2024-03-26T03:25:17.744 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T03:25:17.744 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_adking@teuthology --archive /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451 --name adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi --description orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} --verbose -- /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/orig.config.yaml 2024-03-26T03:25:17.778 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T03:25:17.864 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451 branch: wip-adk-testing-2024-03-25-1628 description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: adking@redhat.com first_in_suite: false job_id: '7622451' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-adk-testing-2024-03-25-1628 ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - MON_DOWN - mons down, quorum - is down \(out of quorum\) - OSDMAP_FLAGS - PG_DEGRADED - OSD_DOWN - POOL_APP_NOT_ENABLED - CACHE_POOL_NO_HIT_SET - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-adk-testing-2024-03-25-1628 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 owner: scheduled_adking@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9192 sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 sleep_before_teardown: 0 subset: 1/16 suite: orch:cephadm suite_branch: wip-adk-testing-2024-03-25-1628 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyiWJ0FQ6XSMIajEJ/7O0IKWmXYqMAG1UOylFEqrCzvT3Q3DPyIL30yjheKqmJShRyRKzzSQXBGZkrtKcYRA/k= smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+f2GjBWCT1ydzKcEaQJWK0YesqjDKanNpmvIou6emAhOAk6hAE/YoDz1vu2Z7hFgRLWIEt89/J800Sl87Ey6g= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_01:50:53 tube: smithi user: adking verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T03:25:17.864 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa; will attempt to use it 2024-03-26T03:25:17.865 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/qa/tasks 2024-03-26T03:25:17.865 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T03:25:17.867 INFO:teuthology.task.internal:Checking packages... 2024-03-26T03:25:17.893 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003' 2024-03-26T03:25:17.893 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T03:25:17.893 INFO:teuthology.packaging:ref: None 2024-03-26T03:25:17.893 INFO:teuthology.packaging:tag: None 2024-03-26T03:25:17.893 INFO:teuthology.packaging:branch: wip-adk-testing-2024-03-25-1628 2024-03-26T03:25:17.893 INFO:teuthology.packaging:sha1: e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:25:17.893 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-adk-testing-2024-03-25-1628 2024-03-26T03:25:18.129 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2475.ge42b8ea3 2024-03-26T03:25:18.132 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T03:25:18.138 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T03:25:18.138 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T03:25:18.162 INFO:teuthology.task.internal:Saving configuration 2024-03-26T03:25:18.174 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T03:25:18.187 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:25:18.212 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:20:17.101050', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyiWJ0FQ6XSMIajEJ/7O0IKWmXYqMAG1UOylFEqrCzvT3Q3DPyIL30yjheKqmJShRyRKzzSQXBGZkrtKcYRA/k='} 2024-03-26T03:25:18.233 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:20:17.102271', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+f2GjBWCT1ydzKcEaQJWK0YesqjDKanNpmvIou6emAhOAk6hAE/YoDz1vu2Z7hFgRLWIEt89/J800Sl87Ey6g='} 2024-03-26T03:25:18.233 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T03:25:18.239 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T03:25:18.240 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T03:25:18.240 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T03:25:18.329 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f65b4b2a160>, signals=[15]) 2024-03-26T03:25:18.330 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T03:25:18.336 INFO:teuthology.task.internal:Opening connections... 2024-03-26T03:25:18.336 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2024-03-26T03:25:18.338 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:25:18.485 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-26T03:25:18.486 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:25:18.577 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T03:25:18.585 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2024-03-26T03:25:18.602 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2024-03-26T03:25:18.603 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="9" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="9" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:25:18.660 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:25:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:25:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:25:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:25:18.661 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:25:18.661 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2024-03-26T03:25:18.682 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-26T03:25:18.700 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-26T03:25:18.701 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:25:18.756 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:25:18.757 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:25:18.757 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-26T03:25:18.782 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T03:25:18.791 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T03:25:18.812 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T03:25:18.812 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:25:18.814 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:25:18.830 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T03:25:18.836 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T03:25:18.836 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:25:18.870 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:25:19.040 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T03:25:19.093 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T03:25:19.108 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T03:25:19.108 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T03:25:19.108 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:25:19.109 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:25:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:25:19.125 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-26T03:25:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:25:19.126 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-26T03:25:19.181 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:25:19.182 INFO:teuthology.task.kernel:Running kernel on smithi006: 5.14.0-361.el9.x86_64 2024-03-26T03:25:19.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2024-03-26T03:25:19.183 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:25:19.183 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-03-26T03:25:19.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-03-26T03:25:21.226 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T03:25:21.240 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T03:25:22.934 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2024-03-26T03:25:23.098 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 2.6 MB/s | 2.0 MB 00:00 2024-03-26T03:25:26.224 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-26T03:25:27.089 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-26T03:25:28.151 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-26T03:25:28.881 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-26T03:25:33.557 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 6.1 MB/s | 6.1 MB 00:01 2024-03-26T03:25:33.770 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 3.2 MB/s | 6.1 MB 00:01 2024-03-26T03:25:35.108 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-03-26T03:25:35.411 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-03-26T03:25:35.934 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-26T03:25:36.879 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-26T03:25:40.809 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-26T03:25:42.220 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-03-26T03:25:42.966 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:25:42.966 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:25:43.064 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:25:43.070 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:25:43.070 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-26T03:25:43.070 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:25:43.070 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:25:43.071 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 76 M 2024-03-26T03:25:43.072 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-03-26T03:25:43.072 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:25:43.477 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 486 kB/s | 66 kB 00:00 2024-03-26T03:25:44.469 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:25:44.469 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:25:44.559 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:25:44.565 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:25:44.565 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-26T03:25:44.565 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:25:44.565 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:25:44.565 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout:Install 4 Packages 2024-03-26T03:25:44.566 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:25:44.567 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 76 M 2024-03-26T03:25:44.567 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 123 M 2024-03-26T03:25:44.567 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:25:45.030 INFO:teuthology.orchestra.run.smithi006.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 190 kB/s | 66 kB 00:00 2024-03-26T03:25:45.244 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.8 MB/s | 15 MB 00:01 2024-03-26T03:25:46.736 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 8.4 MB/s | 27 MB 00:03 2024-03-26T03:25:48.286 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 6.8 MB/s | 33 MB 00:04 2024-03-26T03:25:48.286 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:25:48.286 INFO:teuthology.orchestra.run.smithi086.stdout:Total 15 MB/s | 76 MB 00:05 2024-03-26T03:25:48.951 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:25:49.307 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:25:49.307 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:25:50.324 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:25:50.324 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:25:52.773 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:25:53.366 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:25:53.469 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:25:55.505 INFO:teuthology.orchestra.run.smithi006.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.4 MB/s | 15 MB 00:10 2024-03-26T03:25:56.198 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:25:56.320 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:26:01.029 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:26:01.070 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:03.379 INFO:teuthology.orchestra.run.smithi006.stdout:(3/4): kernel-modules-core-5.14.0-432.el9.x86_6 1.5 MB/s | 27 MB 00:18 2024-03-26T03:26:05.834 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:22.086 INFO:teuthology.orchestra.run.smithi006.stdout:(4/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 916 kB/s | 33 MB 00:37 2024-03-26T03:26:22.087 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:26:22.087 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.0 MB/s | 76 MB 00:37 2024-03-26T03:26:22.776 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:26:23.198 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:26:23.198 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:26:24.248 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:26:24.249 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:26:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:26:27.487 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:26:27.570 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:26:30.472 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:26:30.606 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:26:35.410 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:35.411 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:26:35.411 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:26:35.454 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:35.679 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:26:35.724 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:35.793 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:35.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:26:35.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:26:35.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:26:36.189 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:26:36.189 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:26:36.189 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:26:36.190 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:26:36.512 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:26:36.650 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.650 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:26:36.650 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:26:36.650 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:26:36.650 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:26:36.651 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T03:26:36.651 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:26:36.651 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:26:36.730 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:26:36.730 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:26:36.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-03-26T03:26:36.764 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-03-26T03:26:36.764 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:26:36.765 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:26:36.766 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:26:36.767 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:26:36.769 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:26:36.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:26:37.361 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-03-26T03:26:38.630 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:26:38.668 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-03-26T03:26:38.670 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-26T03:26:38.685 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.aGvB15sG9c 2024-03-26T03:26:38.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.aGvB15sG9c 2024-03-26T03:26:38.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.aGvB15sG9c 2024-03-26T03:26:38.894 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.aGvB15sG9c is 6KB 2024-03-26T03:26:38.943 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.aGvB15sG9c 2024-03-26T03:26:38.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:26:39.028 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:26:39.028 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:26:39.028 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:26:39.028 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:26:39.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:26:39.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-03-26T03:26:40.823 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:27:09.166 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T03:27:09.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-26T03:27:09.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:27:11.869 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:27:11.869 INFO:teuthology.orchestra.run.smithi006.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:27:11.869 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:27:11.913 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:27:12.272 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:27:12.272 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:27:12.272 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:27:12.272 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:27:12.703 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:27:12.703 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:27:12.704 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:27:13.053 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:27:13.160 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:27:13.160 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:27:13.160 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:27:13.161 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:27:13.161 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:27:13.161 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T03:27:13.161 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:27:13.161 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:27:13.243 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:27:13.244 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:27:13.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.06 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 68.el9 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Group : Unspecified 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 8264254 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:27:13.282 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : builder@centos.org 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:27:13.283 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:27:13.284 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:27:13.284 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:27:13.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:27:13.890 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2024-03-26T03:27:15.264 INFO:teuthology.orchestra.run.smithi006.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:27:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:done 2024-03-26T03:27:15.319 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2024-03-26T03:27:15.342 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.ptChSdVx9i 2024-03-26T03:27:15.343 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ptChSdVx9i 2024-03-26T03:27:15.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.ptChSdVx9i 2024-03-26T03:27:15.565 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.ptChSdVx9i is 6KB 2024-03-26T03:27:15.585 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.ptChSdVx9i 2024-03-26T03:27:15.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:27:15.668 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:27:15.668 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:27:15.668 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:27:15.668 INFO:teuthology.orchestra.run.smithi006.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:27:15.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:27:15.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2024-03-26T03:27:45.864 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T03:27:45.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:27:45.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:28:04.383 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-26T03:28:09.175 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T03:28:13.388 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:28:13.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:28:16.446 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2024-03-26T03:28:18.177 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-26T03:28:18.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:28:21.672 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-26T03:28:22.115 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-26T03:28:22.115 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T03:28:22.115 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-26T03:28:22.129 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-432.el9.x86_64 2024-03-26T03:28:22.129 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T03:28:22.129 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T03:28:22.129 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T03:28:23.130 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T03:28:23.130 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T03:28:23.184 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-03-26T03:28:23.221 DEBUG:teuthology.parallel:result is None 2024-03-26T03:28:28.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:28:28.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:29:00.584 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-26T03:29:01.100 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:29:01.100 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T03:29:01.100 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2024-03-26T03:29:01.114 INFO:teuthology.orchestra.run.smithi006.stdout:5.14.0-432.el9.x86_64 2024-03-26T03:29:01.114 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T03:29:01.114 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T03:29:01.114 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T03:29:02.115 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T03:29:02.115 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T03:29:02.174 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2024-03-26T03:29:02.213 DEBUG:teuthology.parallel:result is None 2024-03-26T03:29:02.214 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T03:29:02.226 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T03:29:02.226 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T03:29:02.229 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T03:29:02.248 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T03:29:02.260 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T03:29:02.299 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T03:29:02.299 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T03:29:02.302 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T03:29:02.329 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T03:29:02.343 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T03:29:02.343 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T03:29:02.366 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T03:29:02.405 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:29:02.409 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:29:02.419 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:29:02.423 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:29:02.425 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T03:29:02.436 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T03:29:02.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T03:29:02.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T03:29:02.494 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T03:29:02.505 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T03:29:02.505 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T03:29:02.535 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T03:29:02.551 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:29:02.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:29:02.689 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:29:02.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:29:02.816 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:29:02.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T03:29:02.885 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:29:02.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:29:02.980 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:29:03.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:29:03.109 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:29:03.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T03:29:03.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2024-03-26T03:29:03.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-26T03:29:03.231 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:29:03.277 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:29:03.641 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T03:29:03.649 INFO:teuthology.task.internal:Starting timer... 2024-03-26T03:29:03.649 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T03:29:03.703 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T03:29:03.742 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T03:29:03.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2024-03-26T03:29:03.791 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2024-03-26T03:29:03.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-26T03:29:03.872 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-26T03:29:03.874 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T03:29:03.874 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2024-03-26T03:29:03.897 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2024-03-26T03:29:03.898 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-26T03:29:03.929 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-26T03:29:03.929 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi086.front.sepia.ceph.com': 'permissive'} 2024-03-26T03:29:03.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:29:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:29:03.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:29:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:29:03.998 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T03:29:03.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2024-03-26T03:29:04.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-26T03:29:04.073 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T03:29:04.084 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-26T03:29:04.090 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-26T03:29:04.211 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T03:29:04.228 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T03:29:04.229 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi086.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T03:37:44.755 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi086.front.sepia.ceph.com')] 2024-03-26T03:37:44.757 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:37:44.758 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:37:44.848 DEBUG:teuthology.orchestra.run.smithi006:> true 2024-03-26T03:37:44.933 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2024-03-26T03:37:44.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-26T03:37:44.934 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:37:45.023 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-26T03:37:45.101 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-26T03:37:45.101 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T03:37:45.112 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T03:37:45.112 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T03:37:45.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:37:45.114 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T03:37:45.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T03:37:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T03:37:45.163 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T03:37:45.172 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T03:37:45.188 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T03:37:45.194 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2024-03-26T03:37:45.208 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2024-03-26T03:37:45.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-26T03:37:45.224 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T03:37:45.231 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-26T03:37:45.241 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T03:37:45.247 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T03:37:45.264 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T03:37:45.289 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:37:45.293 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:37:45.293 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-26T03:37:45.293 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.294 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.294 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.294 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.314 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-26T03:37:45.317 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T03:37:45.317 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-26T03:37:45.318 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.318 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.318 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.318 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T03:37:45.318 INFO:teuthology.run_tasks:Running task install... 2024-03-26T03:37:45.326 DEBUG:teuthology.task.install:project ceph 2024-03-26T03:37:45.326 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}} 2024-03-26T03:37:45.326 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'} 2024-03-26T03:37:45.327 INFO:teuthology.task.install:Using flavor: default 2024-03-26T03:37:45.332 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T03:37:45.332 INFO:teuthology.task.install:extra packages: [] 2024-03-26T03:37:45.332 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'tag': None, 'wait_for_package': False} 2024-03-26T03:37:45.333 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:37:45.335 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'tag': None, 'wait_for_package': False} 2024-03-26T03:37:45.335 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:37:45.480 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/ 2024-03-26T03:37:45.480 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2475.ge42b8ea3 2024-03-26T03:37:45.559 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/ 2024-03-26T03:37:45.559 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2475.ge42b8ea3 2024-03-26T03:37:45.591 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T03:37:45.591 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:37:45.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T03:37:45.619 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T03:37:45.619 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T03:37:45.666 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T03:37:45.667 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:37:45.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T03:37:45.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T03:37:45.695 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T03:37:45.695 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T03:37:45.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T03:37:45.766 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T03:37:45.832 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-03-26T03:37:45.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-03-26T03:37:45.843 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T03:37:45.910 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2024-03-26T03:37:45.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2024-03-26T03:37:46.082 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-03-26T03:37:46.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-03-26T03:37:46.164 INFO:teuthology.orchestra.run.smithi006.stdout:74 files removed 2024-03-26T03:37:46.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2024-03-26T03:37:46.818 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 202 kB/s | 83 kB 00:00 2024-03-26T03:37:46.869 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 223 kB/s | 83 kB 00:00 2024-03-26T03:37:47.185 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-03-26T03:37:47.212 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 40 kB/s | 12 kB 00:00 2024-03-26T03:37:47.572 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-26T03:37:47.589 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-26T03:37:48.142 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T03:37:48.166 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-26T03:37:49.733 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 4.7 MB/s | 2.0 MB 00:00 2024-03-26T03:37:49.751 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-03-26T03:37:50.490 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-26T03:37:52.200 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - BaseOS 3.9 MB/s | 8.0 MB 00:02 2024-03-26T03:37:54.101 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-26T03:37:58.326 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-26T03:37:59.794 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-26T03:38:01.029 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-26T03:38:06.129 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-03-26T03:38:07.415 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-26T03:38:08.694 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 9.8 M 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2024-03-26T03:38:08.708 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.4 M 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.4 M 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 21 M 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 25 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 746 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T03:38:08.709 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 501 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.1 M 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 45 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 134 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 161 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T03:38:08.710 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 321 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 298 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 99 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout:Installing weak dependencies: 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout:Install 30 Packages 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2024-03-26T03:38:08.711 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:08.712 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 59 M 2024-03-26T03:38:08.712 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:38:09.681 INFO:teuthology.orchestra.run.smithi006.stdout:(1/32): ceph-base-19.0.0-2475.ge42b8ea3.el9.x86 7.1 MB/s | 5.4 MB 00:00 2024-03-26T03:38:09.739 INFO:teuthology.orchestra.run.smithi006.stdout:(2/32): ceph-selinux-19.0.0-2475.ge42b8ea3.el9. 432 kB/s | 25 kB 00:00 2024-03-26T03:38:09.831 INFO:teuthology.orchestra.run.smithi006.stdout:(3/32): libcephfs2-19.0.0-2475.ge42b8ea3.el9.x8 8.0 MB/s | 746 kB 00:00 2024-03-26T03:38:09.940 INFO:teuthology.orchestra.run.smithi006.stdout:(4/32): libradosstriper1-19.0.0-2475.ge42b8ea3. 4.5 MB/s | 501 kB 00:00 2024-03-26T03:38:10.115 INFO:teuthology.orchestra.run.smithi006.stdout:(5/32): ceph-radosgw-19.0.0-2475.ge42b8ea3.el9. 8.3 MB/s | 9.8 MB 00:01 2024-03-26T03:38:10.182 INFO:teuthology.orchestra.run.smithi006.stdout:(6/32): python3-ceph-argparse-19.0.0-2475.ge42b 679 kB/s | 45 kB 00:00 2024-03-26T03:38:10.240 INFO:teuthology.orchestra.run.smithi006.stdout:(7/32): python3-ceph-common-19.0.0-2475.ge42b8e 2.3 MB/s | 134 kB 00:00 2024-03-26T03:38:10.307 INFO:teuthology.orchestra.run.smithi006.stdout:(8/32): python3-cephfs-19.0.0-2475.ge42b8ea3.el 2.4 MB/s | 161 kB 00:00 2024-03-26T03:38:10.391 INFO:teuthology.orchestra.run.smithi006.stdout:(9/32): librgw2-19.0.0-2475.ge42b8ea3.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-26T03:38:10.416 INFO:teuthology.orchestra.run.smithi006.stdout:(10/32): python3-rados-19.0.0-2475.ge42b8ea3.el 2.9 MB/s | 321 kB 00:00 2024-03-26T03:38:10.441 INFO:teuthology.orchestra.run.smithi006.stdout:(11/32): python3-rbd-19.0.0-2475.ge42b8ea3.el9. 5.8 MB/s | 298 kB 00:00 2024-03-26T03:38:10.467 INFO:teuthology.orchestra.run.smithi006.stdout:(12/32): python3-rgw-19.0.0-2475.ge42b8ea3.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T03:38:10.592 INFO:teuthology.orchestra.run.smithi006.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 704 kB/s | 106 kB 00:00 2024-03-26T03:38:10.618 INFO:teuthology.orchestra.run.smithi006.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 159 kB 00:00 2024-03-26T03:38:10.643 INFO:teuthology.orchestra.run.smithi006.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-26T03:38:10.701 INFO:teuthology.orchestra.run.smithi006.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-26T03:38:10.727 INFO:teuthology.orchestra.run.smithi006.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-26T03:38:10.752 INFO:teuthology.orchestra.run.smithi006.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 823 kB/s | 41 kB 00:00 2024-03-26T03:38:10.836 INFO:teuthology.orchestra.run.smithi006.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 431 kB/s | 36 kB 00:00 2024-03-26T03:38:10.895 INFO:teuthology.orchestra.run.smithi006.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-03-26T03:38:10.920 INFO:teuthology.orchestra.run.smithi006.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.1 MB/s | 221 kB 00:00 2024-03-26T03:38:11.120 INFO:teuthology.orchestra.run.smithi006.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-26T03:38:11.162 INFO:teuthology.orchestra.run.smithi006.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-26T03:38:11.212 INFO:teuthology.orchestra.run.smithi006.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 983 kB/s | 49 kB 00:00 2024-03-26T03:38:11.313 INFO:teuthology.orchestra.run.smithi006.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-26T03:38:11.338 INFO:teuthology.orchestra.run.smithi006.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 550 kB/s | 67 kB 00:00 2024-03-26T03:38:11.397 INFO:teuthology.orchestra.run.smithi006.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 9.8 MB/s | 838 kB 00:00 2024-03-26T03:38:11.421 INFO:teuthology.orchestra.run.smithi006.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-26T03:38:11.440 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-26T03:38:11.622 INFO:teuthology.orchestra.run.smithi006.stdout:(29/32): ceph-common-19.0.0-2475.ge42b8ea3.el9. 7.7 MB/s | 21 MB 00:02 2024-03-26T03:38:11.664 INFO:teuthology.orchestra.run.smithi006.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-03-26T03:38:11.831 INFO:teuthology.orchestra.run.smithi006.stdout:(31/32): librados2-19.0.0-2475.ge42b8ea3.el9.x8 8.3 MB/s | 3.4 MB 00:00 2024-03-26T03:38:11.948 INFO:teuthology.orchestra.run.smithi006.stdout:(32/32): librbd1-19.0.0-2475.ge42b8ea3.el9.x86_ 9.7 MB/s | 3.1 MB 00:00 2024-03-26T03:38:11.952 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:38:11.953 INFO:teuthology.orchestra.run.smithi006.stdout:Total 18 MB/s | 59 MB 00:03 2024-03-26T03:38:11.984 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:38:12.024 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:38:12.024 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:38:12.515 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:38:12.515 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:38:12.966 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 44 kB/s | 16 kB 00:00 2024-03-26T03:38:14.013 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:38:14.085 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T03:38:14.119 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T03:38:14.145 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T03:38:14.167 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T03:38:14.184 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T03:38:14.360 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T03:38:14.415 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T03:38:14.804 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T03:38:14.838 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:38:14.914 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:38:15.004 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 8/34 2024-03-26T03:38:15.031 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T03:38:15.111 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T03:38:15.139 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T03:38:15.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 11/34 2024-03-26T03:38:15.287 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T03:38:15.317 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T03:38:15.381 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T03:38:15.415 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T03:38:15.526 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T03:38:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T03:38:15.615 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T03:38:15.648 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T03:38:15.677 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T03:38:15.710 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T03:38:16.047 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T03:38:16.081 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T03:38:16.156 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T03:38:16.186 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 23/34 2024-03-26T03:38:16.222 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T03:38:16.707 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T03:38:16.734 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T03:38:16.788 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T03:38:16.859 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 27/34 2024-03-26T03:38:16.886 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 28/34 2024-03-26T03:38:19.162 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:38:19.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:38:19.782 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:38:19.819 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T03:38:19.893 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T03:38:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T03:38:19.894 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T03:38:19.894 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:19.929 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:38:29.981 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /root 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2024-03-26T03:38:29.982 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:30.945 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T03:38:30.969 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:30.972 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T03:38:31.067 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T03:38:31.067 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/34 2024-03-26T03:38:32.140 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 6/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 8/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 9/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 10/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 11/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 13/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T03:38:32.141 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T03:38:32.142 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T03:38:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T03:38:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:38:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T03:38:32.143 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/34 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T03:38:33.385 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.386 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T03:38:33.387 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T03:38:33.388 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:33.388 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:38:33.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2024-03-26T03:38:34.292 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:38:34.447 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:38:34.447 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 48 M 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 163 k 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-03-26T03:38:34.448 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:34.449 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 49 M 2024-03-26T03:38:34.449 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 206 M 2024-03-26T03:38:34.449 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:38:34.895 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-26T03:38:34.945 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-26T03:38:34.978 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T03:38:35.112 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): libcephsqlite-19.0.0-2475.ge42b8ea3.el9. 418 kB/s | 163 kB 00:00 2024-03-26T03:38:35.279 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 623 kB/s | 187 kB 00:00 2024-03-26T03:38:37.063 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): ceph-test-19.0.0-2475.ge42b8ea3.el9.x86_ 20 MB/s | 48 MB 00:02 2024-03-26T03:38:37.065 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:38:37.066 INFO:teuthology.orchestra.run.smithi006.stdout:Total 19 MB/s | 49 MB 00:02 2024-03-26T03:38:37.080 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:38:37.103 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:38:37.103 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:38:37.501 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:38:37.501 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:38:38.169 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:38:38.234 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T03:38:38.273 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T03:38:38.304 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T03:38:38.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T03:38:38.374 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T03:38:43.155 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T03:38:43.243 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T03:38:45.668 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T03:38:45.668 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/6 2024-03-26T03:38:45.668 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/6 2024-03-26T03:38:45.668 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T03:38:45.668 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T03:38:45.669 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T03:38:46.034 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T03:38:46.034 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:46.034 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:38:46.034 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-16.el9.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:46.035 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:38:46.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2024-03-26T03:38:46.892 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:40 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:38:47.051 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 6.6 k 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 2.4 M 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 1.5 M 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 249 k 2024-03-26T03:38:47.054 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 4.7 M 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 17 M 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T03:38:47.055 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T03:38:47.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:========================================================================================= 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:Install 36 Packages 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 30 M 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 108 M 2024-03-26T03:38:47.057 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:38:47.502 INFO:teuthology.orchestra.run.smithi006.stdout:(1/36): ceph-19.0.0-2475.ge42b8ea3.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-26T03:38:47.635 INFO:teuthology.orchestra.run.smithi006.stdout:(2/36): ceph-mgr-19.0.0-2475.ge42b8ea3.el9.x86_ 4.3 MB/s | 1.5 MB 00:00 2024-03-26T03:38:47.952 INFO:teuthology.orchestra.run.smithi006.stdout:(3/36): ceph-mds-19.0.0-2475.ge42b8ea3.el9.x86_ 3.5 MB/s | 2.4 MB 00:00 2024-03-26T03:38:48.061 INFO:teuthology.orchestra.run.smithi006.stdout:(4/36): ceph-mgr-modules-core-19.0.0-2475.ge42b 2.2 MB/s | 249 kB 00:00 2024-03-26T03:38:48.203 INFO:teuthology.orchestra.run.smithi006.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-26T03:38:48.245 INFO:teuthology.orchestra.run.smithi006.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T03:38:48.270 INFO:teuthology.orchestra.run.smithi006.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-26T03:38:48.295 INFO:teuthology.orchestra.run.smithi006.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-26T03:38:48.320 INFO:teuthology.orchestra.run.smithi006.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-26T03:38:48.362 INFO:teuthology.orchestra.run.smithi006.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-26T03:38:48.538 INFO:teuthology.orchestra.run.smithi006.stdout:(11/36): ceph-osd-19.0.0-2475.ge42b8ea3.el9.x86 19 MB/s | 17 MB 00:00 2024-03-26T03:38:48.588 INFO:teuthology.orchestra.run.smithi006.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.1 MB/s | 1.1 MB 00:00 2024-03-26T03:38:48.613 INFO:teuthology.orchestra.run.smithi006.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-26T03:38:48.638 INFO:teuthology.orchestra.run.smithi006.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-26T03:38:48.664 INFO:teuthology.orchestra.run.smithi006.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-26T03:38:48.706 INFO:teuthology.orchestra.run.smithi006.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 539 kB/s | 22 kB 00:00 2024-03-26T03:38:48.748 INFO:teuthology.orchestra.run.smithi006.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 1.1 MB/s | 125 kB 00:00 2024-03-26T03:38:48.990 INFO:teuthology.orchestra.run.smithi006.stdout:(18/36): ceph-mon-19.0.0-2475.ge42b8ea3.el9.x86 3.2 MB/s | 4.7 MB 00:01 2024-03-26T03:38:49.032 INFO:teuthology.orchestra.run.smithi006.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 153 kB/s | 43 kB 00:00 2024-03-26T03:38:49.140 INFO:teuthology.orchestra.run.smithi006.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 348 kB/s | 151 kB 00:00 2024-03-26T03:38:49.232 INFO:teuthology.orchestra.run.smithi006.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-26T03:38:49.391 INFO:teuthology.orchestra.run.smithi006.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 111 kB/s | 18 kB 00:00 2024-03-26T03:38:49.416 INFO:teuthology.orchestra.run.smithi006.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 933 kB/s | 358 kB 00:00 2024-03-26T03:38:49.441 INFO:teuthology.orchestra.run.smithi006.stdout:(24/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 381 kB/s | 172 kB 00:00 2024-03-26T03:38:49.483 INFO:teuthology.orchestra.run.smithi006.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 251 kB/s | 23 kB 00:00 2024-03-26T03:38:49.508 INFO:teuthology.orchestra.run.smithi006.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-26T03:38:49.534 INFO:teuthology.orchestra.run.smithi006.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 213 kB/s | 20 kB 00:00 2024-03-26T03:38:49.576 INFO:teuthology.orchestra.run.smithi006.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-03-26T03:38:49.601 INFO:teuthology.orchestra.run.smithi006.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 854 kB/s | 79 kB 00:00 2024-03-26T03:38:49.668 INFO:teuthology.orchestra.run.smithi006.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-26T03:38:49.693 INFO:teuthology.orchestra.run.smithi006.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 979 kB/s | 90 kB 00:00 2024-03-26T03:38:49.718 INFO:teuthology.orchestra.run.smithi006.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-26T03:38:49.760 INFO:teuthology.orchestra.run.smithi006.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 387 kB/s | 36 kB 00:00 2024-03-26T03:38:49.786 INFO:teuthology.orchestra.run.smithi006.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-26T03:38:49.852 INFO:teuthology.orchestra.run.smithi006.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 217 kB/s | 20 kB 00:00 2024-03-26T03:38:49.886 INFO:teuthology.orchestra.run.smithi006.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.5 MB/s | 427 kB 00:00 2024-03-26T03:38:49.890 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:38:49.890 INFO:teuthology.orchestra.run.smithi006.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-26T03:38:49.915 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:38:49.961 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:38:49.961 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:38:50.389 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:38:50.390 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:38:51.841 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:38:51.905 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T03:38:51.945 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T03:38:51.975 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T03:38:52.049 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T03:38:52.099 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T03:38:52.131 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T03:38:52.227 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T03:38:52.321 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T03:38:52.368 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T03:38:52.442 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T03:38:52.620 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T03:38:52.657 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T03:38:52.715 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T03:38:52.778 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T03:38:52.972 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T03:38:53.019 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T03:38:53.058 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T03:38:53.093 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T03:38:53.122 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T03:38:53.201 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T03:38:53.232 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T03:38:53.266 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T03:38:53.306 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T03:38:53.341 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T03:38:53.376 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T03:38:53.469 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T03:38:53.515 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T03:38:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T03:38:53.732 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T03:38:53.855 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T03:38:54.023 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 31/36 2024-03-26T03:38:54.045 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T03:38:54.070 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T03:38:54.070 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:38:54.070 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T03:38:54.070 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T03:38:54.071 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T03:38:54.071 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:55.587 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T03:38:55.613 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T03:38:55.614 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:38:55.614 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T03:38:55.614 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T03:38:55.614 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T03:38:55.614 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:56.088 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T03:38:56.111 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:56.347 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T03:38:56.370 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T03:38:56.370 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:38:56.370 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T03:38:56.370 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T03:38:56.370 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T03:38:56.371 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:56.447 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 6/36 2024-03-26T03:38:58.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T03:38:58.122 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T03:38:58.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T03:38:58.124 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T03:38:58.124 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T03:38:58.124 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T03:38:58.124 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T03:38:58.124 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T03:38:59.185 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T03:38:59.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T03:38:59.186 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T03:38:59.187 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:38:59.188 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:38:59.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2024-03-26T03:39:00.233 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:00.332 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:00.388 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:00.389 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:00.389 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:00.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2024-03-26T03:39:01.021 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:01.178 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:01.178 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 762 k 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 762 k 2024-03-26T03:39:01.179 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 769 k 2024-03-26T03:39:01.180 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:01.463 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-19.0.0-2475.ge42b8ea3.el9.noarch.rpm 2.6 MB/s | 762 kB 00:00 2024-03-26T03:39:01.463 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:01.463 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.6 MB/s | 762 kB 00:00 2024-03-26T03:39:01.464 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:01.469 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:01.469 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:01.475 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:01.475 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:01.549 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:02.035 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:39:02.112 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:39:02.479 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:02.766 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:02.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T03:39:03.531 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:03.688 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 145 k 2024-03-26T03:39:03.689 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 145 k 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 435 k 2024-03-26T03:39:03.690 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:03.918 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-19.0.0-2475.ge42b8e 635 kB/s | 145 kB 00:00 2024-03-26T03:39:03.919 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:03.919 INFO:teuthology.orchestra.run.smithi006.stdout:Total 631 kB/s | 145 kB 00:00 2024-03-26T03:39:03.919 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:03.927 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:03.927 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:03.982 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:03.982 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:04.249 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:04.311 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:39:04.335 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:39:04.335 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:39:04.335 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T03:39:04.335 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:05.291 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:05.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2024-03-26T03:39:06.120 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:06.220 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:06.275 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:06.276 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:06.276 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:06.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T03:39:06.923 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:07.077 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 3.5 M 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 34 k 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 16 k 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T03:39:07.079 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout:====================================================================================== 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-03-26T03:39:07.080 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:07.081 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 8.6 M 2024-03-26T03:39:07.081 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 104 M 2024-03-26T03:39:07.081 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:07.439 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): ceph-grafana-dashboards-19.0.0-2475.ge4 187 kB/s | 34 kB 00:00 2024-03-26T03:39:07.539 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): ceph-prometheus-alerts-19.0.0-2475.ge42 58 kB/s | 16 kB 00:00 2024-03-26T03:39:07.597 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-26T03:39:07.623 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-26T03:39:07.790 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.4 MB/s | 863 kB 00:00 2024-03-26T03:39:07.857 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): ceph-mgr-dashboard-19.0.0-2475.ge42b8ea 5.9 MB/s | 3.5 MB 00:00 2024-03-26T03:39:08.049 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 75 kB/s | 19 kB 00:00 2024-03-26T03:39:08.216 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 925 kB/s | 548 kB 00:00 2024-03-26T03:39:08.474 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 340 kB/s | 144 kB 00:00 2024-03-26T03:39:08.491 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 112 kB/s | 31 kB 00:00 2024-03-26T03:39:08.692 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-26T03:39:08.908 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 435 kB/s | 188 kB 00:00 2024-03-26T03:39:08.913 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:08.913 INFO:teuthology.orchestra.run.smithi006.stdout:Total 4.7 MB/s | 8.6 MB 00:01 2024-03-26T03:39:08.933 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:08.953 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:08.953 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:09.142 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:09.143 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:09.665 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:09.735 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T03:39:09.778 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T03:39:09.825 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T03:39:09.854 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T03:39:09.972 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T03:39:10.102 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T03:39:10.193 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T03:39:10.235 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T03:39:10.263 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T03:39:10.304 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 10/12 2024-03-26T03:39:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 11/12 2024-03-26T03:39:11.445 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 1/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 2/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 3/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T03:39:12.768 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T03:39:13.123 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T03:39:13.123 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:13.123 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:13.124 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:13.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T03:39:13.942 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:14.101 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 7.4 M 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T03:39:14.102 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout:============================================================================================ 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout:Install 12 Packages 2024-03-26T03:39:14.103 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:14.104 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 37 M 2024-03-26T03:39:14.104 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 209 M 2024-03-26T03:39:14.104 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:14.474 INFO:teuthology.orchestra.run.smithi006.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 227 kB/s | 32 kB 00:00 2024-03-26T03:39:14.508 INFO:teuthology.orchestra.run.smithi006.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-26T03:39:14.549 INFO:teuthology.orchestra.run.smithi006.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 842 kB/s | 34 kB 00:00 2024-03-26T03:39:14.608 INFO:teuthology.orchestra.run.smithi006.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-26T03:39:14.767 INFO:teuthology.orchestra.run.smithi006.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2024-03-26T03:39:14.842 INFO:teuthology.orchestra.run.smithi006.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-26T03:39:14.867 INFO:teuthology.orchestra.run.smithi006.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-26T03:39:15.160 INFO:teuthology.orchestra.run.smithi006.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 50 MB/s | 16 MB 00:00 2024-03-26T03:39:15.651 INFO:teuthology.orchestra.run.smithi006.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 397 kB/s | 195 kB 00:00 2024-03-26T03:39:15.702 INFO:teuthology.orchestra.run.smithi006.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 965 kB/s | 803 kB 00:00 2024-03-26T03:39:15.794 INFO:teuthology.orchestra.run.smithi006.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-2 5.1 MB/s | 7.4 MB 00:01 2024-03-26T03:39:15.835 INFO:teuthology.orchestra.run.smithi006.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-26T03:39:15.838 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:15.839 INFO:teuthology.orchestra.run.smithi006.stdout:Total 21 MB/s | 37 MB 00:01 2024-03-26T03:39:15.882 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:15.908 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:15.908 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:16.293 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:16.293 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:17.004 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:17.084 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T03:39:17.123 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T03:39:17.223 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T03:39:17.260 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T03:39:17.444 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T03:39:17.475 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T03:39:17.559 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T03:39:18.250 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T03:39:18.359 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T03:39:19.682 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T03:39:20.771 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T03:39:20.799 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 1/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T03:39:23.329 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T03:39:23.330 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T03:39:23.775 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T03:39:23.775 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:23.775 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T03:39:23.776 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T03:39:23.777 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T03:39:23.777 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T03:39:23.777 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:23.777 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:24.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2024-03-26T03:39:24.723 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:24.881 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 49 k 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:24.882 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:24.883 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2024-03-26T03:39:24.884 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:24.884 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2024-03-26T03:39:24.884 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2024-03-26T03:39:24.884 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:25.225 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 180 kB/s | 24 kB 00:00 2024-03-26T03:39:25.250 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-26T03:39:25.309 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T03:39:25.334 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T03:39:25.360 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T03:39:25.377 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-26T03:39:25.394 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): ceph-mgr-rook-19.0.0-2475.ge42b8ea3.el9 162 kB/s | 49 kB 00:00 2024-03-26T03:39:25.661 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 107 kB/s | 32 kB 00:00 2024-03-26T03:39:25.677 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 50 kB/s | 15 kB 00:00 2024-03-26T03:39:25.853 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 337 kB/s | 59 kB 00:00 2024-03-26T03:39:25.886 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 412 kB/s | 203 kB 00:00 2024-03-26T03:39:26.062 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 430 kB/s | 90 kB 00:00 2024-03-26T03:39:26.103 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.3 MB/s | 1.0 MB 00:00 2024-03-26T03:39:26.105 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:26.106 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-26T03:39:26.106 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:26.123 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:26.123 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:26.225 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:26.227 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:26.948 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T03:39:27.006 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T03:39:27.061 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T03:39:27.097 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T03:39:27.139 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T03:39:27.236 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T03:39:27.345 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T03:39:27.400 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T03:39:28.160 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T03:39:28.213 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T03:39:28.246 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T03:39:28.295 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T03:39:28.324 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T03:39:29.114 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T03:39:29.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/13 2024-03-26T03:39:29.114 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T03:39:29.115 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T03:39:29.500 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T03:39:29.500 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T03:39:29.501 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:29.502 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:29.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T03:39:30.321 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:30.478 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 142 k 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T03:39:30.479 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout:======================================================================================== 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.8 M 2024-03-26T03:39:30.480 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 33 M 2024-03-26T03:39:30.481 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:31.039 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-03-26T03:39:31.206 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-26T03:39:31.256 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): ceph-mgr-cephadm-19.0.0-2475.ge42b8ea3.e 399 kB/s | 142 kB 00:00 2024-03-26T03:39:31.707 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 121 kB/s | 58 kB 00:00 2024-03-26T03:39:31.723 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 109 kB/s | 51 kB 00:00 2024-03-26T03:39:31.923 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 620 kB/s | 548 kB 00:00 2024-03-26T03:39:31.927 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:31.927 INFO:teuthology.orchestra.run.smithi006.stdout:Total 4.7 MB/s | 6.8 MB 00:01 2024-03-26T03:39:31.941 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:31.952 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:31.952 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:32.045 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:32.046 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:32.357 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:32.519 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T03:39:32.569 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T03:39:33.114 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T03:39:33.189 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T03:39:33.259 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T03:39:33.287 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T03:39:34.072 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T03:39:34.380 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T03:39:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T03:39:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T03:39:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T03:39:34.381 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:34.381 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:34.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2024-03-26T03:39:35.220 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:29 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 890 k 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:35.377 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 969 k 2024-03-26T03:39:35.379 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 2.8 M 2024-03-26T03:39:35.379 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:35.500 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 650 kB/s | 79 kB 00:00 2024-03-26T03:39:35.700 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-19.0.0-2475.ge42b8ea3.el9.x86_ 2.7 MB/s | 890 kB 00:00 2024-03-26T03:39:35.701 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:35.702 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.9 MB/s | 969 kB 00:00 2024-03-26T03:39:35.702 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:35.724 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:35.724 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:35.801 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:35.801 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:36.003 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:36.148 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T03:39:36.225 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T03:39:36.869 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T03:39:36.869 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/2 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:37.205 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:37.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2024-03-26T03:39:38.069 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:32 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:38.225 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 264 k 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T03:39:38.226 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 332 k 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.5 M 2024-03-26T03:39:38.227 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:38.347 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 578 kB/s | 69 kB 00:00 2024-03-26T03:39:38.780 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-volume-19.0.0-2475.ge42b8ea3.el9.no 477 kB/s | 264 kB 00:00 2024-03-26T03:39:38.782 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:38.782 INFO:teuthology.orchestra.run.smithi006.stdout:Total 599 kB/s | 332 kB 00:00 2024-03-26T03:39:38.782 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:38.791 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:38.792 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:38.834 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:38.834 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:38.989 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:39.168 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T03:39:39.199 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T03:39:39.215 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T03:39:39.215 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:39:39.215 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T03:39:39.215 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:39.959 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/2 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T03:39:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:40.292 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:40.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2024-03-26T03:39:41.210 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:41.364 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:41.365 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:41.365 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-26T03:39:41.365 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:41.365 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 127 k 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 127 k 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 456 k 2024-03-26T03:39:41.366 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:42.077 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-19.0.0-2475.ge42b8ea3.el9.x86_64 178 kB/s | 127 kB 00:00 2024-03-26T03:39:42.078 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:42.078 INFO:teuthology.orchestra.run.smithi006.stdout:Total 178 kB/s | 127 kB 00:00 2024-03-26T03:39:42.078 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:42.085 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:42.085 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:42.150 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:42.150 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:42.306 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:42.412 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:42.825 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:43.121 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:43.122 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:43.122 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:43.123 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:43.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2024-03-26T03:39:43.931 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:44.032 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:44.088 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:44.089 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:44.089 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:44.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2024-03-26T03:39:44.730 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:44.886 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:44.887 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:44.887 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-26T03:39:44.887 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:44.887 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 32 k 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 32 k 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 142 k 2024-03-26T03:39:44.888 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:45.124 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-19.0.0-2475.ge42b8ea3.el9.x86_6 134 kB/s | 32 kB 00:00 2024-03-26T03:39:45.125 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:45.125 INFO:teuthology.orchestra.run.smithi006.stdout:Total 134 kB/s | 32 kB 00:00 2024-03-26T03:39:45.125 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:45.129 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:45.129 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:45.161 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:45.161 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:45.300 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:45.418 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:45.780 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:46.045 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:46.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2024-03-26T03:39:46.772 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:46.871 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:46.926 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:46.927 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:46.927 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:46.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2024-03-26T03:39:47.539 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:47.638 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:47.693 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:47.694 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:47.694 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:47.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2024-03-26T03:39:48.322 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:48.421 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:48.476 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:48.477 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:48.477 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:48.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2024-03-26T03:39:49.094 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:49.193 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:49.248 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:49.249 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:49.249 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:49.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2024-03-26T03:39:49.867 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:49.967 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:50.022 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:50.023 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:50.023 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:50.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2024-03-26T03:39:50.646 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:50.747 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:39:50.803 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:50.804 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2024-03-26T03:39:50.804 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:50.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2024-03-26T03:39:51.432 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:45 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:51.588 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 85 k 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 85 k 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 231 k 2024-03-26T03:39:51.589 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:52.044 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 187 kB/s | 85 kB 00:00 2024-03-26T03:39:52.045 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:52.045 INFO:teuthology.orchestra.run.smithi006.stdout:Total 187 kB/s | 85 kB 00:00 2024-03-26T03:39:52.045 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:52.053 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:52.053 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:52.112 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:52.112 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:52.253 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:52.362 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:52.747 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:53.037 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:53.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2024-03-26T03:39:53.837 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:47 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:53.996 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:53.996 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:53.997 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:53.998 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.1 M 2024-03-26T03:39:53.998 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 13 M 2024-03-26T03:39:53.998 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:54.464 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 6.6 MB/s | 3.1 MB 00:00 2024-03-26T03:39:54.464 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:54.464 INFO:teuthology.orchestra.run.smithi006.stdout:Total 6.6 MB/s | 3.1 MB 00:00 2024-03-26T03:39:54.465 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:54.475 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:54.475 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:54.552 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:54.552 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:54.998 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:55.047 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:55.072 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:55.073 INFO:teuthology.orchestra.run.smithi006.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:39:55.073 INFO:teuthology.orchestra.run.smithi006.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T03:39:55.073 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T03:39:55.073 INFO:teuthology.orchestra.run.smithi006.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T03:39:55.073 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:56.186 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:56.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2024-03-26T03:39:56.988 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:50 ago on Tue 26 Mar 2024 03:38:06 AM UTC. 2024-03-26T03:39:57.143 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 171 k 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2024-03-26T03:39:57.144 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:57.145 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 171 k 2024-03-26T03:39:57.145 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 494 k 2024-03-26T03:39:57.145 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2024-03-26T03:39:57.450 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 562 kB/s | 171 kB 00:00 2024-03-26T03:39:57.451 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:39:57.451 INFO:teuthology.orchestra.run.smithi006.stdout:Total 559 kB/s | 171 kB 00:00 2024-03-26T03:39:57.451 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2024-03-26T03:39:57.459 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2024-03-26T03:39:57.459 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2024-03-26T03:39:57.518 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2024-03-26T03:39:57.518 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2024-03-26T03:39:57.676 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2024-03-26T03:39:57.784 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:58.220 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:39:58.511 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2024-03-26T03:39:58.763 DEBUG:teuthology.parallel:result is None 2024-03-26T03:40:30.004 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 155 kB/s | 21 MB 02:16 2024-03-26T03:40:35.036 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T03:40:37.489 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:40:37.504 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:40:37.504 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 9.8 M 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.4 M 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options 2024-03-26T03:40:37.505 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.4 M 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 21 M 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 25 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 746 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T03:40:37.506 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 501 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 5.1 M 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 45 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 134 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 161 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T03:40:37.507 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 321 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 298 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 99 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:40:37.508 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:40:37.509 INFO:teuthology.orchestra.run.smithi086.stdout:Install 30 Packages 2024-03-26T03:40:37.509 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-03-26T03:40:37.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:40:37.509 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 59 M 2024-03-26T03:40:37.509 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:40:39.162 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-base-19.0.0-2475.ge42b8ea3.el9.x86 4.3 MB/s | 5.4 MB 00:01 2024-03-26T03:40:39.254 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-selinux-19.0.0-2475.ge42b8ea3.el9. 274 kB/s | 25 kB 00:00 2024-03-26T03:40:39.438 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): libcephfs2-19.0.0-2475.ge42b8ea3.el9.x8 4.0 MB/s | 746 kB 00:00 2024-03-26T03:40:39.654 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): ceph-common-19.0.0-2475.ge42b8ea3.el9.x 12 MB/s | 21 MB 00:01 2024-03-26T03:40:39.687 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): libradosstriper1-19.0.0-2475.ge42b8ea3. 2.0 MB/s | 501 kB 00:00 2024-03-26T03:40:39.789 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): ceph-radosgw-19.0.0-2475.ge42b8ea3.el9. 5.3 MB/s | 9.8 MB 00:01 2024-03-26T03:40:39.805 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-ceph-argparse-19.0.0-2475.ge42b 386 kB/s | 45 kB 00:00 2024-03-26T03:40:39.863 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-ceph-common-19.0.0-2475.ge42b8e 1.8 MB/s | 134 kB 00:00 2024-03-26T03:40:39.889 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-cephfs-19.0.0-2475.ge42b8ea3.el 1.9 MB/s | 161 kB 00:00 2024-03-26T03:40:39.956 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): librgw2-19.0.0-2475.ge42b8ea3.el9.x86_ 17 MB/s | 5.1 MB 00:00 2024-03-26T03:40:39.981 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-rbd-19.0.0-2475.ge42b8ea3.el9. 3.2 MB/s | 298 kB 00:00 2024-03-26T03:40:40.006 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): python3-rados-19.0.0-2475.ge42b8ea3.el 2.2 MB/s | 321 kB 00:00 2024-03-26T03:40:40.032 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): python3-rgw-19.0.0-2475.ge42b8ea3.el9. 1.3 MB/s | 99 kB 00:00 2024-03-26T03:40:40.207 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 469 kB/s | 106 kB 00:00 2024-03-26T03:40:40.232 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 705 kB/s | 159 kB 00:00 2024-03-26T03:40:40.249 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 264 kB/s | 44 kB 00:00 2024-03-26T03:40:40.349 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.6 MB/s | 661 kB 00:00 2024-03-26T03:40:40.366 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 353 kB/s | 41 kB 00:00 2024-03-26T03:40:40.392 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 282 kB 00:00 2024-03-26T03:40:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.457 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.458 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.601 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.616 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.640 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:40:40.670 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.681 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.752 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.810 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.817 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.875 INFO:teuthology.orchestra.run.smithi086.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T03:40:40.917 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 90 kB/s | 47 kB 00:00 2024-03-26T03:40:40.942 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 373 kB/s | 221 kB 00:00 2024-03-26T03:40:40.984 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): qatlib-service-24.02.0-1.el9.x86_64.rp 58 kB/s | 36 kB 00:00 2024-03-26T03:40:41.251 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-26T03:40:41.426 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 606 kB/s | 309 kB 00:00 2024-03-26T03:40:41.602 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 139 kB/s | 49 kB 00:00 2024-03-26T03:40:41.793 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 184 kB/s | 67 kB 00:00 2024-03-26T03:40:42.119 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 4.4 MB 00:01 2024-03-26T03:40:42.302 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-03-26T03:40:42.336 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 838 kB 00:00 2024-03-26T03:40:42.794 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): librados2-19.0.0-2475.ge42b8ea3.el9.x8 6.9 MB/s | 3.4 MB 00:00 2024-03-26T03:40:42.861 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): librbd1-19.0.0-2475.ge42b8ea3.el9.x86_ 6.3 MB/s | 3.1 MB 00:00 2024-03-26T03:40:42.911 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-26T03:40:42.917 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:40:42.917 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 59 MB 00:05 2024-03-26T03:40:42.949 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:40:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:40:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:40:43.471 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:40:43.472 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:40:44.945 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:40:45.027 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T03:40:45.066 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T03:40:45.091 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T03:40:45.115 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T03:40:45.131 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T03:40:45.341 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T03:40:45.396 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T03:40:45.742 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T03:40:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:40:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:40:45.966 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 8/34 2024-03-26T03:40:45.995 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T03:40:46.057 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 9/34 2024-03-26T03:40:46.086 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T03:40:46.130 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 11/34 2024-03-26T03:40:46.200 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T03:40:46.222 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T03:40:46.286 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 13/34 2024-03-26T03:40:46.329 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T03:40:46.436 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T03:40:46.480 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T03:40:46.520 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T03:40:46.553 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T03:40:46.582 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T03:40:46.615 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T03:40:46.949 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T03:40:46.978 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T03:40:47.069 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 22/34 2024-03-26T03:40:47.099 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 23/34 2024-03-26T03:40:47.136 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T03:40:47.599 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T03:40:47.622 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T03:40:47.684 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 26/34 2024-03-26T03:40:47.744 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 27/34 2024-03-26T03:40:47.774 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 28/34 2024-03-26T03:40:50.012 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:40:50.046 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:40:50.616 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 29/34 2024-03-26T03:40:50.658 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T03:40:50.745 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 30/34 2024-03-26T03:40:50.745 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T03:40:50.746 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T03:40:50.746 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:40:50.784 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:41:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:41:00.078 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-03-26T03:41:00.078 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-03-26T03:41:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:01.038 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T03:41:01.062 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/34 2024-03-26T03:41:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T03:41:01.062 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T03:41:01.063 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T03:41:01.063 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:01.064 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T03:41:01.149 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T03:41:01.149 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_6 6/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 7/34 2024-03-26T03:41:02.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9. 8/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x8 9/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 10/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 11/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 12/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 13/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T03:41:02.264 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 31/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T03:41:02.265 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/34 2024-03-26T03:41:03.392 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T03:41:03.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:03.392 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-03-26T03:41:03.392 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.393 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T03:41:03.394 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:03.395 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:03.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-03-26T03:41:04.259 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:29 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:04.407 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 48 M 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T03:41:04.408 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 163 k 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:04.409 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 49 M 2024-03-26T03:41:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 206 M 2024-03-26T03:41:04.410 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:04.844 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-26T03:41:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-26T03:41:04.928 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-26T03:41:05.070 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): libcephsqlite-19.0.0-2475.ge42b8ea3.el9. 419 kB/s | 163 kB 00:00 2024-03-26T03:41:05.170 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 774 kB/s | 187 kB 00:00 2024-03-26T03:41:07.046 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-19.0.0-2475.ge42b8ea3.el9.x86_ 20 MB/s | 48 MB 00:02 2024-03-26T03:41:07.049 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:07.049 INFO:teuthology.orchestra.run.smithi086.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-26T03:41:07.063 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:07.084 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:07.084 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:07.470 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:07.470 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:08.051 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:08.113 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T03:41:08.154 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T03:41:08.186 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T03:41:08.228 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T03:41:08.257 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T03:41:12.841 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/6 2024-03-26T03:41:12.915 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 6/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T03:41:15.423 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T03:41:15.755 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-16.el9.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:15.756 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:16.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-03-26T03:41:16.588 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:41 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:16.745 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 6.6 k 2024-03-26T03:41:16.748 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 2.4 M 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 1.5 M 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 249 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 4.7 M 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 17 M 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T03:41:16.749 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T03:41:16.750 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout:Install 36 Packages 2024-03-26T03:41:16.751 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:16.752 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2024-03-26T03:41:16.752 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 108 M 2024-03-26T03:41:16.752 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:17.148 INFO:teuthology.orchestra.run.smithi086.stdout:(1/36): ceph-19.0.0-2475.ge42b8ea3.el9.x86_64.r 37 kB/s | 6.6 kB 00:00 2024-03-26T03:41:17.565 INFO:teuthology.orchestra.run.smithi086.stdout:(2/36): ceph-mon-19.0.0-2475.ge42b8ea3.el9.x86_ 11 MB/s | 4.7 MB 00:00 2024-03-26T03:41:17.749 INFO:teuthology.orchestra.run.smithi086.stdout:(3/36): ceph-mds-19.0.0-2475.ge42b8ea3.el9.x86_ 3.0 MB/s | 2.4 MB 00:00 2024-03-26T03:41:18.391 INFO:teuthology.orchestra.run.smithi086.stdout:(4/36): ceph-osd-19.0.0-2475.ge42b8ea3.el9.x86_ 20 MB/s | 17 MB 00:00 2024-03-26T03:41:18.416 INFO:teuthology.orchestra.run.smithi086.stdout:(5/36): ceph-mgr-modules-core-19.0.0-2475.ge42b 373 kB/s | 249 kB 00:00 2024-03-26T03:41:18.458 INFO:teuthology.orchestra.run.smithi086.stdout:(6/36): ceph-mgr-19.0.0-2475.ge42b8ea3.el9.x86_ 1.0 MB/s | 1.5 MB 00:01 2024-03-26T03:41:18.600 INFO:teuthology.orchestra.run.smithi086.stdout:(7/36): lua-5.4.4-4.el9.x86_64.rpm 895 kB/s | 187 kB 00:00 2024-03-26T03:41:18.625 INFO:teuthology.orchestra.run.smithi086.stdout:(8/36): python3-mako-1.1.4-6.el9.noarch.rpm 691 kB/s | 144 kB 00:00 2024-03-26T03:41:18.650 INFO:teuthology.orchestra.run.smithi086.stdout:(9/36): python3-markupsafe-1.1.1-12.el9.x86_64. 217 kB/s | 32 kB 00:00 2024-03-26T03:41:18.675 INFO:teuthology.orchestra.run.smithi086.stdout:(10/36): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-26T03:41:18.703 INFO:teuthology.orchestra.run.smithi086.stdout:(11/36): python3-toml-0.10.2-6.el9.noarch.rpm 736 kB/s | 39 kB 00:00 2024-03-26T03:41:18.736 INFO:teuthology.orchestra.run.smithi086.stdout:(12/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 241 kB 00:00 2024-03-26T03:41:18.862 INFO:teuthology.orchestra.run.smithi086.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 6.1 MB/s | 1.1 MB 00:00 2024-03-26T03:41:18.887 INFO:teuthology.orchestra.run.smithi086.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 560 kB/s | 103 kB 00:00 2024-03-26T03:41:18.912 INFO:teuthology.orchestra.run.smithi086.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 709 kB/s | 124 kB 00:00 2024-03-26T03:41:18.938 INFO:teuthology.orchestra.run.smithi086.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-26T03:41:19.021 INFO:teuthology.orchestra.run.smithi086.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 206 kB/s | 22 kB 00:00 2024-03-26T03:41:19.080 INFO:teuthology.orchestra.run.smithi086.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 1.0 MB/s | 151 kB 00:00 2024-03-26T03:41:19.113 INFO:teuthology.orchestra.run.smithi086.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2024-03-26T03:41:19.139 INFO:teuthology.orchestra.run.smithi086.stdout:(20/36): python3-requests-2.25.1-8.el9.noarch.r 553 kB/s | 125 kB 00:00 2024-03-26T03:41:19.164 INFO:teuthology.orchestra.run.smithi086.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-26T03:41:19.197 INFO:teuthology.orchestra.run.smithi086.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-26T03:41:19.214 INFO:teuthology.orchestra.run.smithi086.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-26T03:41:19.239 INFO:teuthology.orchestra.run.smithi086.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.8 MB/s | 358 kB 00:00 2024-03-26T03:41:19.256 INFO:teuthology.orchestra.run.smithi086.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 393 kB/s | 23 kB 00:00 2024-03-26T03:41:19.273 INFO:teuthology.orchestra.run.smithi086.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 330 kB/s | 19 kB 00:00 2024-03-26T03:41:19.290 INFO:teuthology.orchestra.run.smithi086.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-03-26T03:41:19.307 INFO:teuthology.orchestra.run.smithi086.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 915 kB/s | 46 kB 00:00 2024-03-26T03:41:19.341 INFO:teuthology.orchestra.run.smithi086.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-26T03:41:19.366 INFO:teuthology.orchestra.run.smithi086.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 852 kB/s | 79 kB 00:00 2024-03-26T03:41:19.383 INFO:teuthology.orchestra.run.smithi086.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 217 kB/s | 16 kB 00:00 2024-03-26T03:41:19.408 INFO:teuthology.orchestra.run.smithi086.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-26T03:41:19.425 INFO:teuthology.orchestra.run.smithi086.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 859 kB/s | 36 kB 00:00 2024-03-26T03:41:19.467 INFO:teuthology.orchestra.run.smithi086.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-26T03:41:19.484 INFO:teuthology.orchestra.run.smithi086.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-26T03:41:19.517 INFO:teuthology.orchestra.run.smithi086.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-26T03:41:19.521 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:19.521 INFO:teuthology.orchestra.run.smithi086.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-26T03:41:19.544 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:19.584 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:19.584 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:19.986 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:19.987 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:21.309 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:21.362 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T03:41:21.401 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T03:41:21.430 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T03:41:21.492 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T03:41:21.546 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T03:41:21.579 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T03:41:21.669 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T03:41:21.748 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T03:41:21.789 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T03:41:21.936 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T03:41:21.990 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T03:41:22.028 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T03:41:22.076 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T03:41:22.132 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T03:41:22.315 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T03:41:22.365 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T03:41:22.396 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T03:41:22.431 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T03:41:22.461 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T03:41:22.536 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T03:41:22.571 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T03:41:22.604 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T03:41:22.644 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T03:41:22.679 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T03:41:22.722 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T03:41:22.811 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T03:41:22.861 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T03:41:22.933 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T03:41:23.069 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T03:41:23.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T03:41:23.353 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 31/36 2024-03-26T03:41:23.382 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T03:41:23.408 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 32/36 2024-03-26T03:41:23.408 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:23.408 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T03:41:23.408 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T03:41:23.408 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T03:41:23.409 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:24.842 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 33/36 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T03:41:24.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:25.309 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 34/36 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T03:41:25.333 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:25.560 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T03:41:25.582 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 35/36 2024-03-26T03:41:25.583 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:25.583 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T03:41:25.583 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T03:41:25.583 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T03:41:25.583 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:25.648 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 36/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 3/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 4/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 5/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9. 6/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T03:41:27.260 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T03:41:27.261 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T03:41:27.262 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T03:41:27.263 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:28.207 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T03:41:28.208 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T03:41:28.209 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:28.210 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:28.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-03-26T03:41:29.150 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:29.246 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:41:29.298 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:29.299 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:41:29.299 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:29.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-03-26T03:41:29.914 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:54 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:30.064 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 762 k 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 762 k 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 769 k 2024-03-26T03:41:30.065 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:30.722 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.0.0-2475.ge42b8ea3.el9.noarch.rpm 1.1 MB/s | 762 kB 00:00 2024-03-26T03:41:30.723 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:30.723 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.1 MB/s | 762 kB 00:00 2024-03-26T03:41:30.724 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:30.729 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:30.729 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:30.735 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:30.736 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:30.814 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:31.255 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:41:31.336 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:41:31.698 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/1 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:31.972 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:32.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T03:41:32.738 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:57 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:32.893 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 145 k 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:32.894 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2024-03-26T03:41:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 435 k 2024-03-26T03:41:32.895 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:33.544 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.0.0-2475.ge42b8e 223 kB/s | 145 kB 00:00 2024-03-26T03:41:33.545 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:33.545 INFO:teuthology.orchestra.run.smithi086.stdout:Total 223 kB/s | 145 kB 00:00 2024-03-26T03:41:33.545 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:33.553 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:33.553 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:33.606 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:33.606 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:33.744 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:33.812 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:41:33.836 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:41:33.837 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:41:33.837 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T03:41:33.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:34.828 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3. 1/1 2024-03-26T03:41:34.828 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:34.829 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:34.829 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:41:34.829 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:34.829 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:35.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-03-26T03:41:35.641 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:00 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:35.738 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:41:35.790 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:35.791 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:41:35.791 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:35.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T03:41:36.413 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:36.564 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 3.5 M 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T03:41:36.566 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 34 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 16 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:36.567 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 8.6 M 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 104 M 2024-03-26T03:41:36.568 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:37.172 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-prometheus-alerts-19.0.0-2475.ge42 83 kB/s | 16 kB 00:00 2024-03-26T03:41:37.231 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): ceph-grafana-dashboards-19.0.0-2475.ge4 132 kB/s | 34 kB 00:00 2024-03-26T03:41:37.306 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 3.1 MB/s | 238 kB 00:00 2024-03-26T03:41:37.356 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-03-26T03:41:37.515 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.1 MB/s | 863 kB 00:00 2024-03-26T03:41:37.632 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-26T03:41:37.824 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): ceph-mgr-dashboard-19.0.0-2475.ge42b8ea 4.2 MB/s | 3.5 MB 00:00 2024-03-26T03:41:37.849 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-26T03:41:37.891 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 739 kB/s | 31 kB 00:00 2024-03-26T03:41:37.949 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 6.4 MB/s | 2.0 MB 00:00 2024-03-26T03:41:37.975 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-26T03:41:38.033 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 691 kB/s | 144 kB 00:00 2024-03-26T03:41:38.036 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:38.036 INFO:teuthology.orchestra.run.smithi086.stdout:Total 5.9 MB/s | 8.6 MB 00:01 2024-03-26T03:41:38.054 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:38.071 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:38.071 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:38.266 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:38.266 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:38.796 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:38.883 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T03:41:38.919 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T03:41:38.965 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T03:41:38.995 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T03:41:39.100 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T03:41:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T03:41:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T03:41:39.375 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T03:41:39.403 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T03:41:39.436 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 10/12 2024-03-26T03:41:40.491 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 11/12 2024-03-26T03:41:40.527 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 12/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el 1/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noa 2/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9 3/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T03:41:41.792 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T03:41:41.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T03:41:41.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T03:41:41.793 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T03:41:42.134 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T03:41:42.135 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T03:41:42.136 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T03:41:42.136 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:42.136 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:42.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T03:41:42.947 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:43.097 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:43.098 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-26T03:41:43.098 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:43.098 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-26T03:41:43.098 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 7.4 M 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T03:41:43.099 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-03-26T03:41:43.100 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-03-26T03:41:43.101 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:43.599 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 182 kB/s | 32 kB 00:00 2024-03-26T03:41:43.623 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 697 kB/s | 17 kB 00:00 2024-03-26T03:41:43.649 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-26T03:41:43.799 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2024-03-26T03:41:43.941 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-03-26T03:41:44.000 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-26T03:41:44.025 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-26T03:41:44.333 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 47 MB/s | 16 MB 00:00 2024-03-26T03:41:44.834 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 391 kB/s | 195 kB 00:00 2024-03-26T03:41:44.892 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 928 kB/s | 803 kB 00:00 2024-03-26T03:41:45.084 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 981 kB/s | 245 kB 00:00 2024-03-26T03:41:45.434 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-2 3.7 MB/s | 7.4 MB 00:02 2024-03-26T03:41:45.437 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:45.437 INFO:teuthology.orchestra.run.smithi086.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-26T03:41:45.478 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:45.502 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:45.502 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:45.864 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:45.864 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:46.583 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:46.657 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T03:41:46.723 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T03:41:46.823 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T03:41:46.858 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T03:41:47.034 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T03:41:47.074 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T03:41:47.156 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T03:41:47.800 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T03:41:47.918 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T03:41:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T03:41:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T03:41:50.206 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T03:41:52.753 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 12/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8 1/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T03:41:52.754 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T03:41:53.196 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:53.197 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:53.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-03-26T03:41:54.129 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:19 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:54.277 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 49 k 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T03:41:54.279 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-03-26T03:41:54.280 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:54.281 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-03-26T03:41:54.281 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-26T03:41:54.281 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:54.522 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 147 kB/s | 24 kB 00:00 2024-03-26T03:41:54.539 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 99 kB/s | 18 kB 00:00 2024-03-26T03:41:54.564 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): ceph-mgr-rook-19.0.0-2475.ge42b8ea3.el9 237 kB/s | 49 kB 00:00 2024-03-26T03:41:54.664 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 174 kB 00:00 2024-03-26T03:41:54.706 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 792 kB/s | 132 kB 00:00 2024-03-26T03:41:54.731 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 211 kB 00:00 2024-03-26T03:41:54.757 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 466 kB/s | 43 kB 00:00 2024-03-26T03:41:54.807 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-26T03:41:54.882 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 183 kB/s | 32 kB 00:00 2024-03-26T03:41:55.007 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 474 kB/s | 59 kB 00:00 2024-03-26T03:41:55.033 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 735 kB/s | 203 kB 00:00 2024-03-26T03:41:55.058 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.1 MB/s | 1.0 MB 00:00 2024-03-26T03:41:55.116 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 824 kB/s | 90 kB 00:00 2024-03-26T03:41:55.118 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:41:55.118 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-26T03:41:55.119 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:41:55.131 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:41:55.131 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:41:55.220 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:41:55.221 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:41:55.734 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:41:55.902 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T03:41:56.042 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T03:41:56.105 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T03:41:56.135 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T03:41:56.175 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T03:41:56.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T03:41:56.386 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T03:41:56.463 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T03:41:57.175 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T03:41:57.224 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T03:41:57.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T03:41:57.306 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T03:41:57.320 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T03:41:57.985 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 13/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T03:41:57.986 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T03:41:58.343 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:58.344 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:41:58.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T03:41:59.154 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:24 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:41:59.304 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 142 k 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:41:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:41:59.306 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2024-03-26T03:41:59.307 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-03-26T03:41:59.307 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:41:59.884 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-26T03:41:59.909 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-19.0.0-2475.ge42b8ea3.e 392 kB/s | 142 kB 00:00 2024-03-26T03:41:59.934 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 641 kB/s | 248 kB 00:00 2024-03-26T03:42:00.101 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 301 kB/s | 58 kB 00:00 2024-03-26T03:42:00.126 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 266 kB/s | 51 kB 00:00 2024-03-26T03:42:00.235 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-26T03:42:00.236 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:00.236 INFO:teuthology.orchestra.run.smithi086.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-03-26T03:42:00.249 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:00.260 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:00.260 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:00.348 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:00.349 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:00.656 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:00.834 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T03:42:00.878 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T03:42:01.408 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T03:42:01.489 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T03:42:01.552 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T03:42:01.582 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 6/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T03:42:02.425 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T03:42:02.725 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T03:42:02.725 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:02.726 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:02.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-03-26T03:42:03.541 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:28 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:03.690 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 890 k 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:03.691 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-26T03:42:03.692 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 969 k 2024-03-26T03:42:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2024-03-26T03:42:03.692 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:03.827 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 582 kB/s | 79 kB 00:00 2024-03-26T03:42:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-19.0.0-2475.ge42b8ea3.el9.x86_ 1.0 MB/s | 890 kB 00:00 2024-03-26T03:42:04.571 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.1 MB/s | 969 kB 00:00 2024-03-26T03:42:04.572 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:04.594 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:04.669 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:04.670 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:04.850 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T03:42:05.053 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T03:42:05.768 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2/2 2024-03-26T03:42:05.769 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/2 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:06.072 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:06.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-03-26T03:42:06.931 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:07.080 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-2475.ge42b8ea3.el9 ceph-noarch 264 k 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-26T03:42:07.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:07.082 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 332 k 2024-03-26T03:42:07.082 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.5 M 2024-03-26T03:42:07.082 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:07.207 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 551 kB/s | 69 kB 00:00 2024-03-26T03:42:07.732 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-volume-19.0.0-2475.ge42b8ea3.el9.no 406 kB/s | 264 kB 00:00 2024-03-26T03:42:07.733 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:07.734 INFO:teuthology.orchestra.run.smithi086.stdout:Total 509 kB/s | 332 kB 00:00 2024-03-26T03:42:07.734 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:07.743 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:07.743 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:07.783 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:07.783 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:07.932 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:08.120 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T03:42:08.159 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T03:42:08.175 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2/2 2024-03-26T03:42:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:42:08.175 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T03:42:08.175 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:08.894 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 1/2 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-2475.ge42b8ea3.el9.noarch 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:09.225 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:09.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-03-26T03:42:10.119 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:35 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:10.267 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 127 k 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:42:10.268 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:10.269 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 127 k 2024-03-26T03:42:10.269 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-03-26T03:42:10.269 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:10.579 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.0.0-2475.ge42b8ea3.el9.x86_64 409 kB/s | 127 kB 00:00 2024-03-26T03:42:10.579 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:10.580 INFO:teuthology.orchestra.run.smithi086.stdout:Total 407 kB/s | 127 kB 00:00 2024-03-26T03:42:10.580 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:10.585 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:10.585 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:10.645 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:10.645 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:10.797 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:10.910 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:11.356 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:11.623 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:11.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-03-26T03:42:12.395 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:12.494 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:12.548 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:12.549 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:12.549 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:12.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-03-26T03:42:13.170 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:13.319 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:13.319 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:13.319 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-26T03:42:13.319 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 32 k 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:13.320 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 32 k 2024-03-26T03:42:13.321 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 142 k 2024-03-26T03:42:13.321 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:13.491 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.0.0-2475.ge42b8ea3.el9.x86_6 186 kB/s | 32 kB 00:00 2024-03-26T03:42:13.492 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:13.492 INFO:teuthology.orchestra.run.smithi086.stdout:Total 184 kB/s | 32 kB 00:00 2024-03-26T03:42:13.492 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:13.496 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:13.496 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:13.532 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:13.533 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:13.652 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:13.747 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:14.124 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:14.394 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:14.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:14.395 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:14.395 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:42:14.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:14.395 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:14.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-03-26T03:42:15.133 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:15.229 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:15.280 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:15.281 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:15.281 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:15.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-03-26T03:42:15.893 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:15.990 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:16.042 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:16.043 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:16.043 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:16.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-03-26T03:42:16.655 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:16.752 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:16.803 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:16.804 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:16.805 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:16.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-03-26T03:42:17.412 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:17.507 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:17.560 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:17.561 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:17.561 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:17.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-03-26T03:42:18.162 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:18.258 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:18.309 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:18.310 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:18.310 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:18.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-03-26T03:42:18.929 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:43 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:19.025 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 is already installed. 2024-03-26T03:42:19.076 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:19.077 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-26T03:42:19.077 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:19.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-03-26T03:42:19.694 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:44 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:19.845 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:19.845 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:19.845 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 85 k 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2024-03-26T03:42:19.846 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:20.044 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 427 kB/s | 85 kB 00:00 2024-03-26T03:42:20.045 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:20.045 INFO:teuthology.orchestra.run.smithi086.stdout:Total 424 kB/s | 85 kB 00:00 2024-03-26T03:42:20.045 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:20.052 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:20.053 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:20.109 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:20.109 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:20.253 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:20.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:20.738 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:21.062 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:21.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-03-26T03:42:21.860 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:22.013 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 3.1 M 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:42:22.014 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:22.015 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2024-03-26T03:42:22.015 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 13 M 2024-03-26T03:42:22.015 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:23.097 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-26T03:42:23.098 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:23.098 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.8 MB/s | 3.1 MB 00:01 2024-03-26T03:42:23.098 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:23.108 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:23.108 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:23.181 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:23.181 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:23.602 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:23.681 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:23.706 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:23.706 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T03:42:23.706 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T03:42:23.706 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T03:42:23.706 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T03:42:23.707 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:24.837 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:24.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:24.837 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:24.838 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:42:24.838 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:24.838 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:25.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-03-26T03:42:25.647 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:50 ago on Tue 26 Mar 2024 03:40:35 AM UTC. 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-2475.ge42b8ea3.el9 ceph 171 k 2024-03-26T03:42:25.796 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 171 k 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-03-26T03:42:25.797 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-26T03:42:26.492 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.0.0-2475.ge42b8ea3.el9.x86_64.rpm 246 kB/s | 171 kB 00:00 2024-03-26T03:42:26.492 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:42:26.492 INFO:teuthology.orchestra.run.smithi086.stdout:Total 246 kB/s | 171 kB 00:00 2024-03-26T03:42:26.493 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-26T03:42:26.500 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-26T03:42:26.501 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-26T03:42:26.559 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-26T03:42:26.560 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-26T03:42:26.711 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-26T03:42:26.808 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:27.235 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 1/1 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-2475.ge42b8ea3.el9.x86_64 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:42:27.520 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-26T03:42:27.765 DEBUG:teuthology.parallel:result is None 2024-03-26T03:42:27.766 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:42:28.005 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T03:42:28.030 INFO:teuthology.orchestra.run.smithi006.stdout:19.0.0-2475.ge42b8ea3.el9 2024-03-26T03:42:28.031 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2475.ge42b8ea3.el9 2024-03-26T03:42:28.031 INFO:teuthology.task.install:The correct ceph version 19.0.0-2475.ge42b8ea3 is installed. 2024-03-26T03:42:28.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:42:28.260 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T03:42:28.280 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-2475.ge42b8ea3.el9 2024-03-26T03:42:28.280 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2475.ge42b8ea3.el9 2024-03-26T03:42:28.281 INFO:teuthology.task.install:The correct ceph version 19.0.0-2475.ge42b8ea3 is installed. 2024-03-26T03:42:28.282 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T03:42:28.282 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:42:28.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T03:42:28.315 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:42:28.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T03:42:28.347 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T03:42:28.348 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:42:28.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T03:42:28.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T03:42:28.458 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:42:28.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T03:42:28.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T03:42:28.551 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T03:42:28.551 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:42:28.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T03:42:28.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T03:42:28.652 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:42:28.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T03:42:28.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T03:42:28.812 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T03:42:28.812 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:42:28.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T03:42:28.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T03:42:28.905 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:42:28.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T03:42:28.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T03:42:28.999 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T03:42:29.097 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'cephadm_mode': 'root'} 2024-03-26T03:42:29.098 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:42:29.098 INFO:tasks.cephadm:Cluster fsid is de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:42:29.098 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T03:42:29.098 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.6', 'mon.c': '[v2:172.21.15.6:3301,v1:172.21.15.6:6790]', 'mon.b': '172.21.15.86'} 2024-03-26T03:42:29.099 INFO:tasks.cephadm:First mon is mon.a on smithi006 2024-03-26T03:42:29.099 INFO:tasks.cephadm:First mgr is y 2024-03-26T03:42:29.099 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T03:42:29.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo hostname $(hostname -s) 2024-03-26T03:42:29.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-03-26T03:42:29.153 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-26T03:42:29.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:42:29.375 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77960/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2475-ge42b8ea3', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2475.ge42b8ea3'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-26 00:31:23.988041', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-adk-testing-2024-03-25-1628', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-26T03:42:29.526 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-adk-testing-2024-03-25-1628, sha1 e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:42:29.527 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm 2024-03-26T03:42:29.527 DEBUG:teuthology.orchestra.run.smithi006:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:30.588 INFO:teuthology.orchestra.run.smithi006.stdout:-rw-r--r--. 1 ubuntu ubuntu 781509 Mar 26 03:42 /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:30.589 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-adk-testing-2024-03-25-1628/e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:32.107 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 781509 Mar 26 03:42 /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:32.108 DEBUG:teuthology.orchestra.run.smithi006:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:32.126 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-26T03:42:32.155 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 on all hosts... 2024-03-26T03:42:32.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 pull 2024-03-26T03:42:32.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 pull 2024-03-26T03:42:32.968 INFO:teuthology.orchestra.run.smithi006.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T03:42:33.025 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)", 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354", 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717" 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:43:04.123 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:43:05.562 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:43:05.562 INFO:teuthology.orchestra.run.smithi006.stdout: "ceph_version": "ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev)", 2024-03-26T03:43:05.562 INFO:teuthology.orchestra.run.smithi006.stdout: "image_id": "f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354", 2024-03-26T03:43:05.563 INFO:teuthology.orchestra.run.smithi006.stdout: "repo_digests": [ 2024-03-26T03:43:05.563 INFO:teuthology.orchestra.run.smithi006.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717" 2024-03-26T03:43:05.563 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:43:05.563 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:43:05.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph 2024-03-26T03:43:05.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-03-26T03:43:05.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /etc/ceph 2024-03-26T03:43:05.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-03-26T03:43:05.731 INFO:tasks.cephadm:Writing seed config... 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-26T03:43:05.732 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T03:43:05.733 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-26T03:43:05.734 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:43:05.734 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T03:43:05.748 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = de980954-eb22-11ee-95d0-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-26T03:43:05.749 DEBUG:teuthology.orchestra.run.smithi006:mon.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T03:43:05.791 DEBUG:teuthology.orchestra.run.smithi006:mgr.y> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T03:43:05.833 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T03:43:05.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 -v bootstrap --fsid de980954-eb22-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.6 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:43:06.030 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:43:06.030 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003', '-v', 'bootstrap', '--fsid', 'de980954-eb22-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.6', '--skip-admin-label'] 2024-03-26T03:43:06.031 INFO:teuthology.orchestra.run.smithi006.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T03:43:06.031 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying podman|docker is present... 2024-03-26T03:43:06.058 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T03:43:06.058 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying lvm2 is present... 2024-03-26T03:43:06.058 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying time synchronization is in place... 2024-03-26T03:43:06.065 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T03:43:06.065 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T03:43:06.071 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T03:43:06.071 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:43:06.077 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-03-26T03:43:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-03-26T03:43:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-03-26T03:43:06.083 INFO:teuthology.orchestra.run.smithi006.stdout:Repeating the final host check... 2024-03-26T03:43:06.105 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T03:43:06.106 INFO:teuthology.orchestra.run.smithi006.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T03:43:06.106 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl is present 2024-03-26T03:43:06.106 INFO:teuthology.orchestra.run.smithi006.stdout:lvcreate is present 2024-03-26T03:43:06.112 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T03:43:06.112 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T03:43:06.117 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T03:43:06.117 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:43:06.123 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout enabled 2024-03-26T03:43:06.129 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout active 2024-03-26T03:43:06.129 INFO:teuthology.orchestra.run.smithi006.stdout:Unit chronyd.service is enabled and running 2024-03-26T03:43:06.129 INFO:teuthology.orchestra.run.smithi006.stdout:Host looks OK 2024-03-26T03:43:06.129 INFO:teuthology.orchestra.run.smithi006.stdout:Cluster fsid: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Acquiring lock 139880099680944 on /run/cephadm/de980954-eb22-11ee-95d0-87774f69a715.lock 2024-03-26T03:43:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Lock 139880099680944 acquired on /run/cephadm/de980954-eb22-11ee-95d0-87774f69a715.lock 2024-03-26T03:43:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 3300 ... 2024-03-26T03:43:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying IP 172.21.15.6 port 6789 ... 2024-03-26T03:43:06.130 INFO:teuthology.orchestra.run.smithi006.stdout:Base mon IP(s) is [172.21.15.6:3300, 172.21.15.6:6789], mon addrv is [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-03-26T03:43:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.6 metric 100 2024-03-26T03:43:06.133 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.6 metric 100 2024-03-26T03:43:06.135 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T03:43:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T03:43:06.136 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15e4/64 scope link noprefixroute 2024-03-26T03:43:06.138 INFO:teuthology.orchestra.run.smithi006.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T03:43:06.139 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-03-26T03:43:06.139 INFO:teuthology.orchestra.run.smithi006.stdout:Mon IP `172.21.15.6` is in CIDR network `172.21.0.0/20` 2024-03-26T03:43:06.139 INFO:teuthology.orchestra.run.smithi006.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-26T03:43:06.139 INFO:teuthology.orchestra.run.smithi006.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T03:43:06.140 INFO:teuthology.orchestra.run.smithi006.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stdout f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003... 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying blob sha256:225af1ab3ea34577a62e5c6f6ac1e3381fc0d9eac77fbb4e31c233b967549b54 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Copying config sha256:f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 2024-03-26T03:43:06.634 INFO:teuthology.orchestra.run.smithi006.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T03:43:07.536 INFO:teuthology.orchestra.run.smithi006.stdout:ceph: stdout ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev) 2024-03-26T03:43:07.536 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph version: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev) 2024-03-26T03:43:07.536 INFO:teuthology.orchestra.run.smithi006.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T03:43:08.367 INFO:teuthology.orchestra.run.smithi006.stdout:stat: stdout 167 167 2024-03-26T03:43:08.368 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial keys... 2024-03-26T03:43:09.365 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBMRAJmQ5VCLxAA2J8KrSPk19gY2UaRWOtLrA== 2024-03-26T03:43:10.270 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBNRAJmLkNFKhAAkBeXohgkGXrM6dLB0wxK1Q== 2024-03-26T03:43:11.302 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph-authtool: stdout AQBORAJmBAqtJRAAhErI9w84XyF4LiYEYv9zOw== 2024-03-26T03:43:11.302 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial monmap... 2024-03-26T03:43:12.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T03:43:12.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T03:43:12.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:12.175 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool for a [v2:172.21.15.6:3300,v1:172.21.15.6:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = quincy 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: set fsid to de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:43:12.176 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mon... 2024-03-26T03:43:13.114 INFO:teuthology.orchestra.run.smithi006.stdout:create mon.a on 2024-03-26T03:43:13.329 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T03:43:13.481 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T03:43:13.662 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-de980954-eb22-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715.target. 2024-03-26T03:43:13.662 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-de980954-eb22-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715.target. 2024-03-26T03:43:13.915 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a 2024-03-26T03:43:13.916 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T03:43:14.075 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715.target.wants/ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715@.service. 2024-03-26T03:43:14.555 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T03:43:14.555 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-26T03:43:14.561 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T03:43:14.561 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:43:14.561 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-26T03:43:14.561 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T03:43:14.562 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon to start... 2024-03-26T03:43:14.562 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mon... 2024-03-26T03:43:15.563 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout id: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout services: 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.549151s) 2024-03-26T03:43:15.564 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout data: 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:mon is available 2024-03-26T03:43:15.565 INFO:teuthology.orchestra.run.smithi006.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.6:3300,v1:172.21.15.6:6789] 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T03:43:16.610 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T03:43:16.611 INFO:teuthology.orchestra.run.smithi006.stdout:Generating new minimal ceph.conf... 2024-03-26T03:43:17.598 INFO:teuthology.orchestra.run.smithi006.stdout:Restarting the monitor... 2024-03-26T03:43:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29067]: 2024-03-26T03:43:17.856+0000 7f84cf031700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T03:43:18.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 podman[29317]: 2024-03-26 03:43:18.021407047 +0000 UTC m=+0.255716757 container died 582ca6eb4768bfc89a516f6eaf642c0ef3e1f7310708728d587c6596081eff93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T03:43:18.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 podman[29317]: 2024-03-26 03:43:18.306925815 +0000 UTC m=+0.541235525 container cleanup 582ca6eb4768bfc89a516f6eaf642c0ef3e1f7310708728d587c6596081eff93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T03:43:18.635 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 bash[29317]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a 2024-03-26T03:43:18.635 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 podman[29328]: 2024-03-26 03:43:18.524723779 +0000 UTC m=+0.506166837 container remove 582ca6eb4768bfc89a516f6eaf642c0ef3e1f7310708728d587c6596081eff93 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T03:43:18.939 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T03:43:18.940 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 systemd[1]: Stopped Ceph mon.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:43:18.940 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 systemd[1]: Starting Ceph mon.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:43:18.940 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 podman[29427]: 2024-03-26 03:43:18.842415753 +0000 UTC m=+0.098214195 container create 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:43:18.940 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:18 smithi006 podman[29427]: 2024-03-26 03:43:18.761907337 +0000 UTC m=+0.017705777 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:43:19.099 INFO:teuthology.orchestra.run.smithi006.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T03:43:19.221 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 podman[29427]: 2024-03-26 03:43:19.027962239 +0000 UTC m=+0.283760679 container init 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 podman[29427]: 2024-03-26 03:43:19.030742644 +0000 UTC m=+0.286541085 container start 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: pidfile_write: ignore empty --pid-file 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: load: jerasure load: lrc 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: RocksDB version: 7.9.2 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Git sha 0 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: DB SUMMARY 2024-03-26T03:43:19.222 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: DB Session ID: F1CLI8J5WVVFIV8WRMIX 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: CURRENT file: CURRENT 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 96141 ; 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.error_if_exists: 0 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.create_if_missing: 0 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.paranoid_checks: 1 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.env: 0x55eade41bc60 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.info_log: 0x55eadf935ae0 2024-03-26T03:43:19.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T03:43:19.224 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.statistics: (nil) 2024-03-26T03:43:19.224 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.use_fsync: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_log_file_size: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_fallocate: 1 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.use_direct_reads: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T03:43:19.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.db_log_dir: 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.wal_dir: 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.write_buffer_manager: 0x55eadf9061e0 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T03:43:19.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.unordered_write: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.row_cache: None 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.wal_filter: None 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.two_write_queues: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.wal_compression: 0 2024-03-26T03:43:19.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.atomic_flush: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.log_readahead_size: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_background_jobs: 2 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_background_compactions: -1 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_subcompactions: 1 2024-03-26T03:43:19.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_open_files: -1 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_background_flushes: -1 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Compression algorithms supported: 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T03:43:19.229 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kZSTD supported: 0 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kXpressCompression supported: 0 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kZlibCompression supported: 1 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kSnappyCompression supported: 1 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kLZ4Compression supported: 1 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: kBZip2Compression supported: 0 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.merge_operator: 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_filter: None 2024-03-26T03:43:19.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55eadf935cc0) 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_type: 0 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_index_type: 0 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_shortening: 1 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: checksum: 4 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: no_block_cache: 0 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache: 0x55eadf939090 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-03-26T03:43:19.231 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_options: 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: capacity : 536870912 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_shard_bits : 4 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: strict_capacity_limit : 0 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_cache_compressed: (nil) 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: persistent_cache: (nil) 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size: 4096 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_size_deviation: 10 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_restart_interval: 16 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: index_block_restart_interval: 1 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: metadata_block_size: 4096 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: partition_filters: 0 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: use_delta_encoding: 1 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: filter_policy: bloomfilter 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: whole_key_filtering: 1 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: verify_compression: 0 2024-03-26T03:43:19.232 INFO:journalctl@ceph.mon.a.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: format_version: 5 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_index_compression: 1 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: block_align: 0 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: max_auto_readahead_size: 262144 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: prepopulate_block_cache: 0 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: initial_auto_readahead_size: 8192 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression: NoCompression 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.num_levels: 7 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T03:43:19.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T03:43:19.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T03:43:19.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T03:43:19.236 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.inplace_update_support: 0 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T03:43:19.237 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.bloom_locality: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.max_successive_merges: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.ttl: 2592000 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enable_blob_files: false 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.min_blob_size: 0 2024-03-26T03:43:19.238 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 85ba09e6-8949-4015-963e-b5ca18427c8d 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424599058926, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424599059862, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92006, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 288, "table_properties": {"data_size": 90054, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11976, "raw_average_key_size": 48, "raw_value_size": 83487, "raw_average_value_size": 338, "num_data_blocks": 12, "num_entries": 247, "num_filter_entries": 247, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711424599, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "85ba09e6-8949-4015-963e-b5ca18427c8d", "db_session_id": "F1CLI8J5WVVFIV8WRMIX", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424599059951, "job": 1, "event": "recovery_finished"} 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T03:43:19.239 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55eadfa28000 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: DB pointer 0x55eadfa12000 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** DB Stats ** 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T03:43:19.240 INFO:journalctl@ceph.mon.a.smithi006.stdout: L0 2/0 91.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Sum 2/0 91.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T03:43:19.241 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: Cumulative compaction: 0.00 GB write, 30.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: Interval compaction: 0.00 GB write, 30.37 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache BinnedLRUCache@0x55eadf939090#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: starting mon.a rank 0 at public addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] at bind addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???) e1 preinit fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).mds e1 new map 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).mds e1 print_map 2024-03-26T03:43:19.242 INFO:journalctl@ceph.mon.a.smithi006.stdout: e1 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout: legacy client fscid: -1 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout: 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout: No filesystems configured 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 bash[29427]: 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 systemd[1]: Started Ceph mon.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:43:19.243 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: monmap epoch 1 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: last_changed 2024-03-26T03:43:11.617705+0000 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: min_mon_release 19 (squid) 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: election_strategy: 1 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: fsmap 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T03:43:19.244 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:19 smithi006 ceph-mon[29446]: mgrmap e1: no daemons active 2024-03-26T03:43:20.305 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T03:43:20.327 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:43:20.327 INFO:teuthology.orchestra.run.smithi006.stdout:Creating mgr... 2024-03-26T03:43:20.327 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T03:43:20.327 INFO:teuthology.orchestra.run.smithi006.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T03:43:20.576 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y 2024-03-26T03:43:20.577 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Failed to reset failed state of unit ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T03:43:20.759 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715.target.wants/ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-de980954-eb22-11ee-95d0-87774f69a715@.service. 2024-03-26T03:43:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1725176649' entity='client.admin' 2024-03-26T03:43:20.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:20 smithi006 systemd[1]: Starting Ceph mgr.y for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:43:21.240 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:20 smithi006 podman[29663]: 2024-03-26 03:43:20.979177663 +0000 UTC m=+0.097047784 container create 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T03:43:21.240 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:20 smithi006 podman[29663]: 2024-03-26 03:43:20.899579533 +0000 UTC m=+0.017449626 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 2024-03-26T03:43:21.240 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 podman[29663]: 2024-03-26 03:43:21.164981964 +0000 UTC m=+0.282852044 container init 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=) 2024-03-26T03:43:21.240 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 podman[29663]: 2024-03-26 03:43:21.167355562 +0000 UTC m=+0.285225650 container start 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T03:43:21.240 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 bash[29663]: 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 2024-03-26T03:43:21.253 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T03:43:21.253 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-26T03:43:21.261 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T03:43:21.261 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:43:21.261 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-26T03:43:21.261 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T03:43:21.269 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T03:43:21.269 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr to start... 2024-03-26T03:43:21.276 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr... 2024-03-26T03:43:21.543 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 systemd[1]: Started Ceph mgr.y for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:43:21.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3527907063' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T03:43:21.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:21.736+0000 7f69df54c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:43:21.836 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:21 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:21.834+0000 7f69df54c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:43:22.271 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T03:43:22.272 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T03:43:22.273 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T03:43:22.274 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T03:43:14.534881+0000", 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:22.275 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (1/15)... 2024-03-26T03:43:22.596 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:22.501+0000 7f69df54c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:43:22.596 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:22.595+0000 7f69df54c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:43:22.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:22.737+0000 7f69df54c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:43:22.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:22.835+0000 7f69df54c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:43:23.188 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:22.920+0000 7f69df54c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:43:23.188 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:23.187+0000 7f69df54c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:43:23.483 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:23.299+0000 7f69df54c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:43:23.484 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:23.386+0000 7f69df54c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:43:23.484 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:23.482+0000 7f69df54c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:43:23.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:23.659+0000 7f69df54c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:43:24.749 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:24.680+0000 7f69df54c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:43:25.094 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1794446432' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T03:43:25.095 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:24.822+0000 7f69df54c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:43:25.352 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:25 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:25.160+0000 7f69df54c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:43:25.353 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:25 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:25.251+0000 7f69df54c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T03:43:25.357 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T03:43:25.358 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:25.359 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T03:43:25.359 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T03:43:25.359 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T03:43:25.362 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T03:43:25.363 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T03:43:14.534881+0000", 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:25.364 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (2/15)... 2024-03-26T03:43:25.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:25 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:25.454+0000 7f69df54c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:43:25.936 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:25 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:25.643+0000 7f69df54c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:43:26.196 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:25 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:25.935+0000 7f69df54c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:43:26.196 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:26 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:26.194+0000 7f69df54c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:43:26.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:26 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:26.438+0000 7f69df54c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:43:27.574 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:27 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:27.263+0000 7f69df54c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:43:28.133 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:27 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3155120077' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T03:43:28.134 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:27 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:27.853+0000 7f69df54c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:43:28.134 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:27 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:27.955+0000 7f69df54c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:43:28.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:28.178+0000 7f69df54c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T03:43:28.414 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T03:43:28.415 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T03:43:28.416 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T03:43:28.419 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T03:43:28.420 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T03:43:14.534881+0000", 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:28.421 INFO:teuthology.orchestra.run.smithi006.stdout:mgr not available, waiting (3/15)... 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: Activating manager daemon y 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: mgrmap e2: y(active, starting, since 0.00323756s) 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:43:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: Manager daemon y is now available 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' 2024-03-26T03:43:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:28 smithi006 ceph-mon[29446]: from='mgr.14100 172.21.15.6:0/1557158655' entity='mgr.y' 2024-03-26T03:43:30.541 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:30 smithi006 ceph-mon[29446]: mgrmap e3: y(active, since 1.00708s) 2024-03-26T03:43:32.459 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:32 smithi006 ceph-mon[29446]: mgrmap e4: y(active, since 3s) 2024-03-26T03:43:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3795311677' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T03:43:39.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:39.905 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 0 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T03:43:39.906 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T03:43:39.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T03:43:39.911 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T03:43:39.912 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ], 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T03:43:14.534881+0000", 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout }, 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:39.913 INFO:teuthology.orchestra.run.smithi006.stdout:mgr is available 2024-03-26T03:43:41.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:41 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2665194309' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T03:43:41.558 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout fsid = de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T03:43:41.559 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T03:43:41.560 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling cephadm module... 2024-03-26T03:43:42.585 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:42 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3301051257' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T03:43:42.586 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:42 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setuser ceph since I am not root 2024-03-26T03:43:42.586 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:42 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setgroup ceph since I am not root 2024-03-26T03:43:43.273 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:42 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:42.849+0000 7f81c6c78200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:43:43.273 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:42 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:42.944+0000 7f81c6c78200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:43:43.540 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3301051257' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T03:43:43.540 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-mon[29446]: mgrmap e5: y(active, since 14s) 2024-03-26T03:43:43.854 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:43.609+0000 7f81c6c78200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:43:43.854 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:43.706+0000 7f81c6c78200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:43:43.854 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:43.853+0000 7f81c6c78200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:43:44.114 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:43 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:43.951+0000 7f81c6c78200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:43:44.114 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.037+0000 7f81c6c78200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:43:44.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2863022402' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T03:43:44.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.310+0000 7f81c6c78200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:43:44.694 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.423+0000 7f81c6c78200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:43:44.694 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.512+0000 7f81c6c78200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:43:44.694 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.609+0000 7f81c6c78200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:43:44.698 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:44.698 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T03:43:44.698 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T03:43:44.699 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T03:43:44.699 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T03:43:44.699 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:44.699 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-03-26T03:43:44.699 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 5... 2024-03-26T03:43:45.005 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:44.789+0000 7f81c6c78200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:43:46.140 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:45 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:45.780+0000 7f81c6c78200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:43:46.140 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:45 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:45.918+0000 7f81c6c78200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:43:46.554 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:46.255+0000 7f81c6c78200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:43:46.554 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:46.346+0000 7f81c6c78200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:43:46.554 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:46.553+0000 7f81c6c78200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:43:46.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:46.741+0000 7f81c6c78200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:43:47.297 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:47 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:47.035+0000 7f81c6c78200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:43:47.297 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:47 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:47.295+0000 7f81c6c78200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:43:47.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:47 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:47.542+0000 7f81c6c78200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:43:48.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:48 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:48.370+0000 7f81c6c78200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:43:49.276 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:48 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:48.953+0000 7f81c6c78200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:43:49.276 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:49.053+0000 7f81c6c78200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:43:49.276 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:43:49.275+0000 7f81c6c78200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: Active manager daemon y restarted 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: Activating manager daemon y 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: mgrmap e6: y(active, starting, since 0.00322919s) 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:43:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: Manager daemon y is now available 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: Found migration_current of "None". Setting to last migration. 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:43:49.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:49 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:43:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:50 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:43:50.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:50 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:43:50.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:50 smithi006 ceph-mon[29446]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T03:43:50.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:50 smithi006 ceph-mon[29446]: mgrmap e7: y(active, since 1.00529s) 2024-03-26T03:43:50.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:50 smithi006 ceph-mon[29446]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 5 is available 2024-03-26T03:43:50.904 INFO:teuthology.orchestra.run.smithi006.stdout:Setting orchestrator backend to cephadm... 2024-03-26T03:43:52.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:52 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:52.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:52 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:52.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:52 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:52.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:52 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:43:53.208 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T03:43:53.208 INFO:teuthology.orchestra.run.smithi006.stdout:Generating ssh key... 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: [26/Mar/2024:03:43:51] ENGINE Bus STARTING 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: [26/Mar/2024:03:43:51] ENGINE Serving on https://172.21.15.6:7150 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: [26/Mar/2024:03:43:51] ENGINE Serving on http://172.21.15.6:8765 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: [26/Mar/2024:03:43:51] ENGINE Bus STARTED 2024-03-26T03:43:53.323 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:53 smithi006 ceph-mon[29446]: mgrmap e8: y(active, since 2s) 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-mon[29446]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: Generating public/private rsa key pair. 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: Your identification has been saved in /tmp/tmpffpke_s3/key. 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: Your public key has been saved in /tmp/tmpffpke_s3/key.pub. 2024-03-26T03:43:54.320 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: The key fingerprint is: 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: SHA256:qjXqNxZREXN7uV0GXdNflvGdJpNGBq6g2x+WCKMJqf8 ceph-de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: The key's randomart image is: 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: +---[RSA 3072]----+ 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | +o...o.+*| 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | .o..o..oX| 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | o ..o= +B| 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | . o . ...o+o.| 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | o + .S. . . | 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: |. . o *.. . | 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: |. o .++ + | 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | . ++.o . | 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: | .oEo . . | 2024-03-26T03:43:54.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:43:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: +----[SHA256]-----+ 2024-03-26T03:43:55.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:55 smithi006 ceph-mon[29446]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:43:55.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:55 smithi006 ceph-mon[29446]: Generating ssh key... 2024-03-26T03:43:55.763 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNbWhdN231U+cMsIY6s00DxySYz7fQsLham5lhGf69PM98B1wnHlBE4/+PosxIrAmlTHu7RfXC31s781cu20/5FzhHN1rLi7VxeUv1xIXmkYynFxOa1EuRtdP+i1sfe5SO4VsJUsl+dcHqg9dShhGhVseROlNbLr4rUbROmSSDwhmLSZpeYdgGnWWmF+ge33aSipjgXXU80PGHECWepGUHQDTJXMayNiSOESx/Ke19/IU2KxJwxEbCkFBYYxA2z7tNBJQsBf/YZkZuF/LWSPC7XMElleKhhOrv9PNq4pLeNOuhblKTSn25eQCmqN8+GA+oxDmi0tcA1cMhrNFmRCL+LZYd2llN28+ufTLZlKUqcgWXgn0BEdVGKUR9Bavho2Y7/P1GDRDI+MEoxol0UyD4hf1YWfHXlXldy7hXJ88Xiz3tbeaBYJyDV60OuWtK65aRyrJX1+gjiH+qcyScZGb5GFaDnjDuU+HoarzNAYwQscLU+o6z2jNiVZ44yhy9aC8= ceph-de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:43:55.764 INFO:teuthology.orchestra.run.smithi006.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T03:43:55.764 INFO:teuthology.orchestra.run.smithi006.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T03:43:55.764 INFO:teuthology.orchestra.run.smithi006.stdout:Adding host smithi006... 2024-03-26T03:43:56.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:56 smithi006 ceph-mon[29446]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:43:57.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:57 smithi006 ceph-mon[29446]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi006", "addr": "172.21.15.6", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:43:57.571 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:43:57 smithi006 ceph-mon[29446]: Deploying cephadm binary to smithi006 2024-03-26T03:44:02.151 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Added host 'smithi006' with addr '172.21.15.6' 2024-03-26T03:44:02.151 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mon service... 2024-03-26T03:44:02.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:02 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:02.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:02 smithi006 ceph-mon[29446]: Added host smithi006 2024-03-26T03:44:02.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:02 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:03.438 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T03:44:03.439 INFO:teuthology.orchestra.run.smithi006.stdout:Deploying unmanaged mgr service... 2024-03-26T03:44:04.024 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:03 smithi006 ceph-mon[29446]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:04.024 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:03 smithi006 ceph-mon[29446]: Saving service mon spec with placement count:5 2024-03-26T03:44:04.024 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:03 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:04.907 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T03:44:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:05 smithi006 ceph-mon[29446]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:05 smithi006 ceph-mon[29446]: Saving service mgr spec with placement count:2 2024-03-26T03:44:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:05 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:06.552 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/61522169' entity='client.admin' 2024-03-26T03:44:06.552 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:06 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:06.552 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:06 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:07.700 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling the dashboard module... 2024-03-26T03:44:08.116 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:08 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1597591004' entity='client.admin' 2024-03-26T03:44:09.339 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3521005254' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T03:44:09.339 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-mon[29446]: from='mgr.14120 172.21.15.6:0/421904826' entity='mgr.y' 2024-03-26T03:44:09.339 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setuser ceph since I am not root 2024-03-26T03:44:09.339 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setgroup ceph since I am not root 2024-03-26T03:44:09.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:09.623+0000 7f7977ce9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:44:09.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:09 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:09.719+0000 7f7977ce9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:44:10.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3521005254' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T03:44:10.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-mon[29446]: mgrmap e9: y(active, since 19s) 2024-03-26T03:44:10.492 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:10.395+0000 7f7977ce9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:44:10.493 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:10.491+0000 7f7977ce9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:44:10.771 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:10.636+0000 7f7977ce9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:44:10.771 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:10.735+0000 7f7977ce9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for the mgr to restart... 2024-03-26T03:44:10.775 INFO:teuthology.orchestra.run.smithi006.stdout:Waiting for mgr epoch 9... 2024-03-26T03:44:11.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3065112858' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T03:44:11.099 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:10.826+0000 7f7977ce9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:44:11.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:11.102+0000 7f7977ce9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:44:11.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:11.219+0000 7f7977ce9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:44:11.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:11.308+0000 7f7977ce9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:44:11.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:11.404+0000 7f7977ce9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:44:11.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:11.583+0000 7f7977ce9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:44:12.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:12.572+0000 7f7977ce9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:44:12.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:12.707+0000 7f7977ce9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:44:13.341 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:13.046+0000 7f7977ce9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:44:13.341 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:13.137+0000 7f7977ce9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:44:13.342 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:13.340+0000 7f7977ce9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:44:13.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:13.530+0000 7f7977ce9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:44:14.087 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:13.824+0000 7f7977ce9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:44:14.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:14.086+0000 7f7977ce9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:44:14.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:14.331+0000 7f7977ce9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:44:15.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:15.165+0000 7f7977ce9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:44:16.083 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:15.755+0000 7f7977ce9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:44:16.083 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:15.856+0000 7f7977ce9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:44:16.083 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:44:16.082+0000 7f7977ce9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:44:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: Active manager daemon y restarted 2024-03-26T03:44:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: Activating manager daemon y 2024-03-26T03:44:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T03:44:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: mgrmap e10: y(active, starting, since 0.00310558s) 2024-03-26T03:44:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: Manager daemon y is now available 2024-03-26T03:44:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:17.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:44:17.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:44:17.306 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: [26/Mar/2024:03:44:16] ENGINE Bus STARTING 2024-03-26T03:44:17.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: [26/Mar/2024:03:44:16] ENGINE Serving on https://172.21.15.6:7150 2024-03-26T03:44:17.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: [26/Mar/2024:03:44:16] ENGINE Serving on http://172.21.15.6:8765 2024-03-26T03:44:17.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: [26/Mar/2024:03:44:16] ENGINE Bus STARTED 2024-03-26T03:44:17.307 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:17 smithi006 ceph-mon[29446]: mgrmap e11: y(active, since 1.00543s) 2024-03-26T03:44:17.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout { 2024-03-26T03:44:17.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T03:44:17.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T03:44:17.884 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout } 2024-03-26T03:44:17.885 INFO:teuthology.orchestra.run.smithi006.stdout:mgr epoch 9 is available 2024-03-26T03:44:17.885 INFO:teuthology.orchestra.run.smithi006.stdout:Generating a dashboard self-signed certificate... 2024-03-26T03:44:19.207 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T03:44:19.207 INFO:teuthology.orchestra.run.smithi006.stdout:Creating initial admin user... 2024-03-26T03:44:19.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:18 smithi006 ceph-mon[29446]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T03:44:19.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:18 smithi006 ceph-mon[29446]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T03:44:19.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:19.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:19.329 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:20.185 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:19 smithi006 ceph-mon[29446]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:20.185 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:19 smithi006 ceph-mon[29446]: mgrmap e12: y(active, since 2s) 2024-03-26T03:44:20.185 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:20.876 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$5ZNYdC8NjkV0JWwjP0U.6uRka8zmNX5AHYYZQEXgic/oVh04katMO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711424660, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T03:44:20.876 INFO:teuthology.orchestra.run.smithi006.stdout:Fetching dashboard port number... 2024-03-26T03:44:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:20 smithi006 ceph-mon[29446]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:20 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:22.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1706954663' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T03:44:22.484 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T03:44:22.495 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T03:44:22.495 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout disabled 2024-03-26T03:44:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T03:44:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:systemctl: stdout inactive 2024-03-26T03:44:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:firewalld.service is not enabled 2024-03-26T03:44:22.502 INFO:teuthology.orchestra.run.smithi006.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout:Ceph Dashboard is now available at: 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: URL: https://smithi006.front.sepia.ceph.com:8443/ 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: User: admin 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: Password: dr4qw120ed 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:22.504 INFO:teuthology.orchestra.run.smithi006.stdout:Saving cluster configuration to /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config directory 2024-03-26T03:44:22.505 INFO:teuthology.orchestra.run.smithi006.stdout:Enabling autotune for osd_memory_target 2024-03-26T03:44:24.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:23 smithi006 ceph-mon[29446]: mgrmap e13: y(active, since 6s) 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: ceph telemetry on 2024-03-26T03:44:25.000 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.001 INFO:teuthology.orchestra.run.smithi006.stdout:For more information see: 2024-03-26T03:44:25.001 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.001 INFO:teuthology.orchestra.run.smithi006.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T03:44:25.001 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:25.001 INFO:teuthology.orchestra.run.smithi006.stdout:Bootstrap complete. 2024-03-26T03:44:25.043 INFO:tasks.cephadm:Fetching config... 2024-03-26T03:44:25.044 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:44:25.044 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T03:44:25.059 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T03:44:25.059 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:44:25.059 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T03:44:25.114 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T03:44:25.115 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:44:25.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T03:44:25.181 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T03:44:25.181 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:44:25.181 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T03:44:25.237 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T03:44:25.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNbWhdN231U+cMsIY6s00DxySYz7fQsLham5lhGf69PM98B1wnHlBE4/+PosxIrAmlTHu7RfXC31s781cu20/5FzhHN1rLi7VxeUv1xIXmkYynFxOa1EuRtdP+i1sfe5SO4VsJUsl+dcHqg9dShhGhVseROlNbLr4rUbROmSSDwhmLSZpeYdgGnWWmF+ge33aSipjgXXU80PGHECWepGUHQDTJXMayNiSOESx/Ke19/IU2KxJwxEbCkFBYYxA2z7tNBJQsBf/YZkZuF/LWSPC7XMElleKhhOrv9PNq4pLeNOuhblKTSn25eQCmqN8+GA+oxDmi0tcA1cMhrNFmRCL+LZYd2llN28+ufTLZlKUqcgWXgn0BEdVGKUR9Bavho2Y7/P1GDRDI+MEoxol0UyD4hf1YWfHXlXldy7hXJ88Xiz3tbeaBYJyDV60OuWtK65aRyrJX1+gjiH+qcyScZGb5GFaDnjDuU+HoarzNAYwQscLU+o6z2jNiVZ44yhy9aC8= ceph-de980954-eb22-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T03:44:25.328 INFO:teuthology.orchestra.run.smithi006.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNbWhdN231U+cMsIY6s00DxySYz7fQsLham5lhGf69PM98B1wnHlBE4/+PosxIrAmlTHu7RfXC31s781cu20/5FzhHN1rLi7VxeUv1xIXmkYynFxOa1EuRtdP+i1sfe5SO4VsJUsl+dcHqg9dShhGhVseROlNbLr4rUbROmSSDwhmLSZpeYdgGnWWmF+ge33aSipjgXXU80PGHECWepGUHQDTJXMayNiSOESx/Ke19/IU2KxJwxEbCkFBYYxA2z7tNBJQsBf/YZkZuF/LWSPC7XMElleKhhOrv9PNq4pLeNOuhblKTSn25eQCmqN8+GA+oxDmi0tcA1cMhrNFmRCL+LZYd2llN28+ufTLZlKUqcgWXgn0BEdVGKUR9Bavho2Y7/P1GDRDI+MEoxol0UyD4hf1YWfHXlXldy7hXJ88Xiz3tbeaBYJyDV60OuWtK65aRyrJX1+gjiH+qcyScZGb5GFaDnjDuU+HoarzNAYwQscLU+o6z2jNiVZ44yhy9aC8= ceph-de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:25.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNbWhdN231U+cMsIY6s00DxySYz7fQsLham5lhGf69PM98B1wnHlBE4/+PosxIrAmlTHu7RfXC31s781cu20/5FzhHN1rLi7VxeUv1xIXmkYynFxOa1EuRtdP+i1sfe5SO4VsJUsl+dcHqg9dShhGhVseROlNbLr4rUbROmSSDwhmLSZpeYdgGnWWmF+ge33aSipjgXXU80PGHECWepGUHQDTJXMayNiSOESx/Ke19/IU2KxJwxEbCkFBYYxA2z7tNBJQsBf/YZkZuF/LWSPC7XMElleKhhOrv9PNq4pLeNOuhblKTSn25eQCmqN8+GA+oxDmi0tcA1cMhrNFmRCL+LZYd2llN28+ufTLZlKUqcgWXgn0BEdVGKUR9Bavho2Y7/P1GDRDI+MEoxol0UyD4hf1YWfHXlXldy7hXJ88Xiz3tbeaBYJyDV60OuWtK65aRyrJX1+gjiH+qcyScZGb5GFaDnjDuU+HoarzNAYwQscLU+o6z2jNiVZ44yhy9aC8= ceph-de980954-eb22-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T03:44:25.388 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDNbWhdN231U+cMsIY6s00DxySYz7fQsLham5lhGf69PM98B1wnHlBE4/+PosxIrAmlTHu7RfXC31s781cu20/5FzhHN1rLi7VxeUv1xIXmkYynFxOa1EuRtdP+i1sfe5SO4VsJUsl+dcHqg9dShhGhVseROlNbLr4rUbROmSSDwhmLSZpeYdgGnWWmF+ge33aSipjgXXU80PGHECWepGUHQDTJXMayNiSOESx/Ke19/IU2KxJwxEbCkFBYYxA2z7tNBJQsBf/YZkZuF/LWSPC7XMElleKhhOrv9PNq4pLeNOuhblKTSn25eQCmqN8+GA+oxDmi0tcA1cMhrNFmRCL+LZYd2llN28+ufTLZlKUqcgWXgn0BEdVGKUR9Bavho2Y7/P1GDRDI+MEoxol0UyD4hf1YWfHXlXldy7hXJ88Xiz3tbeaBYJyDV60OuWtK65aRyrJX1+gjiH+qcyScZGb5GFaDnjDuU+HoarzNAYwQscLU+o6z2jNiVZ44yhy9aC8= ceph-de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:25.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T03:44:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:25 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1046100247' entity='client.admin' 2024-03-26T03:44:25.652 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi006", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:28.517 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T03:44:28.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T03:44:28.761 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:44:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2500877929' entity='client.admin' 2024-03-26T03:44:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:31.213 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi086 2024-03-26T03:44:31.213 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:44:31.213 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.conf 2024-03-26T03:44:31.232 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:44:31.232 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:44:31.289 INFO:tasks.cephadm:Adding host smithi086 to orchestrator... 2024-03-26T03:44:31.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch host add smithi086 2024-03-26T03:44:31.524 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:44:31.636 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:44:31.637 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:44:31.637 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:44:31.637 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:31.637 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:31.637 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:34.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:34 smithi006 ceph-mon[29446]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:35 smithi006 ceph-mon[29446]: Deploying cephadm binary to smithi086 2024-03-26T03:44:37.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:37 smithi006 ceph-mon[29446]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:37.827 INFO:teuthology.orchestra.run.smithi006.stdout:Added host 'smithi086' with addr '172.21.15.86' 2024-03-26T03:44:38.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T03:44:38.661 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:44:39.118 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:39.118 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:38 smithi006 ceph-mon[29446]: Added host smithi086 2024-03-26T03:44:39.118 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:39.118 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:40.107 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:39 smithi006 ceph-mon[29446]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:40.107 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:44:40.362 INFO:teuthology.orchestra.run.smithi006.stdout:[{"addr": "172.21.15.6", "hostname": "smithi006", "labels": [], "status": ""}, {"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}] 2024-03-26T03:44:40.905 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T03:44:40.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T03:44:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:44:41.522 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:41 smithi006 ceph-mon[29446]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:41.522 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:41 smithi006 ceph-mon[29446]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:44:41.522 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:41 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:43.476 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2024-03-26T03:44:43.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:43 smithi006 ceph-mon[29446]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:43.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/191535065' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T03:44:44.294 INFO:tasks.cephadm:Adding mon.a on smithi006 2024-03-26T03:44:44.294 INFO:tasks.cephadm:Adding mon.c on smithi006 2024-03-26T03:44:44.295 INFO:tasks.cephadm:Adding mon.b on smithi086 2024-03-26T03:44:44.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi086:172.21.15.86=b' 2024-03-26T03:44:44.533 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/191535065' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:44:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:44:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:44:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:44 smithi006 ceph-mon[29446]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:44:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:45 smithi006 ceph-mon[29446]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:45 smithi006 ceph-mon[29446]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:44:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:46.281 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mon update... 2024-03-26T03:44:46.906 DEBUG:teuthology.orchestra.run.smithi006:mon.c> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T03:44:46.909 DEBUG:teuthology.orchestra.run.smithi086:mon.b> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T03:44:46.911 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T03:44:46.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi086:172.21.15.86=b;count:3 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:44:47.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:47.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:44:47.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:47.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:47 smithi006 ceph-mon[29446]: Deploying daemon mon.b on smithi086 2024-03-26T03:44:47.660 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:44:49.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:49 smithi006 ceph-mon[29446]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:51.025 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:44:51.025 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":1,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","modified":"2024-03-26T03:43:11.617705Z","created":"2024-03-26T03:43:11.617705Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T03:44:51.027 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 1 2024-03-26T03:44:51.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:51 smithi086 systemd[1]: Started Ceph mon.b for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:44:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:51 smithi006 ceph-mon[29446]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/3357139704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:44:52.850 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T03:44:52.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T03:44:53.095 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:44:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:44:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: mon.a calling monitor election 2024-03-26T03:44:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: mon.b calling monitor election 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: monmap epoch 2 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: last_changed 2024-03-26T03:44:51.343697+0000 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: min_mon_release 19 (squid) 2024-03-26T03:44:56.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: election_strategy: 1 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: fsmap 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: mgrmap e13: y(active, since 40s) 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: overall HEALTH_OK 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:44:56.840 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:44:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: mon.a calling monitor election 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: mon.b calling monitor election 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: monmap epoch 2 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: last_changed 2024-03-26T03:44:51.343697+0000 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:44:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: min_mon_release 19 (squid) 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: election_strategy: 1 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: fsmap 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: mgrmap e13: y(active, since 40s) 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: overall HEALTH_OK 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:44:56.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:57.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:57 smithi086 ceph-mon[32965]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:57.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:57 smithi086 ceph-mon[32965]: Deploying daemon mon.c on smithi006 2024-03-26T03:44:57.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:57 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:57.864 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:57 smithi006 ceph-mon[29446]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:57.864 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:57 smithi006 ceph-mon[29446]: Deploying daemon mon.c on smithi006 2024-03-26T03:44:57.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:57 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:57.877 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:44:57.877 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":2,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","modified":"2024-03-26T03:44:51.343697Z","created":"2024-03-26T03:43:11.617705Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T03:44:57.879 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 2 2024-03-26T03:44:58.763 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/44657625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:44:58.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/44657625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:44:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 systemd[1]: Starting Ceph mon.c for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:44:59.490 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T03:44:59.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T03:44:59.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[29446]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.658 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 podman[37477]: 2024-03-26 03:44:59.308657524 +0000 UTC m=+0.017854787 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 podman[37477]: 2024-03-26 03:44:59.415233846 +0000 UTC m=+0.124431087 container create e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 podman[37477]: 2024-03-26 03:44:59.609528271 +0000 UTC m=+0.318725525 container init e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 podman[37477]: 2024-03-26 03:44:59.612781852 +0000 UTC m=+0.321979096 container start e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: ceph version 19.0.0-2475-ge42b8ea3 (e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003) squid (dev), process ceph-mon, pid 2 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pidfile_write: ignore empty --pid-file 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: load: jerasure load: lrc 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: RocksDB version: 7.9.2 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Git sha 0 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Compile date 2024-03-25 23:25:11 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: DB SUMMARY 2024-03-26T03:44:59.659 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: DB Session ID: S4UN937AVZOCRLZN5D2R 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: CURRENT file: CURRENT 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.error_if_exists: 0 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.create_if_missing: 0 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.paranoid_checks: 1 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.env: 0x561f23ca1c60 2024-03-26T03:44:59.660 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.info_log: 0x561f25845ae0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.statistics: (nil) 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.use_fsync: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_log_file_size: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_fallocate: 1 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.use_direct_reads: 0 2024-03-26T03:44:59.661 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.db_log_dir: 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.wal_dir: 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.write_buffer_manager: 0x561f258161e0 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T03:44:59.662 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T03:44:59.663 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T03:44:59.663 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T03:44:59.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T03:44:59.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T03:44:59.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.unordered_write: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.row_cache: None 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.wal_filter: None 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.two_write_queues: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.wal_compression: 0 2024-03-26T03:44:59.665 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.atomic_flush: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.log_readahead_size: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_background_jobs: 2 2024-03-26T03:44:59.666 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_background_compactions: -1 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_subcompactions: 1 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_open_files: -1 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T03:44:59.667 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_background_flushes: -1 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Compression algorithms supported: 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kZSTD supported: 0 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kXpressCompression supported: 0 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kZlibCompression supported: 1 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kSnappyCompression supported: 1 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kLZ4Compression supported: 1 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: kBZip2Compression supported: 0 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T03:44:59.668 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.merge_operator: 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_filter: None 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561f25845cc0) 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: cache_index_and_filter_blocks: 1 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: pin_top_level_index_and_filter: 1 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_type: 0 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: data_block_index_type: 0 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_shortening: 1 2024-03-26T03:44:59.669 INFO:journalctl@ceph.mon.c.smithi006.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: checksum: 4 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: no_block_cache: 0 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache: 0x561f25849090 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_name: BinnedLRUCache 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_options: 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: capacity : 536870912 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: num_shard_bits : 4 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: strict_capacity_limit : 0 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: high_pri_pool_ratio: 0.000 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_cache_compressed: (nil) 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: persistent_cache: (nil) 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_size: 4096 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_size_deviation: 10 2024-03-26T03:44:59.670 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_restart_interval: 16 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: index_block_restart_interval: 1 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: metadata_block_size: 4096 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: partition_filters: 0 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: use_delta_encoding: 1 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: filter_policy: bloomfilter 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: whole_key_filtering: 1 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: verify_compression: 0 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: read_amp_bytes_per_bit: 0 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: format_version: 5 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: enable_index_compression: 1 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: block_align: 0 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: max_auto_readahead_size: 262144 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: prepopulate_block_cache: 0 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: initial_auto_readahead_size: 8192 2024-03-26T03:44:59.671 INFO:journalctl@ceph.mon.c.smithi006.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression: NoCompression 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.num_levels: 7 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T03:44:59.672 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T03:44:59.673 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T03:44:59.674 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T03:44:59.675 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.inplace_update_support: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.bloom_locality: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.max_successive_merges: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T03:44:59.676 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.ttl: 2592000 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enable_blob_files: false 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.min_blob_size: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T03:44:59.677 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 05101dd5-22a1-4e0e-b2b2-031f4bca4ec3 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424699642798, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424699643220, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711424699, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "05101dd5-22a1-4e0e-b2b2-031f4bca4ec3", "db_session_id": "S4UN937AVZOCRLZN5D2R", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711424699643309, "job": 1, "event": "recovery_finished"} 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561f25938000 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: DB pointer 0x561f25922000 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T03:44:59.678 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** DB Stats ** 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T03:44:59.679 INFO:journalctl@ceph.mon.c.smithi006.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** Compaction Stats [default] ** 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T03:44:59.680 INFO:journalctl@ceph.mon.c.smithi006.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: Block cache BinnedLRUCache@0x561f25849090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: using public_addrv [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: starting mon.c rank -1 at public addrs [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] at bind addrs [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(???) e0 preinit fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: e1 2024-03-26T03:44:59.681 INFO:journalctl@ceph.mon.c.smithi006.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout: legacy client fscid: -1 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout: 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout: No filesystems configured 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:44:59.682 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/191535065' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/191535065' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:59.683 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi086:172.21.15.86=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Saving service mon spec with placement smithi006:172.21.15.6=a;smithi006:[v2:172.21.15.6:3301,v1:172.21.15.6:6790]=c;smithi086:172.21.15.86=b;count:3 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:44:59.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Deploying daemon mon.b on smithi086 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/3357139704' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.a calling monitor election 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.685 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.b calling monitor election 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: monmap epoch 2 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: last_changed 2024-03-26T03:44:51.343697+0000 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: min_mon_release 19 (squid) 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: election_strategy: 1 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:44:59.686 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: fsmap 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mgrmap e13: y(active, since 40s) 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: overall HEALTH_OK 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: Deploying daemon mon.c on smithi006 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/44657625' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:44:59.687 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 ceph-mon[37501]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T03:44:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:44:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:44:59 smithi086 ceph-mon[32965]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 bash[37477]: e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f 2024-03-26T03:45:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:44:59 smithi006 systemd[1]: Started Ceph mon.c for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:45:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:45:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: mon.a calling monitor election 2024-03-26T03:45:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: mon.b calling monitor election 2024-03-26T03:45:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:45:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: mon.c calling monitor election 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: monmap epoch 3 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: last_changed 2024-03-26T03:44:59.665249+0000 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: min_mon_release 19 (squid) 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: election_strategy: 1 2024-03-26T03:45:05.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: 2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: fsmap 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: mgrmap e13: y(active, since 48s) 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: overall HEALTH_OK 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:45:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: mon.a calling monitor election 2024-03-26T03:45:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: mon.b calling monitor election 2024-03-26T03:45:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:45:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: mon.c calling monitor election 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: monmap epoch 3 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: last_changed 2024-03-26T03:44:59.665249+0000 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: min_mon_release 19 (squid) 2024-03-26T03:45:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: election_strategy: 1 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: 2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: fsmap 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: mgrmap e13: y(active, since 48s) 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: overall HEALTH_OK 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:45:05.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: mon.a calling monitor election 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: mon.b calling monitor election 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: mon.c calling monitor election 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: monmap epoch 3 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: last_changed 2024-03-26T03:44:59.665249+0000 2024-03-26T03:45:05.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: created 2024-03-26T03:43:11.617705+0000 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: min_mon_release 19 (squid) 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: election_strategy: 1 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: 0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: 1: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.b 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: 2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: fsmap 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: mgrmap e13: y(active, since 48s) 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: overall HEALTH_OK 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.765 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-26T03:45:05.765 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":3,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","modified":"2024-03-26T03:44:59.665249Z","created":"2024-03-26T03:43:11.617705Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3300","nonce":0},{"type":"v1","addr":"172.21.15.6:6789","nonce":0}]},"addr":"172.21.15.6:6789/0","public_addr":"172.21.15.6:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:3301","nonce":0},{"type":"v1","addr":"172.21.15.6:6790","nonce":0}]},"addr":"172.21.15.6:6790/0","public_addr":"172.21.15.6:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T03:45:05.767 INFO:teuthology.orchestra.run.smithi086.stderr:dumped monmap epoch 3 2024-03-26T03:45:05.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:05 smithi086 ceph-mon[32965]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:05.986 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:05.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[29446]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[37501]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:06.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:06.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:45:06.783 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T03:45:06.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T03:45:07.038 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/376970694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:06 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/376970694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/376970694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:45:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:07.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:06 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:08.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:07 smithi086 ceph-mon[32965]: Reconfiguring daemon mon.a on smithi006 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:08.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[29446]: Reconfiguring daemon mon.a on smithi006 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:08.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:07 smithi006 ceph-mon[37501]: Reconfiguring daemon mon.a on smithi006 2024-03-26T03:45:08.979 INFO:teuthology.orchestra.run.smithi006.stdout:# minimal ceph.conf for de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:45:08.979 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2024-03-26T03:45:08.979 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:45:08.979 INFO:teuthology.orchestra.run.smithi006.stdout: mon_host = [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2024-03-26T03:45:09.635 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T03:45:09.635 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:45:09.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T03:45:09.666 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:45:09.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:45:09.734 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:45:09.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T03:45:09.769 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:45:09.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:45:09.840 INFO:tasks.cephadm:Adding mgr.y on smithi006 2024-03-26T03:45:09.840 INFO:tasks.cephadm:Adding mgr.x on smithi086 2024-03-26T03:45:09.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi006=y;smithi086=x' 2024-03-26T03:45:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:09 smithi006 ceph-mon[29446]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4176366923' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:09 smithi006 ceph-mon[37501]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4176366923' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:09 smithi086 ceph-mon[32965]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4176366923' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:10.116 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: Reconfiguring mon.c (monmap changed)... 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:11.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:11.323 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:11 smithi086 ceph-mon[32965]: Reconfiguring daemon mon.c on smithi006 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: Reconfiguring mon.c (monmap changed)... 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:11.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[29446]: Reconfiguring daemon mon.c on smithi006 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: Reconfiguring mon.c (monmap changed)... 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:11.360 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:11 smithi006 ceph-mon[37501]: Reconfiguring daemon mon.c on smithi006 2024-03-26T03:45:11.790 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled mgr update... 2024-03-26T03:45:12.401 DEBUG:teuthology.orchestra.run.smithi086:mgr.x> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T03:45:12.403 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T03:45:12.404 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:45:12.404 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T03:45:12.422 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T03:45:12.422 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2024-03-26T03:45:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T03:45:12.480 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:12.480 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-26 03:44:59.130834016 +0000 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-26 03:37:22.224720111 +0000 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-26 03:37:22.224720111 +0000 2024-03-26T03:45:12.481 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-26 03:37:22.224720111 +0000 2024-03-26T03:45:12.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T03:45:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-26T03:45:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-26T03:45:12.550 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000279586 s, 1.8 MB/s 2024-03-26T03:45:12.552 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T03:45:12.609 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2024-03-26T03:45:12.664 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-26 03:44:59.131834005 +0000 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-26 03:37:22.556716827 +0000 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-26 03:37:22.556716827 +0000 2024-03-26T03:45:12.665 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-26 03:37:22.556716827 +0000 2024-03-26T03:45:12.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T03:45:12.732 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-26T03:45:12.732 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-26T03:45:12.732 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000201904 s, 2.5 MB/s 2024-03-26T03:45:12.733 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T03:45:12.789 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-26 03:44:59.131834005 +0000 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-26 03:37:22.892713503 +0000 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-26 03:37:22.892713503 +0000 2024-03-26T03:45:12.845 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-26 03:37:22.892713503 +0000 2024-03-26T03:45:12.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T03:45:12.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: Saving service mgr spec with placement smithi006=y;smithi086=x;count:2 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: Reconfiguring mon.b (monmap changed)... 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[29446]: Reconfiguring daemon mon.b on smithi086 2024-03-26T03:45:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: Saving service mgr spec with placement smithi006=y;smithi086=x;count:2 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: Reconfiguring mon.b (monmap changed)... 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:12.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:12 smithi006 ceph-mon[37501]: Reconfiguring daemon mon.b on smithi086 2024-03-26T03:45:12.912 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-26T03:45:12.912 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-26T03:45:12.912 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000200144 s, 2.6 MB/s 2024-03-26T03:45:12.914 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T03:45:12.970 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2024-03-26T03:45:13.024 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2024-03-26 03:44:59.131834005 +0000 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2024-03-26 03:37:23.241710051 +0000 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2024-03-26 03:37:23.241710051 +0000 2024-03-26T03:45:13.025 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: 2024-03-26 03:37:23.241710051 +0000 2024-03-26T03:45:13.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T03:45:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi006=y;smithi086=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: Saving service mgr spec with placement smithi006=y;smithi086=x;count:2 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: Reconfiguring mon.b (monmap changed)... 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:12 smithi086 ceph-mon[32965]: Reconfiguring daemon mon.b on smithi086 2024-03-26T03:45:13.090 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2024-03-26T03:45:13.091 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2024-03-26T03:45:13.091 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000183954 s, 2.8 MB/s 2024-03-26T03:45:13.092 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T03:45:13.151 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:45:13.151 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T03:45:13.167 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T03:45:13.167 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:13.223 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-26 03:44:50.259462392 +0000 2024-03-26T03:45:13.224 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-26 03:33:14.575701450 +0000 2024-03-26T03:45:13.224 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-26 03:33:14.575701450 +0000 2024-03-26T03:45:13.224 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-26 03:33:14.575701450 +0000 2024-03-26T03:45:13.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T03:45:13.290 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-26T03:45:13.290 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-26T03:45:13.290 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000210404 s, 2.4 MB/s 2024-03-26T03:45:13.291 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T03:45:13.348 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-26T03:45:13.402 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-26 03:44:50.260462369 +0000 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-26 03:33:14.925693520 +0000 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-26 03:33:14.925693520 +0000 2024-03-26T03:45:13.403 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-26 03:33:14.925693520 +0000 2024-03-26T03:45:13.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T03:45:13.468 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-26T03:45:13.469 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-26T03:45:13.469 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000171363 s, 3.0 MB/s 2024-03-26T03:45:13.470 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T03:45:13.526 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-26 03:44:50.260462369 +0000 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-26 03:33:15.260685929 +0000 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-26 03:33:15.260685929 +0000 2024-03-26T03:45:13.581 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-26 03:33:15.260685929 +0000 2024-03-26T03:45:13.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T03:45:13.647 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-26T03:45:13.647 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-26T03:45:13.647 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000214665 s, 2.4 MB/s 2024-03-26T03:45:13.648 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T03:45:13.704 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-26T03:45:13.759 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T03:45:13.759 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-26 03:44:50.260462369 +0000 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-26 03:33:15.610677999 +0000 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-26 03:33:15.610677999 +0000 2024-03-26T03:45:13.760 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-26 03:33:15.610677999 +0000 2024-03-26T03:45:13.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T03:45:13.825 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-26T03:45:13.825 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-26T03:45:13.825 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000175876 s, 2.9 MB/s 2024-03-26T03:45:13.826 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T03:45:13.882 INFO:tasks.cephadm:Deploying osd.0 on smithi006 with /dev/vg_nvme/lv_4... 2024-03-26T03:45:13.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T03:45:14.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:14.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:14.080 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:14.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:14.134 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:45:14.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:14.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:15.251 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:15 smithi086 ceph-mon[32965]: Deploying daemon mgr.x on smithi086 2024-03-26T03:45:15.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:15 smithi006 ceph-mon[29446]: Deploying daemon mgr.x on smithi086 2024-03-26T03:45:15.336 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:15 smithi006 ceph-mon[37501]: Deploying daemon mgr.x on smithi086 2024-03-26T03:45:15.588 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 systemd[1]: Starting Ceph mgr.x for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:45:15.892 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 podman[34806]: 2024-03-26 03:45:15.533318673 +0000 UTC m=+0.017723706 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:45:15.892 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 podman[34806]: 2024-03-26 03:45:15.670493892 +0000 UTC m=+0.154898958 container create ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True) 2024-03-26T03:45:15.892 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 podman[34806]: 2024-03-26 03:45:15.864245262 +0000 UTC m=+0.348650286 container init ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T03:45:15.892 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 podman[34806]: 2024-03-26 03:45:15.86685832 +0000 UTC m=+0.351263347 container start ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0135836 s, 772 MB/s 2024-03-26T03:45:16.007 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-26T03:45:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-mon[32965]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.339 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 ceph-mgr[34826]: -- 172.21.15.86:0/4163878588 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56245263e5a0 con 0x562452613000 2024-03-26T03:45:16.339 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 bash[34806]: ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab 2024-03-26T03:45:16.339 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:15 smithi086 systemd[1]: Started Ceph mgr.x for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[29446]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[37501]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:16.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_4 2024-03-26T03:45:16.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:16.431+0000 7feb17eae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:45:16.838 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:16.527+0000 7feb17eae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:45:16.907 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:45:17.210 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-mon[32965]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:17.210 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:17.210 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:17.292 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[37501]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:17.292 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:17.293 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:17.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[29446]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:17.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:17.293 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:17 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:17.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.208+0000 7feb17eae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:45:17.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.301+0000 7feb17eae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:45:17.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.453+0000 7feb17eae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:45:17.552 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.550+0000 7feb17eae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:45:17.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.635+0000 7feb17eae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:45:18.187 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:17.897+0000 7feb17eae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:45:18.187 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:18.006+0000 7feb17eae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:45:18.187 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:18.093+0000 7feb17eae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:45:18.188 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:18.186+0000 7feb17eae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:45:18.587 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:18.359+0000 7feb17eae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:18 smithi086 ceph-mon[32965]: Reconfiguring daemon mgr.y on smithi006 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[37501]: Reconfiguring daemon mgr.y on smithi006 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:18.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:18 smithi006 ceph-mon[29446]: Reconfiguring daemon mgr.y on smithi006 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:19.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:19.587 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:19.325+0000 7feb17eae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:19.455+0000 7feb17eae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='client.14220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:20.071 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:19.785+0000 7feb17eae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:45:20.071 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:19.874+0000 7feb17eae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:45:20.071 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:20.070+0000 7feb17eae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:45:20.337 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:20.257+0000 7feb17eae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:45:20.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:20.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:20.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:20.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:20.625 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:20.625 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:20 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:20.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:20.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:20.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:20.797 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:20.539+0000 7feb17eae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:45:20.798 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:20.796+0000 7feb17eae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:45:21.087 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:21.034+0000 7feb17eae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:45:21.834 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:21 smithi086 ceph-mon[32965]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:21.834 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]: dispatch 2024-03-26T03:45:21.834 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]': finished 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[29446]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]: dispatch 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]': finished 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[37501]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]: dispatch 2024-03-26T03:45:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2652578612' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e"}]': finished 2024-03-26T03:45:22.087 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:21.833+0000 7feb17eae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:45:22.714 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-mon[32965]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T03:45:22.714 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:22.714 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/347870646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:22.714 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:22.398+0000 7feb17eae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:45:22.715 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:22.495+0000 7feb17eae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:45:22.715 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:45:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:45:22.713+0000 7feb17eae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[29446]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/347870646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[37501]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/347870646' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: Standby manager daemon x started 2024-03-26T03:45:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:45:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:45:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:45:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:23 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: Standby manager daemon x started 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:45:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: Standby manager daemon x started 2024-03-26T03:45:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:45:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:45:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:45:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:23 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/47024390' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:45:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:24 smithi006 ceph-mon[37501]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T03:45:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:24 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:45:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:24 smithi006 ceph-mon[29446]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T03:45:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:24 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:45:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:24 smithi086 ceph-mon[32965]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T03:45:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:24 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:45:25.733 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:25 smithi006 ceph-mon[37501]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:25.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:25 smithi006 ceph-mon[29446]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:25 smithi086 ceph-mon[32965]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:27.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:27 smithi006 ceph-mon[29446]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:27.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:27 smithi006 ceph-mon[37501]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:27 smithi086 ceph-mon[32965]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:28.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T03:45:28.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:28.550 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[29446]: Deploying daemon osd.0 on smithi006 2024-03-26T03:45:28.550 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T03:45:28.551 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:28.551 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:28 smithi006 ceph-mon[37501]: Deploying daemon osd.0 on smithi006 2024-03-26T03:45:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T03:45:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:28 smithi086 ceph-mon[32965]: Deploying daemon osd.0 on smithi006 2024-03-26T03:45:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:29 smithi006 ceph-mon[29446]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:29 smithi006 ceph-mon[37501]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:29 smithi086 ceph-mon[32965]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:31.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:31 smithi006 ceph-mon[29446]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:31.820 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:31 smithi006 ceph-mon[37501]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:31 smithi086 ceph-mon[32965]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[29446]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[37501]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:33.558 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:33.559 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:33.559 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:33 smithi086 ceph-mon[32965]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:36.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:36.253 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[29446]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.254 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:36 smithi006 ceph-mon[37501]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:36 smithi086 ceph-mon[32965]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T03:45:36.605 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 0 on host 'smithi006' 2024-03-26T03:45:37.350 DEBUG:teuthology.orchestra.run.smithi006:osd.0> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T03:45:37.353 INFO:tasks.cephadm:Deploying osd.1 on smithi006 with /dev/vg_nvme/lv_3... 2024-03-26T03:45:37.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:37.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:38.053 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:45:38.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0[41524]: 2024-03-26T03:45:37.718+0000 7f472b0ff700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T03:45:38.140 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:45:37 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0[41524]: 2024-03-26T03:45:37.721+0000 7f4722eea700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.474 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:45:38.475 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T03:45:38.475 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.475 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.475 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' 2024-03-26T03:45:38.475 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:45:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T03:45:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: from='osd.0 v2:172.21.15.6:6801/1246039834' entity='osd.0' 2024-03-26T03:45:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: osd.0 v2:172.21.15.6:6801/1246039834 boot 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T03:45:40.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: osd.0 v2:172.21.15.6:6801/1246039834 boot 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T03:45:40.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: osd.0 v2:172.21.15.6:6801/1246039834 boot 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T03:45:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:45:40.420 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T03:45:40.420 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T03:45:40.420 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-26T03:45:40.421 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-26T03:45:40.421 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012122 s, 865 MB/s 2024-03-26T03:45:40.421 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-26T03:45:41.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_3 2024-03-26T03:45:41.371 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:41 smithi006 ceph-mon[37501]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T03:45:41.371 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:41 smithi006 ceph-mon[37501]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:41.372 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:41 smithi006 ceph-mon[29446]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T03:45:41.372 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:41 smithi006 ceph-mon[29446]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:41.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:41 smithi086 ceph-mon[32965]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T03:45:41.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:41 smithi086 ceph-mon[32965]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:45:42.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi006 2024-03-26T03:45:42.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 16069M 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi006 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 16069M 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:42.931 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:42.932 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:42 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi006 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 16069M 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:43.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:45:43.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:42 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:43.965 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[29446]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[37501]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:43.966 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:43 smithi086 ceph-mon[32965]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:45:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:45:44.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:44.863 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:44 smithi006 ceph-mon[37501]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:44.863 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:44 smithi006 ceph-mon[29446]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:44 smithi086 ceph-mon[32965]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:45:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:45 smithi086 ceph-mon[32965]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:45 smithi006 ceph-mon[29446]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:46.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:45 smithi006 ceph-mon[37501]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]: dispatch 2024-03-26T03:45:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]': finished 2024-03-26T03:45:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:46 smithi086 ceph-mon[32965]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T03:45:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:46 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]: dispatch 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]': finished 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[29446]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]: dispatch 2024-03-26T03:45:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1756047753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "753f7459-62ec-447b-a0da-64d06e813f16"}]': finished 2024-03-26T03:45:47.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[37501]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T03:45:47.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:46 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:45:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:47 smithi086 ceph-mon[32965]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1518436483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:47 smithi006 ceph-mon[29446]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1518436483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:47 smithi006 ceph-mon[37501]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1518436483' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:45:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:49 smithi006 ceph-mon[29446]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:49.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:49 smithi006 ceph-mon[37501]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:49 smithi086 ceph-mon[32965]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:51 smithi086 ceph-mon[32965]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:51 smithi006 ceph-mon[29446]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:51 smithi006 ceph-mon[37501]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T03:45:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:53.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T03:45:53.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:53.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T03:45:53.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:45:53.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:53 smithi006 ceph-mon[29446]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:53.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:53 smithi006 ceph-mon[29446]: Deploying daemon osd.1 on smithi006 2024-03-26T03:45:53.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:53 smithi006 ceph-mon[37501]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:53.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:53 smithi006 ceph-mon[37501]: Deploying daemon osd.1 on smithi006 2024-03-26T03:45:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:53 smithi086 ceph-mon[32965]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:53 smithi086 ceph-mon[32965]: Deploying daemon osd.1 on smithi006 2024-03-26T03:45:55.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:55 smithi006 ceph-mon[37501]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:55 smithi006 ceph-mon[29446]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:55 smithi086 ceph-mon[32965]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:57.276 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:57 smithi006 ceph-mon[37501]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:57.276 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:57 smithi006 ceph-mon[29446]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:57 smithi086 ceph-mon[32965]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:58 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:58 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:58 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:45:59.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:58 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:45:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:45:59 smithi006 ceph-mon[29446]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:45:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:45:59 smithi006 ceph-mon[37501]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:00.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:45:59 smithi086 ceph-mon[32965]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:01.354 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 1 on host 'smithi006' 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:01.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:01.991 DEBUG:teuthology.orchestra.run.smithi006:osd.1> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T03:46:01.993 INFO:tasks.cephadm:Deploying osd.2 on smithi006 with /dev/vg_nvme/lv_2... 2024-03-26T03:46:01.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:02 smithi086 ceph-mon[32965]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T03:46:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:02 smithi086 ceph-mon[32965]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T03:46:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:02 smithi086 ceph-mon[32965]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:02 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.098 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[29446]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[29446]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[29446]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[37501]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[37501]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[37501]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:03.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:02 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:46:03.386 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1[47368]: 2024-03-26T03:46:03.248+0000 7fd25661a700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T03:46:03.386 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1[47368]: 2024-03-26T03:46:03.251+0000 7fd24e405700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:46:03.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:03.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:03.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T03:46:03.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[29446]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:03.948 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:03 smithi006 ceph-mon[37501]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:03 smithi086 ceph-mon[32965]: from='osd.1 v2:172.21.15.6:6805/260311411' entity='osd.1' 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: osd.1 v2:172.21.15.6:6805/260311411 boot 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T03:46:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: osd.1 v2:172.21.15.6:6805/260311411 boot 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:46:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:46:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: osd.1 v2:172.21.15.6:6805/260311411 boot 2024-03-26T03:46:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T03:46:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118634 s, 884 MB/s 2024-03-26T03:46:06.026 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-26T03:46:06.070 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:06.070 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi006 2024-03-26T03:46:06.070 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.070 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.070 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 8034M 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi006 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 8034M 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:06.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi006 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 8034M 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:06.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_2 2024-03-26T03:46:06.830 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:46:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:06 smithi006 ceph-mon[29446]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T03:46:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:06 smithi006 ceph-mon[37501]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T03:46:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:06 smithi086 ceph-mon[32965]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T03:46:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:07 smithi086 ceph-mon[32965]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:08.091 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:07 smithi006 ceph-mon[29446]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:08.092 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:07 smithi006 ceph-mon[37501]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:09.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:08 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:08 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:08 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:08 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:09 smithi006 ceph-mon[37501]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:09 smithi006 ceph-mon[37501]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:10.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:09 smithi006 ceph-mon[29446]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:10.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:09 smithi006 ceph-mon[29446]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:09 smithi086 ceph-mon[32965]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:09 smithi086 ceph-mon[32965]: from='client.24170 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]: dispatch 2024-03-26T03:46:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]': finished 2024-03-26T03:46:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1484117487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]: dispatch 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]': finished 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:12.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1484117487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]: dispatch 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4062761906' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b7d7409f-341a-4711-8570-92fe9082a528"}]': finished 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1484117487' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:13 smithi006 ceph-mon[29446]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:13 smithi006 ceph-mon[37501]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:13 smithi086 ceph-mon[32965]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:15.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:15 smithi006 ceph-mon[29446]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:15.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:15 smithi006 ceph-mon[37501]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:15 smithi086 ceph-mon[32965]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:17.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:17 smithi006 ceph-mon[29446]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:17 smithi006 ceph-mon[37501]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:17 smithi086 ceph-mon[32965]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[37501]: Deploying daemon osd.2 on smithi006 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:18.432 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:18 smithi006 ceph-mon[29446]: Deploying daemon osd.2 on smithi006 2024-03-26T03:46:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T03:46:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:18 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:18 smithi086 ceph-mon[32965]: Deploying daemon osd.2 on smithi006 2024-03-26T03:46:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:19 smithi006 ceph-mon[29446]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:19.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:19 smithi006 ceph-mon[37501]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:19 smithi086 ceph-mon[32965]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:21.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:21 smithi006 ceph-mon[29446]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:21.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:21 smithi006 ceph-mon[37501]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:21.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:21 smithi086 ceph-mon[32965]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[37501]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[29446]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.325 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:24 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:24 smithi086 ceph-mon[32965]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:24 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:24 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:24.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:24 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:26 smithi086 ceph-mon[32965]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:26.378 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:26 smithi006 ceph-mon[29446]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:26.378 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:26 smithi006 ceph-mon[37501]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:26.423 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 2 on host 'smithi006' 2024-03-26T03:46:27.065 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[37501]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.065 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.065 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.065 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[29446]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[29446]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:27 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.093 DEBUG:teuthology.orchestra.run.smithi006:osd.2> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T03:46:27.096 INFO:tasks.cephadm:Deploying osd.3 on smithi006 with /dev/vg_nvme/lv_1... 2024-03-26T03:46:27.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T03:46:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:27 smithi086 ceph-mon[32965]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:27 smithi086 ceph-mon[32965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T03:46:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:27 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:27 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:27.388 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='osd.2 v2:172.21.15.6:6809/3103276794' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:28.701 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[52984]: 2024-03-26T03:46:28.487+0000 7f757f0ba700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T03:46:28.702 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:28 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[52984]: 2024-03-26T03:46:28.492+0000 7f7576ea5700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[37501]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[37501]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[37501]: from='osd.2 ' entity='osd.2' 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[29446]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[29446]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.272 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:29 smithi006 ceph-mon[29446]: from='osd.2 ' entity='osd.2' 2024-03-26T03:46:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:29 smithi086 ceph-mon[32965]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:29.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:29 smithi086 ceph-mon[32965]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T03:46:29.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:29.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:29 smithi086 ceph-mon[32965]: from='osd.2 ' entity='osd.2' 2024-03-26T03:46:29.397 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T03:46:29.397 INFO:teuthology.orchestra.run.smithi006.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T03:46:29.397 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10+0 records in 2024-03-26T03:46:29.398 INFO:teuthology.orchestra.run.smithi006.stderr:10+0 records out 2024-03-26T03:46:29.398 INFO:teuthology.orchestra.run.smithi006.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121001 s, 867 MB/s 2024-03-26T03:46:29.398 INFO:teuthology.orchestra.run.smithi006.stderr:--> Zapping successful for: 2024-03-26T03:46:30.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi006:vg_nvme/lv_1 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: osd.2 v2:172.21.15.6:6809/3103276794 boot 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T03:46:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:30 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: osd.2 v2:172.21.15.6:6809/3103276794 boot 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T03:46:30.379 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: osd.2 v2:172.21.15.6:6809/3103276794 boot 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T03:46:30.380 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:30 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:46:30.865 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:46:31.297 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:31 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:31.297 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:31 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:32.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[29446]: pgmap v75: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:32.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:32.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[29446]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T03:46:32.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:32.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[37501]: pgmap v75: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:32.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:32.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[37501]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T03:46:32.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 ceph-mon[32965]: pgmap v75: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:32.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:32.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 ceph-mon[32965]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T03:46:32.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:46:32.426 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56785]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.426 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56785]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.426 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56785]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.426 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56785]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56788]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56788]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56788]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56788]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56791]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.427 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56791]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.427 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56749]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56749]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56749]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56749]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56767]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56767]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56767]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56767]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56773]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56773]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56773]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.428 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56773]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56776]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56776]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56776]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56776]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56779]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56779]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56779]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56782]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56782]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.429 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56782]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.823 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56791]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.823 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56795]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56795]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56795]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:32 smithi006 sudo[56795]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35462]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35462]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35462]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35462]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35465]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35465]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35465]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T03:46:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:32 smithi086 sudo[35465]: pam_unix(sudo:session): session closed for user root 2024-03-26T03:46:33.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:33.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T03:46:33.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:33.341 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.341 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.341 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T03:46:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:46:33.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:46:33.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.395 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:33.395 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.395 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.396 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:33.396 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:33.396 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:33.396 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:34.263 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi006 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 5356M 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:34.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[37501]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi006 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 5356M 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:34.265 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:34.266 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:34 smithi006 ceph-mon[29446]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi006 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 5356M 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:34 smithi086 ceph-mon[32965]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T03:46:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:35 smithi006 ceph-mon[29446]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:35 smithi006 ceph-mon[37501]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:35 smithi086 ceph-mon[32965]: from='client.14307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi006:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[29446]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]: dispatch 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]': finished 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[29446]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[37501]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]: dispatch 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]': finished 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[37501]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T03:46:36.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:36 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:36.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:36 smithi086 ceph-mon[32965]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:36.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]: dispatch 2024-03-26T03:46:36.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2494847635' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6"}]': finished 2024-03-26T03:46:36.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:36 smithi086 ceph-mon[32965]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T03:46:36.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:36 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3662426818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:37.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3662426818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3662426818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:46:38.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:38 smithi006 ceph-mon[29446]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:38.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:38 smithi006 ceph-mon[37501]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:38.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:38 smithi086 ceph-mon[32965]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:39.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:39 smithi006 ceph-mon[29446]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:39 smithi006 ceph-mon[37501]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:39.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:39 smithi086 ceph-mon[32965]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:41.365 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:41 smithi006 ceph-mon[37501]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:41.365 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:41 smithi006 ceph-mon[29446]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:41 smithi086 ceph-mon[32965]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:43.386 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[37501]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[37501]: Deploying daemon osd.3 on smithi006 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[29446]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:43.387 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:43 smithi006 ceph-mon[29446]: Deploying daemon osd.3 on smithi006 2024-03-26T03:46:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:43 smithi086 ceph-mon[32965]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T03:46:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:43 smithi086 ceph-mon[32965]: Deploying daemon osd.3 on smithi006 2024-03-26T03:46:45.429 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:45 smithi006 ceph-mon[37501]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:45.429 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:45 smithi006 ceph-mon[29446]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:45.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:45 smithi086 ceph-mon[32965]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:47.298 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:47 smithi006 ceph-mon[29446]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:47.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:47 smithi006 ceph-mon[37501]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:47.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:47 smithi086 ceph-mon[32965]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:48 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:48 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:48 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:48 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:49.988 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:49 smithi006 ceph-mon[37501]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:49.989 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:49 smithi006 ceph-mon[29446]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:49 smithi086 ceph-mon[32965]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:51.133 INFO:teuthology.orchestra.run.smithi006.stdout:Created osd(s) 3 on host 'smithi006' 2024-03-26T03:46:51.708 DEBUG:teuthology.orchestra.run.smithi006:osd.3> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T03:46:51.711 INFO:tasks.cephadm:Deploying osd.4 on smithi086 with /dev/vg_nvme/lv_4... 2024-03-26T03:46:51.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T03:46:51.955 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:46:52.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:51 smithi086 ceph-mon[32965]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:52.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:51 smithi086 ceph-mon[32965]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:51 smithi086 ceph-mon[32965]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:51 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:52.077 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:51 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[29446]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[29446]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[29446]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[37501]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[37501]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[37501]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:51 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='osd.3 v2:172.21.15.6:6813/2795883342' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T03:46:53.180 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.181 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:53.619 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T03:46:53.301+0000 7fa5e3632700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T03:46:53.619 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T03:46:53.304+0000 7fa5db41d700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136865 s, 766 MB/s 2024-03-26T03:46:53.699 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-26T03:46:53.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:53.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:53.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T03:46:53.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[37501]: from='osd.3 ' entity='osd.3' 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:53.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:53 smithi006 ceph-mon[29446]: from='osd.3 ' entity='osd.3' 2024-03-26T03:46:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:54.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]': finished 2024-03-26T03:46:54.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T03:46:54.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:54.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:54.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:53 smithi086 ceph-mon[32965]: from='osd.3 ' entity='osd.3' 2024-03-26T03:46:54.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-03-26T03:46:54.637 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: osd.3 v2:172.21.15.6:6813/2795883342 boot 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T03:46:54.953 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:54 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: osd.3 v2:172.21.15.6:6813/2795883342 boot 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:46:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:46:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: osd.3 v2:172.21.15.6:6813/2795883342 boot 2024-03-26T03:46:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T03:46:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:54 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:46:56.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi006 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 4017M 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi006 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 4017M 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi006 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 4017M 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:46:57.212 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:46:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:57 smithi086 ceph-mon[32965]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:46:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:57 smithi086 ceph-mon[32965]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:57 smithi006 ceph-mon[29446]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:46:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:57 smithi006 ceph-mon[29446]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:46:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:57 smithi006 ceph-mon[37501]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:46:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:57 smithi006 ceph-mon[37501]: from='client.24196 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/3096398068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]': finished 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T03:47:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/51008027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/3096398068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]': finished 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:46:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/51008027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/3096398068' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]: dispatch 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8"}]': finished 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:46:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/51008027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:00 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:00 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:01.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:00 smithi086 ceph-mon[32965]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T03:47:01.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:00 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[29446]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[37501]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T03:47:01.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:00 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:02.056 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:01 smithi086 ceph-mon[32965]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:02.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:01 smithi006 ceph-mon[29446]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:01 smithi006 ceph-mon[37501]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:04.277 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:03 smithi086 ceph-mon[32965]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:04.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:03 smithi006 ceph-mon[37501]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:03 smithi006 ceph-mon[29446]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:05 smithi086 ceph-mon[32965]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T03:47:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:06.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[37501]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:06.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T03:47:06.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:06.192 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[29446]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:06.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T03:47:06.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:07.010 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:06 smithi086 ceph-mon[32965]: Deploying daemon osd.4 on smithi086 2024-03-26T03:47:07.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:06 smithi006 ceph-mon[37501]: Deploying daemon osd.4 on smithi086 2024-03-26T03:47:07.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:06 smithi006 ceph-mon[29446]: Deploying daemon osd.4 on smithi086 2024-03-26T03:47:08.288 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:07 smithi086 ceph-mon[32965]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:08.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:07 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:08.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:07 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:08.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:07 smithi086 ceph-mon[32965]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T03:47:08.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:07 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[37501]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[37501]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[29446]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:08.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:08.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[29446]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T03:47:08.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:07 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:10.206 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:09 smithi086 ceph-mon[32965]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:09 smithi006 ceph-mon[29446]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:09 smithi006 ceph-mon[37501]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:11.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:11 smithi086 ceph-mon[32965]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:11.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:11.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:11.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:11 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[37501]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[29446]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:11.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:11 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:13.707 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:13 smithi086 ceph-mon[32965]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:13.763 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 4 on host 'smithi086' 2024-03-26T03:47:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:13 smithi006 ceph-mon[37501]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:13 smithi006 ceph-mon[29446]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:14.270 DEBUG:teuthology.orchestra.run.smithi086:osd.4> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T03:47:14.273 INFO:tasks.cephadm:Deploying osd.5 on smithi086 with /dev/vg_nvme/lv_3... 2024-03-26T03:47:14.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T03:47:14.556 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:47:14.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T03:47:14.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:14.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:15.634 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-mon[32965]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:15.635 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T03:47:15.453+0000 7fb88a866700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T03:47:15.635 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 03:47:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T03:47:15.460+0000 7fb886659700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[29446]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:15.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T03:47:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: from='osd.4 v2:172.21.15.86:6800/931538411' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: osdmap e30: 5 total, 4 up, 5 in 2024-03-26T03:47:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:15 smithi006 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0181402 s, 578 MB/s 2024-03-26T03:47:16.453 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-26T03:47:16.480 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:16 smithi086 ceph-mon[32965]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:16.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:16 smithi086 ceph-mon[32965]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T03:47:16.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:16 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.481 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:16 smithi086 ceph-mon[32965]: osd.4 v2:172.21.15.86:6800/931538411 boot 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[29446]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[29446]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[29446]: osd.4 v2:172.21.15.86:6800/931538411 boot 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[37501]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[37501]: osdmap e31: 5 total, 4 up, 5 in 2024-03-26T03:47:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:16.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:16 smithi006 ceph-mon[37501]: osd.4 v2:172.21.15.86:6800/931538411 boot 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:17.552 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:17 smithi086 ceph-mon[32965]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[29446]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T03:47:17.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:17.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:17.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T03:47:17.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T03:47:17.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:47:17.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:17 smithi006 ceph-mon[37501]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T03:47:18.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-03-26T03:47:18.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:47:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:19.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T03:47:19.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi086 2024-03-26T03:47:19.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.842 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.843 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:19 smithi086 ceph-mon[32965]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi086 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-26T03:47:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[37501]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi086 2024-03-26T03:47:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 17088M 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:19.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:19 smithi006 ceph-mon[29446]: osdmap e35: 5 total, 5 up, 5 in 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 157 KiB/s, 0 objects/s recovering 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T03:47:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 157 KiB/s, 0 objects/s recovering 2024-03-26T03:47:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:21 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: osdmap e36: 5 total, 5 up, 5 in 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: pgmap v118: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 157 KiB/s, 0 objects/s recovering 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: from='client.24223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:21.420 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:21 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:23.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:23 smithi086 ceph-mon[32965]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-26T03:47:23.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:23 smithi006 ceph-mon[37501]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-26T03:47:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:23 smithi006 ceph-mon[29446]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/759163466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]': finished 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/1657047246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/759163466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]': finished 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/1657047246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/759163466' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]: dispatch 2024-03-26T03:47:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc"}]': finished 2024-03-26T03:47:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T03:47:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/1657047246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:25.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:25 smithi086 ceph-mon[32965]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:47:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:25 smithi006 ceph-mon[29446]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:47:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:25 smithi006 ceph-mon[37501]: pgmap v121: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:47:27.414 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:27 smithi086 ceph-mon[32965]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-26T03:47:27.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:27 smithi006 ceph-mon[29446]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-26T03:47:27.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:27 smithi006 ceph-mon[37501]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-26T03:47:29.581 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:29 smithi086 ceph-mon[32965]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:29 smithi006 ceph-mon[29446]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:29 smithi006 ceph-mon[37501]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:30.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:30 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T03:47:30.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:30 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:30.510 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:30 smithi086 ceph-mon[32965]: Deploying daemon osd.5 on smithi086 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[29446]: Deploying daemon osd.5 on smithi086 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:30.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:30 smithi006 ceph-mon[37501]: Deploying daemon osd.5 on smithi086 2024-03-26T03:47:31.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:31 smithi086 ceph-mon[32965]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:31.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:31 smithi006 ceph-mon[37501]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:31.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:31 smithi006 ceph-mon[29446]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:33 smithi086 ceph-mon[32965]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:33.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:33 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:33.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:33 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[37501]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:33.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[29446]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:33.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:33.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:33 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:34.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:34 smithi086 ceph-mon[32965]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T03:47:34.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:34 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:34.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:34 smithi006 ceph-mon[29446]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T03:47:34.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:34 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:34 smithi006 ceph-mon[37501]: osdmap e38: 6 total, 5 up, 6 in 2024-03-26T03:47:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:34 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:35 smithi086 ceph-mon[32965]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:35 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:35 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:35 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[37501]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[29446]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:36.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:35 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:37.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:36 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:47:37.049 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:36 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:47:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:36 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:47:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:36 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:47:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:36 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:47:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:36 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:38.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:37 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:37 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.352 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 5 on host 'smithi086' 2024-03-26T03:47:38.916 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:38 smithi086 ceph-mon[32965]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:38.917 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:38 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:38.917 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:38.917 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.917 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:38 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:38.941 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T03:47:38.943 INFO:tasks.cephadm:Deploying osd.6 on smithi086 with /dev/vg_nvme/lv_2... 2024-03-26T03:47:38.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T03:47:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[37501]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[29446]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:47:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:47:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:38 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:39.702 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:40.089 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[43916]: 2024-03-26T03:47:39.787+0000 7f5d0ca9b700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T03:47:40.089 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:47:39 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[43916]: 2024-03-26T03:47:39.793+0000 7f5d0888e700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:47:40.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:40.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:47:40.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: from='osd.5 v2:172.21.15.86:6804/529279948' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: osdmap e39: 6 total, 5 up, 6 in 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:40.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:39 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:47:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:40 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:40 smithi086 ceph-mon[32965]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T03:47:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:40 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[29446]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[37501]: osdmap e40: 6 total, 5 up, 6 in 2024-03-26T03:47:41.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:40 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: osd.5 v2:172.21.15.86:6804/529279948 boot 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:41 smithi086 ceph-mon[32965]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: osd.5 v2:172.21.15.86:6804/529279948 boot 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[29446]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: osd.5 v2:172.21.15.86:6804/529279948 boot 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:47:42.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:41 smithi006 ceph-mon[37501]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T03:47:42.398 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T03:47:42.398 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T03:47:42.398 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-26T03:47:42.398 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-26T03:47:42.399 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0147425 s, 711 MB/s 2024-03-26T03:47:42.399 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-26T03:47:43.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-03-26T03:47:43.587 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:47:44.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:44.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi086 2024-03-26T03:47:44.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.022 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.023 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:43 smithi086 ceph-mon[32965]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi086 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[29446]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi086 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 8544M 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:47:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:43 smithi006 ceph-mon[37501]: osdmap e43: 6 total, 6 up, 6 in 2024-03-26T03:47:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:44 smithi086 ceph-mon[32965]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T03:47:45.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:44 smithi006 ceph-mon[29446]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T03:47:45.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:44 smithi006 ceph-mon[37501]: osdmap e44: 6 total, 6 up, 6 in 2024-03-26T03:47:46.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:45 smithi086 ceph-mon[32965]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:46.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:45 smithi086 ceph-mon[32965]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T03:47:46.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:45 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:46.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:45 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:46.081 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:45 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[37501]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[37501]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[29446]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[29446]: osdmap e45: 6 total, 6 up, 6 in 2024-03-26T03:47:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:47:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:47:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:45 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:46.950 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:46 smithi086 ceph-mon[32965]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:46 smithi006 ceph-mon[29446]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:46 smithi006 ceph-mon[37501]: from='client.24250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/2922302414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]': finished 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T03:47:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:47 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/2922302414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]': finished 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 171 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/2922302414' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]: dispatch 2024-03-26T03:47:48.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "58b58555-59a3-4620-a15f-024f7041713f"}]': finished 2024-03-26T03:47:48.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: osdmap e46: 7 total, 6 up, 7 in 2024-03-26T03:47:48.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:47 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:49.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:48 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/1518578966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:48 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/1518578966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:48 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/1518578966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:47:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:49 smithi086 ceph-mon[32965]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:50.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:49 smithi006 ceph-mon[29446]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:49 smithi006 ceph-mon[37501]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:52.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:51 smithi086 ceph-mon[32965]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:51 smithi006 ceph-mon[37501]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:51 smithi006 ceph-mon[29446]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:54.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:53 smithi086 ceph-mon[32965]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:54.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:53 smithi006 ceph-mon[29446]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:53 smithi006 ceph-mon[37501]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:55 smithi086 ceph-mon[32965]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T03:47:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:55 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:55 smithi086 ceph-mon[32965]: Deploying daemon osd.6 on smithi086 2024-03-26T03:47:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[29446]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T03:47:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[29446]: Deploying daemon osd.6 on smithi086 2024-03-26T03:47:55.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[37501]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:55.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T03:47:55.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:47:55.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:55 smithi006 ceph-mon[37501]: Deploying daemon osd.6 on smithi086 2024-03-26T03:47:56.156 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:56 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:47:56.156 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:56 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:47:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:56 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:47:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:56 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:47:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:56 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:47:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:56 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:47:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:57 smithi086 ceph-mon[32965]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T03:47:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:57 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:57 smithi086 ceph-mon[32965]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[29446]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T03:47:57.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:57.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[29446]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:57.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[37501]: osdmap e47: 7 total, 6 up, 7 in 2024-03-26T03:47:57.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:47:57.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:57 smithi006 ceph-mon[37501]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:47:59 smithi086 ceph-mon[32965]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:47:59 smithi006 ceph-mon[29446]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:47:59.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:47:59 smithi006 ceph-mon[37501]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:00 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:00 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:00 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:00.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:00.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:00 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:01 smithi086 ceph-mon[32965]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:01 smithi006 ceph-mon[29446]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:01 smithi006 ceph-mon[37501]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:02.883 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 6 on host 'smithi086' 2024-03-26T03:48:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:03 smithi086 ceph-mon[32965]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:03.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:03 smithi086 ceph-mon[32965]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:03 smithi086 ceph-mon[32965]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:03 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:03 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.650 DEBUG:teuthology.orchestra.run.smithi086:osd.6> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T03:48:03.653 INFO:tasks.cephadm:Deploying osd.7 on smithi086 with /dev/vg_nvme/lv_1... 2024-03-26T03:48:03.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[37501]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[37501]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[37501]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[29446]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[29446]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[29446]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T03:48:03.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:03 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:03.938 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:04.679 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.680 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-mon[32965]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[29446]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='osd.6 v2:172.21.15.86:6808/1580133777' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: osdmap e48: 7 total, 6 up, 7 in 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:04 smithi006 ceph-mon[37501]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:04.938 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T03:48:04.914+0000 7f3f74604700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T03:48:04.938 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 03:48:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T03:48:04.922+0000 7f3f6c3ef700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:48:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T03:48:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: from='osd.6 ' entity='osd.6' 2024-03-26T03:48:05.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: osd.6 v2:172.21.15.86:6808/1580133777 boot 2024-03-26T03:48:05.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T03:48:05.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:05 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: from='osd.6 ' entity='osd.6' 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: osd.6 v2:172.21.15.86:6808/1580133777 boot 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T03:48:05.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: osdmap e49: 7 total, 6 up, 7 in 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: from='osd.6 ' entity='osd.6' 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: osd.6 v2:172.21.15.86:6808/1580133777 boot 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T03:48:05.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:05 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:48:06.781 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:06 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:06.781 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:06 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:06.781 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:06 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:06.781 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:06 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:06.781 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:06 smithi086 ceph-mon[32965]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T03:48:06.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:06.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:06.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:06.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:06.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[37501]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T03:48:06.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:06.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:06.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:06.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:06.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:06 smithi006 ceph-mon[29446]: osdmap e51: 7 total, 7 up, 7 in 2024-03-26T03:48:06.914 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T03:48:06.915 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T03:48:06.915 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-03-26T03:48:06.915 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-03-26T03:48:06.915 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138382 s, 758 MB/s 2024-03-26T03:48:06.915 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-03-26T03:48:07.546 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:07 smithi086 ceph-mon[32965]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:07.546 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:07 smithi086 ceph-mon[32965]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T03:48:07.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-03-26T03:48:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:07 smithi006 ceph-mon[29446]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:07 smithi006 ceph-mon[29446]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T03:48:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:07 smithi006 ceph-mon[37501]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:07.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:07 smithi006 ceph-mon[37501]: osdmap e52: 7 total, 7 up, 7 in 2024-03-26T03:48:08.305 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi086 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.557 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.558 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:09 smithi086 ceph-mon[32965]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi086 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-26T03:48:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[29446]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi086 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 5696M 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:09.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:09 smithi006 ceph-mon[37501]: osdmap e53: 7 total, 7 up, 7 in 2024-03-26T03:48:10.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:10 smithi086 ceph-mon[32965]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T03:48:10.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:10 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:48:10.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:10 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:48:10.753 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:10 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[29446]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[37501]: osdmap e54: 7 total, 7 up, 7 in 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T03:48:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:10 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:11.766 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:11 smithi086 ceph-mon[32965]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:11.766 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:11 smithi086 ceph-mon[32965]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:11 smithi006 ceph-mon[37501]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:11 smithi006 ceph-mon[37501]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:11.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:11 smithi006 ceph-mon[29446]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 353 MiB used, 625 GiB / 626 GiB avail 2024-03-26T03:48:11.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:11 smithi006 ceph-mon[29446]: from='client.24277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:48:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/3140179546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]': finished 2024-03-26T03:48:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T03:48:13.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:13.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:13 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/699349818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:48:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:48:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/3140179546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]': finished 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/699349818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/3140179546' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]: dispatch 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7"}]': finished 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: osdmap e55: 8 total, 7 up, 8 in 2024-03-26T03:48:13.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:13.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:13 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/699349818' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T03:48:15.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:15 smithi006 ceph-mon[29446]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:15.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:15 smithi006 ceph-mon[37501]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:15 smithi086 ceph-mon[32965]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:17 smithi086 ceph-mon[32965]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-26T03:48:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:17 smithi006 ceph-mon[29446]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-26T03:48:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:17 smithi006 ceph-mon[37501]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-03-26T03:48:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:19 smithi086 ceph-mon[32965]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-26T03:48:19.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T03:48:19.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:19 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[37501]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[29446]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 67 KiB/s, 0 objects/s recovering 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T03:48:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:19 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:20 smithi086 ceph-mon[32965]: Deploying daemon osd.7 on smithi086 2024-03-26T03:48:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:20 smithi006 ceph-mon[29446]: Deploying daemon osd.7 on smithi086 2024-03-26T03:48:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:20 smithi006 ceph-mon[37501]: Deploying daemon osd.7 on smithi086 2024-03-26T03:48:21.541 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:21 smithi086 ceph-mon[32965]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T03:48:21.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:21 smithi006 ceph-mon[29446]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T03:48:21.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:21 smithi006 ceph-mon[37501]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T03:48:23.583 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:23 smithi086 ceph-mon[32965]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:23 smithi006 ceph-mon[29446]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:23.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:23 smithi006 ceph-mon[37501]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[29446]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[37501]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:25 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:25.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:25 smithi086 ceph-mon[32965]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:25.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:25 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.817 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:25 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:25.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:25 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:27.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:27 smithi086 ceph-mon[32965]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:27.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:27 smithi006 ceph-mon[29446]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:27.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:27 smithi006 ceph-mon[37501]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:27.705 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 7 on host 'smithi086' 2024-03-26T03:48:28.356 DEBUG:teuthology.orchestra.run.smithi086:osd.7> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T03:48:28.358 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T03:48:28.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T03:48:28.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:28.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:28 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:28.618 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:28.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: osdmap e56: 8 total, 7 up, 8 in 2024-03-26T03:48:28.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:28.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T03:48:28.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:28.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:28 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T03:48:29.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T03:48:29.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:29 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='osd.7 v2:172.21.15.86:6812/3905426306' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: osdmap e57: 8 total, 7 up, 8 in 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:29.650 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-03-26T03:48:29.651 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: osdmap e58: 8 total, 7 up, 8 in 2024-03-26T03:48:29.651 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.651 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:29.651 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T03:48:29.378+0000 7f86feb52700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T03:48:29.651 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 03:48:29 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T03:48:29.387+0000 7f86fa945700 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:48:30.866 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: osd.7 v2:172.21.15.86:6812/3905426306 boot 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1391563999' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:31.457 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:31.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:31.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: osd.7 v2:172.21.15.86:6812/3905426306 boot 2024-03-26T03:48:31.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T03:48:31.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:31.458 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:31 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1391563999' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T03:48:31.487 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":59,"num_osds":8,"num_up_osds":8,"osd_up_since":1711424910,"num_in_osds":8,"osd_in_since":1711424892,"num_remapped_pgs":1} 2024-03-26T03:48:31.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: osd.7 v2:172.21.15.86:6812/3905426306 boot 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:48:31.661 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:31 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1391563999' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T03:48:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:48:32.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:32 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:32.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:32.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:32 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:33.509 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:48:33.509 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":62,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","created":"2024-03-26T03:43:14.534401+0000","modified":"2024-03-26T03:48:33.388598+0000","last_up_change":"2024-03-26T03:48:30.373726+0000","last_in_change":"2024-03-26T03:48:12.741174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T03:46:30.183658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d1c84cf-5fa7-437e-8004-3b7aca831d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":1246039834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1246039834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1246039834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":1246039834}]},"public_addr":"172.21.15.6:6801/1246039834","cluster_addr":"172.21.15.6:6802/1246039834","heartbeat_back_addr":"172.21.15.6:6804/1246039834","heartbeat_front_addr":"172.21.15.6:6803/1246039834","state":["exists","up"]},{"osd":1,"uuid":"753f7459-62ec-447b-a0da-64d06e813f16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":260311411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":260311411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":260311411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":260311411}]},"public_addr":"172.21.15.6:6805/260311411","cluster_addr":"172.21.15.6:6806/260311411","heartbeat_back_addr":"172.21.15.6:6808/260311411","heartbeat_front_addr":"172.21.15.6:6807/260311411","state":["exists","up"]},{"osd":2,"uuid":"b7d7409f-341a-4711-8570-92fe9082a528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3103276794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3103276794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3103276794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":3103276794}]},"public_addr":"172.21.15.6:6809/3103276794","cluster_addr":"172.21.15.6:6810/3103276794","heartbeat_back_addr":"172.21.15.6:6812/3103276794","heartbeat_front_addr":"172.21.15.6:6811/3103276794","state":["exists","up"]},{"osd":3,"uuid":"a91459a0-dee4-420c-ae2f-940756c985a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2795883342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2795883342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2795883342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2795883342}]},"public_addr":"172.21.15.6:6813/2795883342","cluster_addr":"172.21.15.6:6814/2795883342","heartbeat_back_addr":"172.21.15.6:6816/2795883342","heartbeat_front_addr":"172.21.15.6:6815/2795883342","state":["exists","up"]},{"osd":4,"uuid":"087da212-7bf7-4982-b7cc-11f3f66c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":931538411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":931538411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6803","nonce":931538411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":931538411}]},"public_addr":"172.21.15.86:6800/931538411","cluster_addr":"172.21.15.86:6801/931538411","heartbeat_back_addr":"172.21.15.86:6803/931538411","heartbeat_front_addr":"172.21.15.86:6802/931538411","state":["exists","up"]},{"osd":5,"uuid":"7a688745-b213-4bbc-a735-42ffd2a5bfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":529279948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":529279948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6807","nonce":529279948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":529279948}]},"public_addr":"172.21.15.86:6804/529279948","cluster_addr":"172.21.15.86:6805/529279948","heartbeat_back_addr":"172.21.15.86:6807/529279948","heartbeat_front_addr":"172.21.15.86:6806/529279948","state":["exists","up"]},{"osd":6,"uuid":"58b58555-59a3-4620-a15f-024f7041713f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1580133777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":1580133777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6811","nonce":1580133777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1580133777}]},"public_addr":"172.21.15.86:6808/1580133777","cluster_addr":"172.21.15.86:6809/1580133777","heartbeat_back_addr":"172.21.15.86:6811/1580133777","heartbeat_front_addr":"172.21.15.86:6810/1580133777","state":["exists","up"]},{"osd":7,"uuid":"49b1833d-0a24-4168-a195-47b8ac1d9bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3905426306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":3905426306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6815","nonce":3905426306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3905426306}]},"public_addr":"172.21.15.86:6812/3905426306","cluster_addr":"172.21.15.86:6813/3905426306","heartbeat_back_addr":"172.21.15.86:6815/3905426306","heartbeat_front_addr":"172.21.15.86:6814/3905426306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:05.821110+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:31.479472+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:47:32.423937+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:47:55.404526+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:26.905088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:47:38.848064+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:03.553635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:28.494160+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/601470125":"2024-03-27T03:44:16.095969+0000","172.21.15.6:6800/3870917014":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1892511221":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/435737617":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/2150730999":"2024-03-27T03:43:49.288943+0000","172.21.15.6:6800/1166209788":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/1248481299":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1832156537":"2024-03-27T03:44:16.095969+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: Detected new or changed devices on smithi086 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: Detected new or changed devices on smithi086 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:33 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: Detected new or changed devices on smithi086 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 4272M 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:33 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:34.230 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T03:46:30.183658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T03:48:34.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T03:48:34.475 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:48:34.592 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:34 smithi006 ceph-mon[37501]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T03:48:34.592 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3217268943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:48:34.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:34 smithi006 ceph-mon[29446]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T03:48:34.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3217268943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:48:34.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:34 smithi086 ceph-mon[32965]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T03:48:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3217268943' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:48:35.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:35 smithi006 ceph-mon[37501]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:35.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:35 smithi006 ceph-mon[37501]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T03:48:35.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:35 smithi006 ceph-mon[29446]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:35.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:35 smithi006 ceph-mon[29446]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T03:48:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:35 smithi086 ceph-mon[32965]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:35 smithi086 ceph-mon[32965]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T03:48:36.390 INFO:teuthology.orchestra.run.smithi006.stdout:pg_num: 1 2024-03-26T03:48:36.688 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2433917469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T03:48:36.688 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2433917469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T03:48:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2433917469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T03:48:36.987 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi006 2024-03-26T03:48:36.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi006=foo.a' 2024-03-26T03:48:37.237 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:37.616 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:37 smithi086 ceph-mon[32965]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-26T03:48:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:37 smithi006 ceph-mon[29446]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-26T03:48:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:37 smithi006 ceph-mon[37501]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-03-26T03:48:38.987 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled rgw.foo.a update... 2024-03-26T03:48:39.554 DEBUG:teuthology.orchestra.run.smithi006:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T03:48:39.557 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi086 2024-03-26T03:48:39.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T03:48:39.794 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:40.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:40.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:40.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.205 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:39 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T03:48:40.282 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi006=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T03:48:40.283 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:40.284 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:39 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:41.270 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:40 smithi006 ceph-mon[37501]: Deploying daemon rgw.foo.a on smithi006 2024-03-26T03:48:41.271 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:40 smithi006 ceph-mon[29446]: Deploying daemon rgw.foo.a on smithi006 2024-03-26T03:48:41.302 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:40 smithi086 ceph-mon[32965]: Deploying daemon rgw.foo.a on smithi006 2024-03-26T03:48:41.890 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:41 smithi006 systemd[1]: Starting Ceph rgw.foo.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:48:42.010 INFO:teuthology.orchestra.run.smithi086.stderr:pool 'datapool' created 2024-03-26T03:48:42.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[37501]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T03:48:42.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/1449866401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[29446]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T03:48:42.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/1449866401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.227 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:41 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.227 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:41 smithi006 podman[63488]: 2024-03-26 03:48:41.939916944 +0000 UTC m=+0.110032037 container create cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:48:42.227 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:41 smithi006 podman[63488]: 2024-03-26 03:48:41.847335784 +0000 UTC m=+0.017450888 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:48:42.227 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:42 smithi006 podman[63488]: 2024-03-26 03:48:42.151496138 +0000 UTC m=+0.321611237 container init cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T03:48:42.227 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:42 smithi006 podman[63488]: 2024-03-26 03:48:42.154455759 +0000 UTC m=+0.324570850 container start cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-26T03:48:42.227 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:42 smithi006 bash[63488]: cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 2024-03-26T03:48:42.305 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:41 smithi086 ceph-mon[32965]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-03-26T03:48:42.305 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:41 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/1449866401' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.305 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:41 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T03:48:42.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T03:48:42.639 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 03:48:42 smithi006 systemd[1]: Started Ceph rgw.foo.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:48:42.835 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:43 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.267 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:43.268 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:43 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: pgmap v187: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1374352628' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.173 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T03:48:44.174 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:44 smithi086 ceph-mon[32965]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T03:48:44.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: pgmap v187: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1374352628' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[37501]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: pgmap v187: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: Saving service rgw.foo.a spec with placement smithi006=foo.a;count:1 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1374352628' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T03:48:44.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T03:48:44.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:44 smithi006 ceph-mon[29446]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T03:48:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:45 smithi086 ceph-mon[32965]: Checking dashboard <-> RGW credentials 2024-03-26T03:48:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/2986270216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T03:48:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:45 smithi086 ceph-mon[32965]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T03:48:45.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[37501]: Checking dashboard <-> RGW credentials 2024-03-26T03:48:45.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/2986270216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[37501]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[29446]: Checking dashboard <-> RGW credentials 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/2986270216' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T03:48:45.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:45 smithi006 ceph-mon[29446]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:46 smithi086 ceph-mon[32965]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[37501]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T03:48:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:46 smithi006 ceph-mon[29446]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T03:48:47.278 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: pgmap v193: 68 pgs: 6 creating+peering, 32 active+clean, 30 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:48:47.279 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:47 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:48:47.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.86 --placement '1;smithi086=iscsi.a' 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: pgmap v193: 68 pgs: 6 creating+peering, 32 active+clean, 30 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:48:47.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: pgmap v193: 68 pgs: 6 creating+peering, 32 active+clean, 30 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:48:47.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:47 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:48:47.819 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:49 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:49 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:49 smithi086 ceph-mon[32965]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[29446]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T03:48:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:49 smithi006 ceph-mon[37501]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T03:48:49.411 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled iscsi.datapool update... 2024-03-26T03:48:50.007 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T03:48:50.008 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:48:50.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T03:48:50.052 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:48:50.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T03:48:50.082 DEBUG:teuthology.orchestra.run.smithi086:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T03:48:50.124 INFO:tasks.cephadm:Adding prometheus.a on smithi086 2024-03-26T03:48:50.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi086=a' 2024-03-26T03:48:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: pgmap v196: 100 pgs: 6 creating+peering, 45 active+clean, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T03:48:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:50 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: pgmap v196: 100 pgs: 6 creating+peering, 45 active+clean, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T03:48:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: pgmap v196: 100 pgs: 6 creating+peering, 45 active+clean, 49 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T03:48:50.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:50 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:50.401 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.86", "placement": "1;smithi086=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: Saving service iscsi.datapool spec with placement smithi086=iscsi.a;count:1 2024-03-26T03:48:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T03:48:51.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.86", "placement": "1;smithi086=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: Saving service iscsi.datapool spec with placement smithi086=iscsi.a;count:1 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.86", "placement": "1;smithi086=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: Saving service iscsi.datapool spec with placement smithi086=iscsi.a;count:1 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2414067140' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:51.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T03:48:52.171 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled prometheus update... 2024-03-26T03:48:52.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: pgmap v199: 132 pgs: 20 creating+peering, 81 active+clean, 31 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T03:48:52.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:52 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: pgmap v199: 132 pgs: 20 creating+peering, 81 active+clean, 31 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T03:48:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:52.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: pgmap v199: 132 pgs: 20 creating+peering, 81 active+clean, 31 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3868950920' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T03:48:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T03:48:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:52 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:48:52.762 DEBUG:teuthology.orchestra.run.smithi086:prometheus.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T03:48:52.765 INFO:tasks.cephadm:Adding node-exporter.a on smithi006 2024-03-26T03:48:52.765 INFO:tasks.cephadm:Adding node-exporter.b on smithi086 2024-03-26T03:48:52.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi006=a;smithi086=b' 2024-03-26T03:48:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:53 smithi086 ceph-mon[32965]: Deploying daemon iscsi.iscsi.a on smithi086 2024-03-26T03:48:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:53 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:48:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:53 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:48:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:53 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:53.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[29446]: Deploying daemon iscsi.iscsi.a on smithi086 2024-03-26T03:48:53.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:48:53.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:48:53.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:53.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[37501]: Deploying daemon iscsi.iscsi.a on smithi086 2024-03-26T03:48:53.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:48:53.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:48:53.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:53 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:53.641 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:54.321 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:54 smithi086 ceph-mon[32965]: pgmap v201: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 100 op/s 2024-03-26T03:48:54.321 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:54 smithi086 ceph-mon[32965]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:54.322 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:54 smithi086 ceph-mon[32965]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[29446]: pgmap v201: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 100 op/s 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[29446]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[29446]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[37501]: pgmap v201: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 100 op/s 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[37501]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:54 smithi006 ceph-mon[37501]: Saving service prometheus spec with placement smithi086=a;count:1 2024-03-26T03:48:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:55 smithi006 ceph-mon[29446]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T03:48:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:55 smithi006 ceph-mon[29446]: pgmap v203: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.3 KiB/s wr, 81 op/s 2024-03-26T03:48:55.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:55 smithi006 ceph-mon[37501]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T03:48:55.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:55 smithi006 ceph-mon[37501]: pgmap v203: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.3 KiB/s wr, 81 op/s 2024-03-26T03:48:55.391 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:55 smithi086 ceph-mon[32965]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T03:48:55.391 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:55 smithi086 ceph-mon[32965]: pgmap v203: 132 pgs: 15 creating+peering, 117 active+clean; 579 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 36 KiB/s rd, 2.3 KiB/s wr, 81 op/s 2024-03-26T03:48:55.739 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled node-exporter update... 2024-03-26T03:48:56.356 DEBUG:teuthology.orchestra.run.smithi006:node-exporter.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T03:48:56.359 DEBUG:teuthology.orchestra.run.smithi086:node-exporter.b> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T03:48:56.362 INFO:tasks.cephadm:Adding alertmanager.a on smithi006 2024-03-26T03:48:56.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi006=a' 2024-03-26T03:48:56.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 podman[60620]: 2024-03-26 03:48:56.51784605 +0000 UTC m=+0.979106811 container start 13cf7e641c20cadd6b9d995ac60b5a0076ed12445fb1619d98d54c1333b57297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:48:56.697 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:48:56.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:56.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: Saving service node-exporter spec with placement smithi006=a;smithi086=b;count:2 2024-03-26T03:48:56.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:56.951 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:56.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:56.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:56.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:56.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 bash[60620]: 13cf7e641c20cadd6b9d995ac60b5a0076ed12445fb1619d98d54c1333b57297 2024-03-26T03:48:56.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 systemd[1]: Started Ceph iscsi.iscsi.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:48:56.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Started the configuration object watcher 2024-03-26T03:48:56.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Checking for config object changes every 1s 2024-03-26T03:48:56.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Processing osd blocklist entries for this node 2024-03-26T03:48:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: Saving service node-exporter spec with placement smithi006=a;smithi086=b;count:2 2024-03-26T03:48:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi006=a;smithi086=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: Saving service node-exporter spec with placement smithi006=a;smithi086=b;count:2 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:56 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:48:57.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Reading the configuration object to update local LIO configuration 2024-03-26T03:48:57.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Configuration does not have an entry for this host(smithi086.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T03:48:57.203 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T03:48:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-mon[32965]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-03-26T03:48:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-mon[32965]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T03:48:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/4104021399' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[29446]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[29446]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/4104021399' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[37501]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 117 KiB/s rd, 6.0 KiB/s wr, 261 op/s 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[37501]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T03:48:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:48:58.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/4104021399' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T03:48:59.721 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled alertmanager update... 2024-03-26T03:49:00.063 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:59 smithi086 ceph-mon[32965]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T03:49:00.063 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:59 smithi086 ceph-mon[32965]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T03:49:00.063 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:48:59 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[29446]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[29446]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[37501]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.6 KiB/s wr, 200 op/s 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[37501]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T03:49:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:48:59 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:00.634 DEBUG:teuthology.orchestra.run.smithi006:alertmanager.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T03:49:00.636 INFO:tasks.cephadm:Adding grafana.a on smithi086 2024-03-26T03:49:00.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi086=a' 2024-03-26T03:49:00.965 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:49:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:00 smithi086 ceph-mon[32965]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:00 smithi086 ceph-mon[32965]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-26T03:49:01.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:00 smithi006 ceph-mon[37501]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:01.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:00 smithi006 ceph-mon[37501]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-26T03:49:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:00 smithi006 ceph-mon[29446]: from='client.24436 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi006=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:00 smithi006 ceph-mon[29446]: Saving service alertmanager spec with placement smithi006=a;count:1 2024-03-26T03:49:02.053 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 175 op/s 2024-03-26T03:49:02.053 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.053 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.054 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:02.054 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:02.054 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.054 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:01 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 175 op/s 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s rd, 3.7 KiB/s wr, 175 op/s 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:01 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:02.786 INFO:teuthology.orchestra.run.smithi086.stdout:Scheduled grafana update... 2024-03-26T03:49:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:02 smithi086 ceph-mon[32965]: Deploying daemon prometheus.a on smithi086 2024-03-26T03:49:03.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:02 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:02 smithi006 ceph-mon[37501]: Deploying daemon prometheus.a on smithi086 2024-03-26T03:49:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:02 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:03.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:02 smithi006 ceph-mon[29446]: Deploying daemon prometheus.a on smithi086 2024-03-26T03:49:03.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:02 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:03.868 DEBUG:teuthology.orchestra.run.smithi086:grafana.a> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T03:49:03.870 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T03:49:03.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T03:49:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:03 smithi086 ceph-mon[32965]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:03 smithi086 ceph-mon[32965]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:03 smithi086 ceph-mon[32965]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-26T03:49:04.127 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:04.249 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[37501]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:04.249 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[37501]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:04.249 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[37501]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-26T03:49:04.250 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[29446]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:04.250 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[29446]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi086=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T03:49:04.250 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:03 smithi006 ceph-mon[29446]: Saving service grafana spec with placement smithi086=a;count:1 2024-03-26T03:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2024-03-26T03:49:06.128 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQCyRQJmVUmEBxAAa5j5DLqdlei7ScpPnN09Ig== 2024-03-26T03:49:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:05 smithi006 ceph-mon[37501]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:05 smithi006 ceph-mon[29446]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:05 smithi086 ceph-mon[32965]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 54 KiB/s rd, 2.7 KiB/s wr, 120 op/s 2024-03-26T03:49:06.854 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2024-03-26T03:49:06.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T03:49:06.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T03:49:06.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T03:49:07.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:49:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:49:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1905329354' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:07 smithi086 ceph-mon[32965]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 100 op/s 2024-03-26T03:49:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:07 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:07 smithi006 ceph-mon[37501]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 100 op/s 2024-03-26T03:49:08.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:07 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:07 smithi006 ceph-mon[29446]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 45 KiB/s rd, 2.2 KiB/s wr, 100 op/s 2024-03-26T03:49:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:07 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:08.637 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:49:10.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:09 smithi086 ceph-mon[32965]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T03:49:10.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:10.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:10.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:10.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[37501]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T03:49:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[29446]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T03:49:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:11.189 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:10 smithi086 ceph-mon[32965]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T03:49:11.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:10 smithi006 ceph-mon[37501]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T03:49:11.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:10 smithi006 ceph-mon[29446]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T03:49:12.272 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:11 smithi086 ceph-mon[32965]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T03:49:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:11 smithi006 ceph-mon[37501]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T03:49:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:11 smithi006 ceph-mon[29446]: pgmap v212: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T03:49:13.231 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-03-26T03:49:13.231 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQC5RQJmOx+PDRAAYqo4W+6xJmerJKYkp+stZg== 2024-03-26T03:49:14.129 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-26T03:49:14.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T03:49:14.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T03:49:14.155 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:13 smithi086 ceph-mon[32965]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:14.155 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:13 smithi086 ceph-mon[32965]: from='client.? 172.21.15.86:0/3297017625' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.155 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.155 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:14.156 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 systemd[1]: Starting Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:49:14.174 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T03:49:14.175 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T03:49:14.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T03:49:14.202 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[37501]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[37501]: from='client.? 172.21.15.86:0/3297017625' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[29446]: pgmap v213: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[29446]: from='client.? 172.21.15.86:0/3297017625' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T03:49:14.203 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T03:49:14.430 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:14.467 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 podman[62110]: 2024-03-26 03:49:14.269033862 +0000 UTC m=+0.019502885 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T03:49:14.467 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 podman[62110]: 2024-03-26 03:49:14.369471217 +0000 UTC m=+0.119940275 container create a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:49:14.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 podman[62110]: 2024-03-26 03:49:14.571713925 +0000 UTC m=+0.322182937 container init a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:49:14.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 podman[62110]: 2024-03-26 03:49:14.574241013 +0000 UTC m=+0.324710026 container start a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:49:14.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.597Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T03:49:14.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.597Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T03:49:14.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.597Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi086 (none))" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.597Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.597Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.598Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.598Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.599Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.600Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.601Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.601Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.583µs 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.601Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.602Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.602Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=24.943µs wal_replay_duration=204.273µs wbl_replay_duration=90ns total_replay_duration=247.166µs 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.602Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.602Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T03:49:14.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.602Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T03:49:14.840 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.621Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=18.837836ms db_storage=589ns remote_storage=906ns web_handler=285ns query_engine=463ns scrape=607.566µs scrape_sd=82.892µs notify=772ns notify_sd=1.31µs rules=17.831213ms tracing=5.299µs 2024-03-26T03:49:14.840 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.621Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T03:49:14.840 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:49:14.621Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T03:49:14.840 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 bash[62110]: a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 2024-03-26T03:49:14.840 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:49:14 smithi086 systemd[1]: Started Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:15.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T03:49:15.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:14 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' 2024-03-26T03:49:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:14 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T03:49:15.890 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setuser ceph since I am not root 2024-03-26T03:49:15.891 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ignoring --setgroup ceph since I am not root 2024-03-26T03:49:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-mon[32965]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-mon[32965]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T03:49:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-mon[32965]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T03:49:16.088 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: ignoring --setuser ceph since I am not root 2024-03-26T03:49:16.088 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: ignoring --setgroup ceph since I am not root 2024-03-26T03:49:16.088 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:15 smithi086 ceph-mgr[34826]: -- 172.21.15.86:0/4059509452 <== mon.1 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f55b9145a0 con 0x55f55b8e8c00 2024-03-26T03:49:16.340 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[37501]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:16.340 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[37501]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T03:49:16.340 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[37501]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T03:49:16.340 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:16 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:16.338+0000 7f0aa622a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:49:16.341 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[29446]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:16.341 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[29446]: from='mgr.14152 172.21.15.6:0/1859559401' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T03:49:16.341 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:15 smithi006 ceph-mon[29446]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T03:49:16.424 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:49:16.587 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:16.303+0000 7f518f5b9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T03:49:16.588 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:16.395+0000 7f518f5b9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:49:16.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:16 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:16.439+0000 7f0aa622a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T03:49:17.014 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":2547319348}]},"active_addr":"172.21.15.6:6800/2547319348","active_change":"2024-03-26T03:44:16.096084+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.6:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":93567168}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":3964053112}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1978459317}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":1771670410}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.6:0","nonce":14180211}]}]} 2024-03-26T03:49:17.019 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T03:49:17.019 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T03:49:17.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T03:49:17.213 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:16 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3169524792' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T03:49:17.214 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.114+0000 7f0aa622a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:49:17.214 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:16 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3169524792' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T03:49:17.261 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:16 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3169524792' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T03:49:17.261 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.030+0000 7f518f5b9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T03:49:17.261 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.120+0000 7f518f5b9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:49:17.302 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:17.473 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.212+0000 7f0aa622a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T03:49:17.473 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.368+0000 7f0aa622a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:49:17.473 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.472+0000 7f0aa622a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:49:17.587 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.260+0000 7f518f5b9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T03:49:17.588 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.354+0000 7f518f5b9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T03:49:17.588 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.436+0000 7f518f5b9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:49:17.840 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.563+0000 7f0aa622a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T03:49:17.840 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.839+0000 7f0aa622a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:49:17.980 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.695+0000 7f518f5b9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T03:49:17.980 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.802+0000 7f518f5b9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:49:17.980 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.886+0000 7f518f5b9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:49:17.980 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:17.979+0000 7f518f5b9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:49:18.101 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:17 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:17.952+0000 7f0aa622a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T03:49:18.101 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:18 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:18.045+0000 7f0aa622a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T03:49:18.337 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:18.153+0000 7f518f5b9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:49:18.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:18 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:18.143+0000 7f0aa622a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T03:49:18.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:18 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:18.323+0000 7f0aa622a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T03:49:19.198 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:49:19.198 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":76,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","created":"2024-03-26T03:43:14.534401+0000","modified":"2024-03-26T03:49:14.228916+0000","last_up_change":"2024-03-26T03:48:30.373726+0000","last_in_change":"2024-03-26T03:48:12.741174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T03:46:30.183658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T03:48:41.622497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T03:48:42.191818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T03:48:44.079106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T03:48:46.035261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T03:48:48.086810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d1c84cf-5fa7-437e-8004-3b7aca831d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":1246039834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1246039834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1246039834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":1246039834}]},"public_addr":"172.21.15.6:6801/1246039834","cluster_addr":"172.21.15.6:6802/1246039834","heartbeat_back_addr":"172.21.15.6:6804/1246039834","heartbeat_front_addr":"172.21.15.6:6803/1246039834","state":["exists","up"]},{"osd":1,"uuid":"753f7459-62ec-447b-a0da-64d06e813f16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":260311411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":260311411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":260311411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":260311411}]},"public_addr":"172.21.15.6:6805/260311411","cluster_addr":"172.21.15.6:6806/260311411","heartbeat_back_addr":"172.21.15.6:6808/260311411","heartbeat_front_addr":"172.21.15.6:6807/260311411","state":["exists","up"]},{"osd":2,"uuid":"b7d7409f-341a-4711-8570-92fe9082a528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3103276794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3103276794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3103276794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":3103276794}]},"public_addr":"172.21.15.6:6809/3103276794","cluster_addr":"172.21.15.6:6810/3103276794","heartbeat_back_addr":"172.21.15.6:6812/3103276794","heartbeat_front_addr":"172.21.15.6:6811/3103276794","state":["exists","up"]},{"osd":3,"uuid":"a91459a0-dee4-420c-ae2f-940756c985a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2795883342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2795883342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2795883342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2795883342}]},"public_addr":"172.21.15.6:6813/2795883342","cluster_addr":"172.21.15.6:6814/2795883342","heartbeat_back_addr":"172.21.15.6:6816/2795883342","heartbeat_front_addr":"172.21.15.6:6815/2795883342","state":["exists","up"]},{"osd":4,"uuid":"087da212-7bf7-4982-b7cc-11f3f66c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":931538411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":931538411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6803","nonce":931538411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":931538411}]},"public_addr":"172.21.15.86:6800/931538411","cluster_addr":"172.21.15.86:6801/931538411","heartbeat_back_addr":"172.21.15.86:6803/931538411","heartbeat_front_addr":"172.21.15.86:6802/931538411","state":["exists","up"]},{"osd":5,"uuid":"7a688745-b213-4bbc-a735-42ffd2a5bfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":529279948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":529279948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6807","nonce":529279948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":529279948}]},"public_addr":"172.21.15.86:6804/529279948","cluster_addr":"172.21.15.86:6805/529279948","heartbeat_back_addr":"172.21.15.86:6807/529279948","heartbeat_front_addr":"172.21.15.86:6806/529279948","state":["exists","up"]},{"osd":6,"uuid":"58b58555-59a3-4620-a15f-024f7041713f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1580133777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":1580133777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6811","nonce":1580133777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1580133777}]},"public_addr":"172.21.15.86:6808/1580133777","cluster_addr":"172.21.15.86:6809/1580133777","heartbeat_back_addr":"172.21.15.86:6811/1580133777","heartbeat_front_addr":"172.21.15.86:6810/1580133777","state":["exists","up"]},{"osd":7,"uuid":"49b1833d-0a24-4168-a195-47b8ac1d9bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3905426306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":3905426306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6815","nonce":3905426306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3905426306}]},"public_addr":"172.21.15.86:6812/3905426306","cluster_addr":"172.21.15.86:6813/3905426306","heartbeat_back_addr":"172.21.15.86:6815/3905426306","heartbeat_front_addr":"172.21.15.86:6814/3905426306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:13.800393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:31.479472+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:44.841098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:47:55.404526+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:26.905088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:09.254349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:08.994071+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:28.494160+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/601470125":"2024-03-27T03:44:16.095969+0000","172.21.15.6:6800/3870917014":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1892511221":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/435737617":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/2150730999":"2024-03-27T03:43:49.288943+0000","172.21.15.6:6800/1166209788":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/1248481299":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1832156537":"2024-03-27T03:44:16.095969+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T03:49:19.564 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3263277411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:19.565 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:19.104+0000 7f518f5b9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:49:19.565 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:19.232+0000 7f518f5b9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:49:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3263277411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3263277411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:19.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:19.347+0000 7f0aa622a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T03:49:19.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:19.484+0000 7f0aa622a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T03:49:19.804 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T03:49:19.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T03:49:19.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:19.563+0000 7f518f5b9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:49:19.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:19.652+0000 7f518f5b9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:49:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:20.140 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:19.829+0000 7f0aa622a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T03:49:20.141 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:19.926+0000 7f0aa622a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T03:49:20.317 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:19.848+0000 7f518f5b9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:49:20.317 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:20.033+0000 7f518f5b9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:49:20.430 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:20.147+0000 7f0aa622a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:49:20.430 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:20.343+0000 7f0aa622a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T03:49:20.573 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:20.316+0000 7f518f5b9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:49:20.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:20.573+0000 7f518f5b9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:49:20.837 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:20.810+0000 7f518f5b9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:49:20.889 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:20.644+0000 7f0aa622a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T03:49:21.156 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:20.907+0000 7f0aa622a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T03:49:21.416 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:21 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:21.155+0000 7f0aa622a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T03:49:22.026 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:49:22.027 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":77,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","created":"2024-03-26T03:43:14.534401+0000","modified":"2024-03-26T03:49:21.680197+0000","last_up_change":"2024-03-26T03:48:30.373726+0000","last_in_change":"2024-03-26T03:48:12.741174+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T03:46:30.183658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T03:48:41.622497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T03:48:42.191818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T03:48:44.079106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T03:48:46.035261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T03:48:48.086810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d1c84cf-5fa7-437e-8004-3b7aca831d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":1246039834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1246039834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1246039834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":1246039834}]},"public_addr":"172.21.15.6:6801/1246039834","cluster_addr":"172.21.15.6:6802/1246039834","heartbeat_back_addr":"172.21.15.6:6804/1246039834","heartbeat_front_addr":"172.21.15.6:6803/1246039834","state":["exists","up"]},{"osd":1,"uuid":"753f7459-62ec-447b-a0da-64d06e813f16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":260311411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":260311411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":260311411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":260311411}]},"public_addr":"172.21.15.6:6805/260311411","cluster_addr":"172.21.15.6:6806/260311411","heartbeat_back_addr":"172.21.15.6:6808/260311411","heartbeat_front_addr":"172.21.15.6:6807/260311411","state":["exists","up"]},{"osd":2,"uuid":"b7d7409f-341a-4711-8570-92fe9082a528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":3103276794}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":3103276794}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":3103276794}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":3103276794}]},"public_addr":"172.21.15.6:6809/3103276794","cluster_addr":"172.21.15.6:6810/3103276794","heartbeat_back_addr":"172.21.15.6:6812/3103276794","heartbeat_front_addr":"172.21.15.6:6811/3103276794","state":["exists","up"]},{"osd":3,"uuid":"a91459a0-dee4-420c-ae2f-940756c985a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2795883342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2795883342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2795883342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2795883342}]},"public_addr":"172.21.15.6:6813/2795883342","cluster_addr":"172.21.15.6:6814/2795883342","heartbeat_back_addr":"172.21.15.6:6816/2795883342","heartbeat_front_addr":"172.21.15.6:6815/2795883342","state":["exists","up"]},{"osd":4,"uuid":"087da212-7bf7-4982-b7cc-11f3f66c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":931538411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":931538411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6803","nonce":931538411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":931538411}]},"public_addr":"172.21.15.86:6800/931538411","cluster_addr":"172.21.15.86:6801/931538411","heartbeat_back_addr":"172.21.15.86:6803/931538411","heartbeat_front_addr":"172.21.15.86:6802/931538411","state":["exists","up"]},{"osd":5,"uuid":"7a688745-b213-4bbc-a735-42ffd2a5bfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":529279948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":529279948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6807","nonce":529279948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":529279948}]},"public_addr":"172.21.15.86:6804/529279948","cluster_addr":"172.21.15.86:6805/529279948","heartbeat_back_addr":"172.21.15.86:6807/529279948","heartbeat_front_addr":"172.21.15.86:6806/529279948","state":["exists","up"]},{"osd":6,"uuid":"58b58555-59a3-4620-a15f-024f7041713f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1580133777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":1580133777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6811","nonce":1580133777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1580133777}]},"public_addr":"172.21.15.86:6808/1580133777","cluster_addr":"172.21.15.86:6809/1580133777","heartbeat_back_addr":"172.21.15.86:6811/1580133777","heartbeat_front_addr":"172.21.15.86:6810/1580133777","state":["exists","up"]},{"osd":7,"uuid":"49b1833d-0a24-4168-a195-47b8ac1d9bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3905426306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":3905426306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6815","nonce":3905426306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3905426306}]},"public_addr":"172.21.15.86:6812/3905426306","cluster_addr":"172.21.15.86:6813/3905426306","heartbeat_back_addr":"172.21.15.86:6815/3905426306","heartbeat_front_addr":"172.21.15.86:6814/3905426306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:13.800393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:31.479472+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:44.841098+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:21.629519+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:26.905088+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:09.254349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:49:08.994071+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T03:48:28.494160+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/601470125":"2024-03-27T03:44:16.095969+0000","172.21.15.6:6800/3870917014":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1892511221":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/435737617":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/2150730999":"2024-03-27T03:43:49.288943+0000","172.21.15.6:6800/1166209788":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/1248481299":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1832156537":"2024-03-27T03:44:16.095969+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T03:49:22.087 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:21.604+0000 7f518f5b9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:49:22.333 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:22.015+0000 7f0aa622a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T03:49:22.491 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:22.174+0000 7f518f5b9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:49:22.491 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:22.272+0000 7f518f5b9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:49:22.491 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:22.490+0000 7f518f5b9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:49:22.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T03:49:22.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T03:49:22.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T03:49:22.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T03:49:22.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T03:49:22.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T03:49:22.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T03:49:22.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T03:49:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4258482812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: Standby manager daemon x restarted 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: Standby manager daemon x started 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-mon[32965]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: [26/Mar/2024:03:49:22] ENGINE Bus STARTING 2024-03-26T03:49:22.839 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: CherryPy Checker: 2024-03-26T03:49:22.840 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: The Application mounted at '' has an empty config. 2024-03-26T03:49:22.840 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: 2024-03-26T03:49:22.840 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: [26/Mar/2024:03:49:22] ENGINE Serving on http://:::9283 2024-03-26T03:49:22.840 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 03:49:22 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x[34822]: [26/Mar/2024:03:49:22] ENGINE Bus STARTED 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4258482812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: Standby manager daemon x restarted 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: Standby manager daemon x started 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:49:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[29446]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4258482812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: Standby manager daemon x restarted 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: Standby manager daemon x started 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T03:49:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-mon[37501]: from='mgr.? 172.21.15.86:0/1929752087' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T03:49:22.892 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:22.609+0000 7f0aa622a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T03:49:22.892 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:22 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:22.716+0000 7f0aa622a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T03:49:23.299 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.313 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.317 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.319 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.328 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.329 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.332 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.338 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:23.380 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:23.120+0000 7f0aa622a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T03:49:23.380 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:49:23] ENGINE Bus STARTING 2024-03-26T03:49:23.638 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: CherryPy Checker: 2024-03-26T03:49:23.638 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: The Application mounted at '' has an empty config. 2024-03-26T03:49:23.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: 2024-03-26T03:49:23.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:49:23] ENGINE Serving on http://:::9283 2024-03-26T03:49:23.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:49:23] ENGINE Bus STARTED 2024-03-26T03:49:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T03:49:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: Active manager daemon y restarted 2024-03-26T03:49:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: Activating manager daemon y 2024-03-26T03:49:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: mgrmap e19: y(active, starting, since 0.00960164s), standbys: x 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:49:23.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: Manager daemon y is now available 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:23.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: Active manager daemon y restarted 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: Activating manager daemon y 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: mgrmap e19: y(active, starting, since 0.00960164s), standbys: x 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:49:23.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: Manager daemon y is now available 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:23.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:23.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:23.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: Active manager daemon y restarted 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: Activating manager daemon y 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: mgrmap e19: y(active, starting, since 0.00960164s), standbys: x 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T03:49:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: Manager daemon y is now available 2024-03-26T03:49:24.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:24.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T03:49:24.526 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:49:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:49:24.545 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:49:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:49:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:24 smithi086 ceph-mon[32965]: [26/Mar/2024:03:49:23] ENGINE Bus STARTING 2024-03-26T03:49:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:24 smithi086 ceph-mon[32965]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T03:49:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:24 smithi086 ceph-mon[32965]: mgrmap e20: y(active, since 1.01327s), standbys: x 2024-03-26T03:49:24.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[37501]: [26/Mar/2024:03:49:23] ENGINE Bus STARTING 2024-03-26T03:49:24.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[37501]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T03:49:24.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[37501]: mgrmap e20: y(active, since 1.01327s), standbys: x 2024-03-26T03:49:24.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[29446]: [26/Mar/2024:03:49:23] ENGINE Bus STARTING 2024-03-26T03:49:24.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[29446]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T03:49:24.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:24 smithi006 ceph-mon[29446]: mgrmap e20: y(active, since 1.01327s), standbys: x 2024-03-26T03:49:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:25 smithi086 ceph-mon[32965]: [26/Mar/2024:03:49:23] ENGINE Serving on https://172.21.15.6:7150 2024-03-26T03:49:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:25 smithi086 ceph-mon[32965]: [26/Mar/2024:03:49:23] ENGINE Serving on http://172.21.15.6:8765 2024-03-26T03:49:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:25 smithi086 ceph-mon[32965]: [26/Mar/2024:03:49:23] ENGINE Bus STARTED 2024-03-26T03:49:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:25 smithi086 ceph-mon[32965]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.136 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[37501]: [26/Mar/2024:03:49:23] ENGINE Serving on https://172.21.15.6:7150 2024-03-26T03:49:26.136 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[37501]: [26/Mar/2024:03:49:23] ENGINE Serving on http://172.21.15.6:8765 2024-03-26T03:49:26.136 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[37501]: [26/Mar/2024:03:49:23] ENGINE Bus STARTED 2024-03-26T03:49:26.136 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:26.136 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[37501]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.137 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[29446]: [26/Mar/2024:03:49:23] ENGINE Serving on https://172.21.15.6:7150 2024-03-26T03:49:26.137 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[29446]: [26/Mar/2024:03:49:23] ENGINE Serving on http://172.21.15.6:8765 2024-03-26T03:49:26.137 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[29446]: [26/Mar/2024:03:49:23] ENGINE Bus STARTED 2024-03-26T03:49:26.137 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:26.137 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:25 smithi006 ceph-mon[29446]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.753 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:26 smithi006 ceph-mon[37501]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.753 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:26 smithi006 ceph-mon[37501]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T03:49:26.753 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:26 smithi006 ceph-mon[29446]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.753 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:26 smithi006 ceph-mon[29446]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T03:49:26.946 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:26 smithi086 ceph-mon[32965]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:26.946 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:26 smithi086 ceph-mon[32965]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T03:49:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:28 smithi086 ceph-mon[32965]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:28 smithi086 ceph-mon[32965]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T03:49:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:28 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:28 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[37501]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[37501]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[29446]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[29446]: mgrmap e22: y(active, since 4s), standbys: x 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:28 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 3760M 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 3760M 2024-03-26T03:49:30.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:30 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:30.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 3760M 2024-03-26T03:49:30.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:30 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:32.310 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:32 smithi006 ceph-mon[37501]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T03:49:32.311 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:32 smithi006 ceph-mon[29446]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T03:49:32.340 INFO:teuthology.orchestra.run.smithi006.stdout:253403070477 2024-03-26T03:49:32.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T03:49:32.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:32 smithi086 ceph-mon[32965]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T03:49:32.965 INFO:teuthology.orchestra.run.smithi006.stdout:137438953500 2024-03-26T03:49:32.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T03:49:34.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:34 smithi086 ceph-mon[32965]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T03:49:34.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:34 smithi086 ceph-mon[32965]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T03:49:34.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:49:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:49:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:34 smithi006 ceph-mon[37501]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T03:49:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:34 smithi006 ceph-mon[37501]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T03:49:34.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:49:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:49:34.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:34 smithi006 ceph-mon[29446]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T03:49:34.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:34 smithi006 ceph-mon[29446]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T03:49:34.712 INFO:teuthology.orchestra.run.smithi006.stdout:55834574891 2024-03-26T03:49:34.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T03:49:35.508 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:35.508 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:35.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:36.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:36 smithi006 ceph-mon[29446]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:36.331 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:36 smithi006 ceph-mon[37501]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:36.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:36 smithi086 ceph-mon[32965]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:37.098 INFO:teuthology.orchestra.run.smithi006.stdout:107374182433 2024-03-26T03:49:37.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T03:49:38.308 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:38 smithi006 ceph-mon[37501]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:38.308 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:38 smithi006 ceph-mon[29446]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:38.309 INFO:teuthology.orchestra.run.smithi006.stdout:77309411368 2024-03-26T03:49:38.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T03:49:38.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:38 smithi086 ceph-mon[32965]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 27 KiB/s rd, 0 B/s wr, 15 op/s 2024-03-26T03:49:38.910 INFO:teuthology.orchestra.run.smithi006.stdout:214748364820 2024-03-26T03:49:38.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T03:49:39.488 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:39.489 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:39 smithi006 ceph-mon[37501]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T03:49:39.489 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:39.489 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:39 smithi006 ceph-mon[29446]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T03:49:39.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:39.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:39 smithi086 ceph-mon[32965]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T03:49:40.102 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:40.578 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:40 smithi006 ceph-mon[37501]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:40.578 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:40 smithi006 ceph-mon[29446]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:40.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:40 smithi086 ceph-mon[32965]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:40.604 INFO:teuthology.orchestra.run.smithi006.stdout:176093659159 2024-03-26T03:49:40.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T03:49:40.654 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:40.692 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:40.693 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:41.076 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:41.095 INFO:teuthology.orchestra.run.smithi006.stdout:34359738419 2024-03-26T03:49:41.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T03:49:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:42 smithi006 ceph-mon[29446]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:42.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:42 smithi006 ceph-mon[37501]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:42.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:42 smithi086 ceph-mon[32965]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:49:44.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:44 smithi006 ceph-mon[29446]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T03:49:44.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:44 smithi006 ceph-mon[29446]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T03:49:44.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:44 smithi006 ceph-mon[37501]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T03:49:44.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:44 smithi006 ceph-mon[37501]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T03:49:44.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:44 smithi086 ceph-mon[32965]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 907 B/s rd, 0 op/s 2024-03-26T03:49:44.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:44 smithi086 ceph-mon[32965]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T03:49:44.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:49:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:49:44.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:49:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:49:45.376 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:45.376 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:45.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:46.349 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:46.387 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:46.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:46 smithi006 ceph-mon[29446]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:46.564 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:46 smithi006 ceph-mon[37501]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:46.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:46 smithi086 ceph-mon[32965]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:47 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:47 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:47 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:47 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T03:49:47.590 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T03:49:47.591 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:47 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[37501]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[37501]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[29446]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:48 smithi006 ceph-mon[29446]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:49:48.460 INFO:teuthology.orchestra.run.smithi006.stdout:137438953503 2024-03-26T03:49:48.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:48 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:48.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:48 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:48.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:48 smithi086 ceph-mon[32965]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:48.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:48 smithi086 ceph-mon[32965]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T03:49:49.299 INFO:teuthology.orchestra.run.smithi006.stdout:253403070480 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/853558233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:49:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4256103102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/853558233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:49:49.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:49 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4256103102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:49:49.465 INFO:teuthology.orchestra.run.smithi006.stdout:77309411370 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/853558233' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T03:49:49.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:49 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4256103102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: 3.9 deep-scrub starts 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: 3.9 deep-scrub ok 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:49:50.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3885539465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: 3.9 deep-scrub starts 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: 3.9 deep-scrub ok 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:49:50.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3885539465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: 3.9 deep-scrub starts 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: 3.9 deep-scrub ok 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:49:50.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3885539465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T03:49:51.482 INFO:teuthology.orchestra.run.smithi006.stdout:107374182437 2024-03-26T03:49:52.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070477 got 253403070480 for osd.7 2024-03-26T03:49:52.148 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T03:49:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T03:49:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T03:49:52.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 4.0 deep-scrub starts 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 4.0 deep-scrub ok 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2062898093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/970421669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T03:49:52.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 4.0 deep-scrub starts 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 4.0 deep-scrub ok 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2062898093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/970421669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.393 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:52.394 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 4.0 deep-scrub starts 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 4.0 deep-scrub ok 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2062898093' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/970421669' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:49:52.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:49:52.664 INFO:teuthology.orchestra.run.smithi006.stdout:214748364823 2024-03-26T03:49:52.896 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182433 got 107374182437 for osd.3 2024-03-26T03:49:52.897 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:53.318 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411368 got 77309411370 for osd.2 2024-03-26T03:49:53.318 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:53.418 INFO:teuthology.orchestra.run.smithi006.stdout:55834574896 2024-03-26T03:49:53.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: 3.10 deep-scrub starts 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: 3.10 deep-scrub ok 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1608568770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: 3.10 deep-scrub starts 2024-03-26T03:49:53.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: 3.10 deep-scrub ok 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1608568770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T03:49:53.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 3761M 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi006:/etc/ceph/ceph.conf 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi086:/etc/ceph/ceph.conf 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: 3.10 deep-scrub starts 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: 3.10 deep-scrub ok 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1608568770' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi006:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:49:53.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:49:53.998 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574891 got 55834574896 for osd.1 2024-03-26T03:49:53.998 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:54.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: Deploying daemon alertmanager.a on smithi006 2024-03-26T03:49:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:54 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:49:54.615 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.615 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: Deploying daemon alertmanager.a on smithi006 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:49:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: Updating smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: Updating smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T03:49:54.616 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:49:54.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:49:54.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: Deploying daemon alertmanager.a on smithi006 2024-03-26T03:49:54.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:54 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:49:54.665 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364820 got 214748364823 for osd.6 2024-03-26T03:49:54.666 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:54.952 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:49:54.985 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953500 got 137438953503 for osd.4 2024-03-26T03:49:54.985 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:55.186 INFO:teuthology.orchestra.run.smithi006.stdout:34359738421 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:49:55.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3007781927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:49:55.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:55.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3007781927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:49:55.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:49:55.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:49:55.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T03:49:55.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3007781927' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T03:49:56.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T03:49:56.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:56 smithi086 ceph-mon[32965]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[29446]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:49:56.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T03:49:56.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:56 smithi006 ceph-mon[37501]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:49:58.111 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738419 got 34359738421 for osd.0 2024-03-26T03:49:58.111 DEBUG:teuthology.parallel:result is None 2024-03-26T03:49:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.c deep-scrub starts 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.c deep-scrub ok 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T03:49:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:58 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T03:49:58.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T03:49:58.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.c deep-scrub starts 2024-03-26T03:49:58.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.c deep-scrub ok 2024-03-26T03:49:58.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:58.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T03:49:58.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T03:49:58.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T03:49:58.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.c deep-scrub starts 2024-03-26T03:49:58.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.c deep-scrub ok 2024-03-26T03:49:58.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:49:58.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T03:49:58.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:58 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T03:49:59.336 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T03:49:59.337 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T03:49:59.462 INFO:teuthology.orchestra.run.smithi006.stdout:176093659163 2024-03-26T03:49:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:59 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T03:49:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:59 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T03:49:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:59 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T03:49:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:59 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T03:49:59.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:49:59 smithi086 ceph-mon[32965]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:49:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T03:49:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T03:49:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[29446]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:49:59.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T03:49:59.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T03:49:59.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T03:49:59.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T03:49:59.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:49:59 smithi006 ceph-mon[37501]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T03:50:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 5.a deep-scrub starts 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: 5.a deep-scrub ok 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2400558269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[37501]: overall HEALTH_OK 2024-03-26T03:50:00.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 5.a deep-scrub starts 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: 5.a deep-scrub ok 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2400558269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:50:00.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:00 smithi006 ceph-mon[29446]: overall HEALTH_OK 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 5.a deep-scrub starts 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: 5.a deep-scrub ok 2024-03-26T03:50:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2400558269' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T03:50:00.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:00 smithi086 ceph-mon[32965]: overall HEALTH_OK 2024-03-26T03:50:01.433 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:01 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T03:50:01.433 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:01 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T03:50:01.433 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:01 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T03:50:01.433 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:01 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T03:50:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:01 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T03:50:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:01 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T03:50:02.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[29446]: 5.4 deep-scrub starts 2024-03-26T03:50:02.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[29446]: 5.4 deep-scrub ok 2024-03-26T03:50:02.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[29446]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:02.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T03:50:02.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[37501]: 5.4 deep-scrub starts 2024-03-26T03:50:02.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[37501]: 5.4 deep-scrub ok 2024-03-26T03:50:02.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[37501]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:02.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:02 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T03:50:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:02 smithi086 ceph-mon[32965]: 5.4 deep-scrub starts 2024-03-26T03:50:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:02 smithi086 ceph-mon[32965]: 5.4 deep-scrub ok 2024-03-26T03:50:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:02 smithi086 ceph-mon[32965]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:02 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T03:50:02.851 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659159 got 176093659163 for osd.5 2024-03-26T03:50:02.851 DEBUG:teuthology.parallel:result is None 2024-03-26T03:50:02.851 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T03:50:02.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:50:03.140 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:02 smithi006 systemd[1]: Starting Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 5.f deep-scrub starts 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 5.f deep-scrub ok 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T03:50:03.453 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 5.f deep-scrub starts 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 5.f deep-scrub ok 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T03:50:03.454 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T03:50:03.454 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 podman[71433]: 2024-03-26 03:50:03.044246897 +0000 UTC m=+0.017581775 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T03:50:03.454 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 podman[71433]: 2024-03-26 03:50:03.1846606 +0000 UTC m=+0.157995457 volume create a744f8f103b7829be823a909a79d7855aa844e91855abc522c2029f02aa6365d 2024-03-26T03:50:03.454 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 podman[71433]: 2024-03-26 03:50:03.26013295 +0000 UTC m=+0.233467848 container create 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:03.776 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:03.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T03:50:03.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T03:50:03.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T03:50:03.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 5.f deep-scrub starts 2024-03-26T03:50:03.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 5.f deep-scrub ok 2024-03-26T03:50:03.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T03:50:03.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:03 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T03:50:03.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 podman[71433]: 2024-03-26 03:50:03.554250342 +0000 UTC m=+0.527585198 container init 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:03.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 podman[71433]: 2024-03-26 03:50:03.556917176 +0000 UTC m=+0.530252032 container start 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:03.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.568Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T03:50:03.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.568Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T03:50:03.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.570Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.6 port=9094 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.571Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.594Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.595Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.597Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:03.597Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 bash[71433]: 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 2024-03-26T03:50:03.891 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:03 smithi006 systemd[1]: Started Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:04.200 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:03] ENGINE Bus STOPPING 2024-03-26T03:50:04.200 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:03] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:50:04.200 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:03] ENGINE Bus STOPPED 2024-03-26T03:50:04.201 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:03 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:03] ENGINE Bus STARTING 2024-03-26T03:50:04.201 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:04] ENGINE Serving on http://:::9283 2024-03-26T03:50:04.201 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:04] ENGINE Bus STARTED 2024-03-26T03:50:04.553 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:50:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.1c deep-scrub starts 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.1c deep-scrub ok 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T03:50:04.554 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:04.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.1c deep-scrub starts 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.1c deep-scrub ok 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:04.557 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.1c deep-scrub starts 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.1c deep-scrub ok 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 578 B/s rd, 0 op/s 2024-03-26T03:50:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:04.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T03:50:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T03:50:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[37501]: Deploying daemon grafana.a on smithi086 2024-03-26T03:50:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T03:50:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T03:50:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[29446]: Deploying daemon grafana.a on smithi086 2024-03-26T03:50:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:05.589 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:50:05.592 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:50:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:05 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T03:50:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:05 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T03:50:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:05 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T03:50:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:05 smithi086 ceph-mon[32965]: Deploying daemon grafana.a on smithi086 2024-03-26T03:50:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:05.885 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:05 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:05.572Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000791168s 2024-03-26T03:50:06.249 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-26T03:50:05.163415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1295,"num_read_kb":1109,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":807,"ondisk_log_size":807,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244896,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716320,"statfs":{"total":767960285184,"available":767709511680,"internally_reserved":0,"allocated":8155136,"data_stored":4523416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003208"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159349+0000","last_change":"2024-03-26T03:48:50.035293+0000","last_active":"2024-03-26T03:49:59.159349+0000","last_peered":"2024-03-26T03:49:59.159349+0000","last_clean":"2024-03-26T03:49:59.159349+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:49:59.159349+0000","last_undegraded":"2024-03-26T03:49:59.159349+0000","last_fullsized":"2024-03-26T03:49:59.159349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.235116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328558+0000","last_change":"2024-03-26T03:48:48.025490+0000","last_active":"2024-03-26T03:49:59.328558+0000","last_peered":"2024-03-26T03:49:59.328558+0000","last_clean":"2024-03-26T03:49:59.328558+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:49:59.328558+0000","last_undegraded":"2024-03-26T03:49:59.328558+0000","last_fullsized":"2024-03-26T03:49:59.328558+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.103757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161626+0000","last_change":"2024-03-26T03:49:49.005275+0000","last_active":"2024-03-26T03:49:59.161626+0000","last_peered":"2024-03-26T03:49:59.161626+0000","last_clean":"2024-03-26T03:49:59.161626+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:49:59.161626+0000","last_undegraded":"2024-03-26T03:49:59.161626+0000","last_fullsized":"2024-03-26T03:49:59.161626+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:49.005129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:49.005129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.159629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159200+0000","last_change":"2024-03-26T03:48:46.018459+0000","last_active":"2024-03-26T03:49:59.159200+0000","last_peered":"2024-03-26T03:49:59.159200+0000","last_clean":"2024-03-26T03:49:59.159200+0000","last_became_active":"2024-03-26T03:48:46.018079+0000","last_became_peered":"2024-03-26T03:48:46.018079+0000","last_unstale":"2024-03-26T03:49:59.159200+0000","last_undegraded":"2024-03-26T03:49:59.159200+0000","last_fullsized":"2024-03-26T03:49:59.159200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.894463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253610+0000","last_change":"2024-03-26T03:48:50.262953+0000","last_active":"2024-03-26T03:49:59.253610+0000","last_peered":"2024-03-26T03:49:59.253610+0000","last_clean":"2024-03-26T03:49:59.253610+0000","last_became_active":"2024-03-26T03:48:50.262548+0000","last_became_peered":"2024-03-26T03:48:50.262548+0000","last_unstale":"2024-03-26T03:49:59.253610+0000","last_undegraded":"2024-03-26T03:49:59.253610+0000","last_fullsized":"2024-03-26T03:49:59.253610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159206+0000","last_change":"2024-03-26T03:48:48.025511+0000","last_active":"2024-03-26T03:49:59.159206+0000","last_peered":"2024-03-26T03:49:59.159206+0000","last_clean":"2024-03-26T03:49:59.159206+0000","last_became_active":"2024-03-26T03:48:48.024923+0000","last_became_peered":"2024-03-26T03:48:48.024923+0000","last_unstale":"2024-03-26T03:49:59.159206+0000","last_undegraded":"2024-03-26T03:49:59.159206+0000","last_fullsized":"2024-03-26T03:49:59.159206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.464377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253931+0000","last_change":"2024-03-26T03:48:44.017642+0000","last_active":"2024-03-26T03:49:59.253931+0000","last_peered":"2024-03-26T03:49:59.253931+0000","last_clean":"2024-03-26T03:49:59.253931+0000","last_became_active":"2024-03-26T03:48:44.017341+0000","last_became_peered":"2024-03-26T03:48:44.017341+0000","last_unstale":"2024-03-26T03:49:59.253931+0000","last_undegraded":"2024-03-26T03:49:59.253931+0000","last_fullsized":"2024-03-26T03:49:59.253931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.584504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253773+0000","last_change":"2024-03-26T03:48:46.020802+0000","last_active":"2024-03-26T03:49:59.253773+0000","last_peered":"2024-03-26T03:49:59.253773+0000","last_clean":"2024-03-26T03:49:59.253773+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:49:59.253773+0000","last_undegraded":"2024-03-26T03:49:59.253773+0000","last_fullsized":"2024-03-26T03:49:59.253773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160055+0000","last_change":"2024-03-26T03:48:50.037848+0000","last_active":"2024-03-26T03:49:59.160055+0000","last_peered":"2024-03-26T03:49:59.160055+0000","last_clean":"2024-03-26T03:49:59.160055+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:49:59.160055+0000","last_undegraded":"2024-03-26T03:49:59.160055+0000","last_fullsized":"2024-03-26T03:49:59.160055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.299751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329686+0000","last_change":"2024-03-26T03:49:59.135484+0000","last_active":"2024-03-26T03:49:59.329686+0000","last_peered":"2024-03-26T03:49:59.329686+0000","last_clean":"2024-03-26T03:49:59.329686+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:49:59.329686+0000","last_undegraded":"2024-03-26T03:49:59.329686+0000","last_fullsized":"2024-03-26T03:49:59.329686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.135403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.135403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.439280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329947+0000","last_change":"2024-03-26T03:48:44.015557+0000","last_active":"2024-03-26T03:49:59.329947+0000","last_peered":"2024-03-26T03:49:59.329947+0000","last_clean":"2024-03-26T03:49:59.329947+0000","last_became_active":"2024-03-26T03:48:44.015277+0000","last_became_peered":"2024-03-26T03:48:44.015277+0000","last_unstale":"2024-03-26T03:49:59.329947+0000","last_undegraded":"2024-03-26T03:49:59.329947+0000","last_fullsized":"2024-03-26T03:49:59.329947+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159518+0000","last_change":"2024-03-26T03:49:54.881008+0000","last_active":"2024-03-26T03:49:59.159518+0000","last_peered":"2024-03-26T03:49:59.159518+0000","last_clean":"2024-03-26T03:49:59.159518+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:49:59.159518+0000","last_undegraded":"2024-03-26T03:49:59.159518+0000","last_fullsized":"2024-03-26T03:49:59.159518+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:49:54.880892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.880892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.130490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329449+0000","last_change":"2024-03-26T03:48:50.039963+0000","last_active":"2024-03-26T03:49:59.329449+0000","last_peered":"2024-03-26T03:49:59.329449+0000","last_clean":"2024-03-26T03:49:59.329449+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:49:59.329449+0000","last_undegraded":"2024-03-26T03:49:59.329449+0000","last_fullsized":"2024-03-26T03:49:59.329449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.621578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328906+0000","last_change":"2024-03-26T03:48:48.032017+0000","last_active":"2024-03-26T03:49:59.328906+0000","last_peered":"2024-03-26T03:49:59.328906+0000","last_clean":"2024-03-26T03:49:59.328906+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:49:59.328906+0000","last_undegraded":"2024-03-26T03:49:59.328906+0000","last_fullsized":"2024-03-26T03:49:59.328906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.912589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329810+0000","last_change":"2024-03-26T03:48:44.013770+0000","last_active":"2024-03-26T03:49:59.329810+0000","last_peered":"2024-03-26T03:49:59.329810+0000","last_clean":"2024-03-26T03:49:59.329810+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:49:59.329810+0000","last_undegraded":"2024-03-26T03:49:59.329810+0000","last_fullsized":"2024-03-26T03:49:59.329810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.115985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'11","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159577+0000","last_change":"2024-03-26T03:48:46.023941+0000","last_active":"2024-03-26T03:49:59.159577+0000","last_peered":"2024-03-26T03:49:59.159577+0000","last_clean":"2024-03-26T03:49:59.159577+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:49:59.159577+0000","last_undegraded":"2024-03-26T03:49:59.159577+0000","last_fullsized":"2024-03-26T03:49:59.159577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.340361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160262+0000","last_change":"2024-03-26T03:48:50.261495+0000","last_active":"2024-03-26T03:49:59.160262+0000","last_peered":"2024-03-26T03:49:59.160262+0000","last_clean":"2024-03-26T03:49:59.160262+0000","last_became_active":"2024-03-26T03:48:50.260266+0000","last_became_peered":"2024-03-26T03:48:50.260266+0000","last_unstale":"2024-03-26T03:49:59.160262+0000","last_undegraded":"2024-03-26T03:49:59.160262+0000","last_fullsized":"2024-03-26T03:49:59.160262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.926198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329321+0000","last_change":"2024-03-26T03:48:48.025562+0000","last_active":"2024-03-26T03:49:59.329321+0000","last_peered":"2024-03-26T03:49:59.329321+0000","last_clean":"2024-03-26T03:49:59.329321+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:49:59.329321+0000","last_undegraded":"2024-03-26T03:49:59.329321+0000","last_fullsized":"2024-03-26T03:49:59.329321+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.783166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161082+0000","last_change":"2024-03-26T03:49:58.053896+0000","last_active":"2024-03-26T03:49:59.161082+0000","last_peered":"2024-03-26T03:49:59.161082+0000","last_clean":"2024-03-26T03:49:59.161082+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:49:59.161082+0000","last_undegraded":"2024-03-26T03:49:59.161082+0000","last_fullsized":"2024-03-26T03:49:59.161082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.053811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.053811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.133041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160189+0000","last_change":"2024-03-26T03:49:53.266632+0000","last_active":"2024-03-26T03:49:59.160189+0000","last_peered":"2024-03-26T03:49:59.160189+0000","last_clean":"2024-03-26T03:49:59.160189+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:49:59.160189+0000","last_undegraded":"2024-03-26T03:49:59.160189+0000","last_fullsized":"2024-03-26T03:49:59.160189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:49:53.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:53.266536+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.703552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254274+0000","last_change":"2024-03-26T03:48:50.261258+0000","last_active":"2024-03-26T03:49:59.254274+0000","last_peered":"2024-03-26T03:49:59.254274+0000","last_clean":"2024-03-26T03:49:59.254274+0000","last_became_active":"2024-03-26T03:48:50.260688+0000","last_became_peered":"2024-03-26T03:48:50.260688+0000","last_unstale":"2024-03-26T03:49:59.254274+0000","last_undegraded":"2024-03-26T03:49:59.254274+0000","last_fullsized":"2024-03-26T03:49:59.254274+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159768+0000","last_change":"2024-03-26T03:48:48.030756+0000","last_active":"2024-03-26T03:49:59.159768+0000","last_peered":"2024-03-26T03:49:59.159768+0000","last_clean":"2024-03-26T03:49:59.159768+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:49:59.159768+0000","last_undegraded":"2024-03-26T03:49:59.159768+0000","last_fullsized":"2024-03-26T03:49:59.159768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:08.338651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159541+0000","last_change":"2024-03-26T03:49:58.872361+0000","last_active":"2024-03-26T03:49:59.159541+0000","last_peered":"2024-03-26T03:49:59.159541+0000","last_clean":"2024-03-26T03:49:59.159541+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:49:59.159541+0000","last_undegraded":"2024-03-26T03:49:59.159541+0000","last_fullsized":"2024-03-26T03:49:59.159541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.872258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.872258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.889376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253731+0000","last_change":"2024-03-26T03:48:46.027100+0000","last_active":"2024-03-26T03:49:59.253731+0000","last_peered":"2024-03-26T03:49:59.253731+0000","last_clean":"2024-03-26T03:49:59.253731+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:49:59.253731+0000","last_undegraded":"2024-03-26T03:49:59.253731+0000","last_fullsized":"2024-03-26T03:49:59.253731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.740617+0000","last_change":"2024-03-26T03:50:02.740650+0000","last_active":"2024-03-26T03:50:02.740617+0000","last_peered":"2024-03-26T03:50:02.740617+0000","last_clean":"2024-03-26T03:50:02.740617+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:50:02.740617+0000","last_undegraded":"2024-03-26T03:50:02.740617+0000","last_fullsized":"2024-03-26T03:50:02.740617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.740545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.965068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330438+0000","last_change":"2024-03-26T03:48:48.022793+0000","last_active":"2024-03-26T03:49:59.330438+0000","last_peered":"2024-03-26T03:49:59.330438+0000","last_clean":"2024-03-26T03:49:59.330438+0000","last_became_active":"2024-03-26T03:48:48.022480+0000","last_became_peered":"2024-03-26T03:48:48.022480+0000","last_unstale":"2024-03-26T03:49:59.330438+0000","last_undegraded":"2024-03-26T03:49:59.330438+0000","last_fullsized":"2024-03-26T03:49:59.330438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:14.690568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328667+0000","last_change":"2024-03-26T03:48:44.018169+0000","last_active":"2024-03-26T03:49:59.328667+0000","last_peered":"2024-03-26T03:49:59.328667+0000","last_clean":"2024-03-26T03:49:59.328667+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:49:59.328667+0000","last_undegraded":"2024-03-26T03:49:59.328667+0000","last_fullsized":"2024-03-26T03:49:59.328667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.658563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161712+0000","last_change":"2024-03-26T03:48:46.210691+0000","last_active":"2024-03-26T03:49:59.161712+0000","last_peered":"2024-03-26T03:49:59.161712+0000","last_clean":"2024-03-26T03:49:59.161712+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:49:59.161712+0000","last_undegraded":"2024-03-26T03:49:59.161712+0000","last_fullsized":"2024-03-26T03:49:59.161712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.585779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329047+0000","last_change":"2024-03-26T03:48:50.035147+0000","last_active":"2024-03-26T03:49:59.329047+0000","last_peered":"2024-03-26T03:49:59.329047+0000","last_clean":"2024-03-26T03:49:59.329047+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:49:59.329047+0000","last_undegraded":"2024-03-26T03:49:59.329047+0000","last_fullsized":"2024-03-26T03:49:59.329047+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.410878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161704+0000","last_change":"2024-03-26T03:48:48.020953+0000","last_active":"2024-03-26T03:49:59.161704+0000","last_peered":"2024-03-26T03:49:59.161704+0000","last_clean":"2024-03-26T03:49:59.161704+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:49:59.161704+0000","last_undegraded":"2024-03-26T03:49:59.161704+0000","last_fullsized":"2024-03-26T03:49:59.161704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253266+0000","last_change":"2024-03-26T03:48:44.016177+0000","last_active":"2024-03-26T03:49:59.253266+0000","last_peered":"2024-03-26T03:49:59.253266+0000","last_clean":"2024-03-26T03:49:59.253266+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:49:59.253266+0000","last_undegraded":"2024-03-26T03:49:59.253266+0000","last_fullsized":"2024-03-26T03:49:59.253266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.493384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330349+0000","last_change":"2024-03-26T03:49:58.491993+0000","last_active":"2024-03-26T03:49:59.330349+0000","last_peered":"2024-03-26T03:49:59.330349+0000","last_clean":"2024-03-26T03:49:59.330349+0000","last_became_active":"2024-03-26T03:48:46.023417+0000","last_became_peered":"2024-03-26T03:48:46.023417+0000","last_unstale":"2024-03-26T03:49:59.330349+0000","last_undegraded":"2024-03-26T03:49:59.330349+0000","last_fullsized":"2024-03-26T03:49:59.330349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:49:58.491875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.491875+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.158055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328787+0000","last_change":"2024-03-26T03:48:50.029982+0000","last_active":"2024-03-26T03:49:59.328787+0000","last_peered":"2024-03-26T03:49:59.328787+0000","last_clean":"2024-03-26T03:49:59.328787+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:49:59.328787+0000","last_undegraded":"2024-03-26T03:49:59.328787+0000","last_fullsized":"2024-03-26T03:49:59.328787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.788600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328760+0000","last_change":"2024-03-26T03:48:48.024958+0000","last_active":"2024-03-26T03:49:59.328760+0000","last_peered":"2024-03-26T03:49:59.328760+0000","last_clean":"2024-03-26T03:49:59.328760+0000","last_became_active":"2024-03-26T03:48:48.022532+0000","last_became_peered":"2024-03-26T03:48:48.022532+0000","last_unstale":"2024-03-26T03:49:59.328760+0000","last_undegraded":"2024-03-26T03:49:59.328760+0000","last_fullsized":"2024-03-26T03:49:59.328760+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.189825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161609+0000","last_change":"2024-03-26T03:48:44.012132+0000","last_active":"2024-03-26T03:49:59.161609+0000","last_peered":"2024-03-26T03:49:59.161609+0000","last_clean":"2024-03-26T03:49:59.161609+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:49:59.161609+0000","last_undegraded":"2024-03-26T03:49:59.161609+0000","last_fullsized":"2024-03-26T03:49:59.161609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253280+0000","last_change":"2024-03-26T03:48:46.027982+0000","last_active":"2024-03-26T03:49:59.253280+0000","last_peered":"2024-03-26T03:49:59.253280+0000","last_clean":"2024-03-26T03:49:59.253280+0000","last_became_active":"2024-03-26T03:48:46.027671+0000","last_became_peered":"2024-03-26T03:48:46.027671+0000","last_unstale":"2024-03-26T03:49:59.253280+0000","last_undegraded":"2024-03-26T03:49:59.253280+0000","last_fullsized":"2024-03-26T03:49:59.253280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253482+0000","last_change":"2024-03-26T03:48:50.261731+0000","last_active":"2024-03-26T03:49:59.253482+0000","last_peered":"2024-03-26T03:49:59.253482+0000","last_clean":"2024-03-26T03:49:59.253482+0000","last_became_active":"2024-03-26T03:48:50.261396+0000","last_became_peered":"2024-03-26T03:48:50.261396+0000","last_unstale":"2024-03-26T03:49:59.253482+0000","last_undegraded":"2024-03-26T03:49:59.253482+0000","last_fullsized":"2024-03-26T03:49:59.253482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329302+0000","last_change":"2024-03-26T03:48:48.029980+0000","last_active":"2024-03-26T03:49:59.329302+0000","last_peered":"2024-03-26T03:49:59.329302+0000","last_clean":"2024-03-26T03:49:59.329302+0000","last_became_active":"2024-03-26T03:48:48.029501+0000","last_became_peered":"2024-03-26T03:48:48.029501+0000","last_unstale":"2024-03-26T03:49:59.329302+0000","last_undegraded":"2024-03-26T03:49:59.329302+0000","last_fullsized":"2024-03-26T03:49:59.329302+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.171082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330124+0000","last_change":"2024-03-26T03:48:44.019133+0000","last_active":"2024-03-26T03:49:59.330124+0000","last_peered":"2024-03-26T03:49:59.330124+0000","last_clean":"2024-03-26T03:49:59.330124+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:49:59.330124+0000","last_undegraded":"2024-03-26T03:49:59.330124+0000","last_fullsized":"2024-03-26T03:49:59.330124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'17","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253661+0000","last_change":"2024-03-26T03:48:46.209729+0000","last_active":"2024-03-26T03:49:59.253661+0000","last_peered":"2024-03-26T03:49:59.253661+0000","last_clean":"2024-03-26T03:49:59.253661+0000","last_became_active":"2024-03-26T03:48:46.209381+0000","last_became_peered":"2024-03-26T03:48:46.209381+0000","last_unstale":"2024-03-26T03:49:59.253661+0000","last_undegraded":"2024-03-26T03:49:59.253661+0000","last_fullsized":"2024-03-26T03:49:59.253661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159720+0000","last_change":"2024-03-26T03:48:50.037564+0000","last_active":"2024-03-26T03:49:59.159720+0000","last_peered":"2024-03-26T03:49:59.159720+0000","last_clean":"2024-03-26T03:49:59.159720+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:49:59.159720+0000","last_undegraded":"2024-03-26T03:49:59.159720+0000","last_fullsized":"2024-03-26T03:49:59.159720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.630721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253770+0000","last_change":"2024-03-26T03:49:56.980611+0000","last_active":"2024-03-26T03:49:59.253770+0000","last_peered":"2024-03-26T03:49:59.253770+0000","last_clean":"2024-03-26T03:49:59.253770+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:49:59.253770+0000","last_undegraded":"2024-03-26T03:49:59.253770+0000","last_fullsized":"2024-03-26T03:49:59.253770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.980524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.980524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.615228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329754+0000","last_change":"2024-03-26T03:49:50.773333+0000","last_active":"2024-03-26T03:49:59.329754+0000","last_peered":"2024-03-26T03:49:59.329754+0000","last_clean":"2024-03-26T03:49:59.329754+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:49:59.329754+0000","last_undegraded":"2024-03-26T03:49:59.329754+0000","last_fullsized":"2024-03-26T03:49:59.329754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:50.773202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:50.773202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.850607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328926+0000","last_change":"2024-03-26T03:49:51.011344+0000","last_active":"2024-03-26T03:49:59.328926+0000","last_peered":"2024-03-26T03:49:59.328926+0000","last_clean":"2024-03-26T03:49:59.328926+0000","last_became_active":"2024-03-26T03:48:46.017045+0000","last_became_peered":"2024-03-26T03:48:46.017045+0000","last_unstale":"2024-03-26T03:49:59.328926+0000","last_undegraded":"2024-03-26T03:49:59.328926+0000","last_fullsized":"2024-03-26T03:49:59.328926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:49:51.011226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.011226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:04.692152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.986797+0000","last_change":"2024-03-26T03:50:02.986818+0000","last_active":"2024-03-26T03:50:02.986797+0000","last_peered":"2024-03-26T03:50:02.986797+0000","last_clean":"2024-03-26T03:50:02.986797+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:50:02.986797+0000","last_undegraded":"2024-03-26T03:50:02.986797+0000","last_fullsized":"2024-03-26T03:50:02.986797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.986736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.986736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.565698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328830+0000","last_change":"2024-03-26T03:48:48.030582+0000","last_active":"2024-03-26T03:49:59.328830+0000","last_peered":"2024-03-26T03:49:59.328830+0000","last_clean":"2024-03-26T03:49:59.328830+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:49:59.328830+0000","last_undegraded":"2024-03-26T03:49:59.328830+0000","last_fullsized":"2024-03-26T03:49:59.328830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.060061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159891+0000","last_change":"2024-03-26T03:49:56.925827+0000","last_active":"2024-03-26T03:49:59.159891+0000","last_peered":"2024-03-26T03:49:59.159891+0000","last_clean":"2024-03-26T03:49:59.159891+0000","last_became_active":"2024-03-26T03:48:44.013906+0000","last_became_peered":"2024-03-26T03:48:44.013906+0000","last_unstale":"2024-03-26T03:49:59.159891+0000","last_undegraded":"2024-03-26T03:49:59.159891+0000","last_fullsized":"2024-03-26T03:49:59.159891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.925725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.925725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159677+0000","last_change":"2024-03-26T03:48:46.019374+0000","last_active":"2024-03-26T03:49:59.159677+0000","last_peered":"2024-03-26T03:49:59.159677+0000","last_clean":"2024-03-26T03:49:59.159677+0000","last_became_active":"2024-03-26T03:48:46.019043+0000","last_became_peered":"2024-03-26T03:48:46.019043+0000","last_unstale":"2024-03-26T03:49:59.159677+0000","last_undegraded":"2024-03-26T03:49:59.159677+0000","last_fullsized":"2024-03-26T03:49:59.159677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.864202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161210+0000","last_change":"2024-03-26T03:48:50.046189+0000","last_active":"2024-03-26T03:49:59.161210+0000","last_peered":"2024-03-26T03:49:59.161210+0000","last_clean":"2024-03-26T03:49:59.161210+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:49:59.161210+0000","last_undegraded":"2024-03-26T03:49:59.161210+0000","last_fullsized":"2024-03-26T03:49:59.161210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.027196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330076+0000","last_change":"2024-03-26T03:49:55.131855+0000","last_active":"2024-03-26T03:49:59.330076+0000","last_peered":"2024-03-26T03:49:59.330076+0000","last_clean":"2024-03-26T03:49:59.330076+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:49:59.330076+0000","last_undegraded":"2024-03-26T03:49:59.330076+0000","last_fullsized":"2024-03-26T03:49:59.330076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:55.131726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:55.131726+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.670201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330602+0000","last_change":"2024-03-26T03:49:48.735316+0000","last_active":"2024-03-26T03:49:59.330602+0000","last_peered":"2024-03-26T03:49:59.330602+0000","last_clean":"2024-03-26T03:49:59.330602+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:49:59.330602+0000","last_undegraded":"2024-03-26T03:49:59.330602+0000","last_fullsized":"2024-03-26T03:49:59.330602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.735213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.735213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:52.231017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253828+0000","last_change":"2024-03-26T03:48:46.209727+0000","last_active":"2024-03-26T03:49:59.253828+0000","last_peered":"2024-03-26T03:49:59.253828+0000","last_clean":"2024-03-26T03:49:59.253828+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:49:59.253828+0000","last_undegraded":"2024-03-26T03:49:59.253828+0000","last_fullsized":"2024-03-26T03:49:59.253828+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.096233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160356+0000","last_change":"2024-03-26T03:48:50.038644+0000","last_active":"2024-03-26T03:49:59.160356+0000","last_peered":"2024-03-26T03:49:59.160356+0000","last_clean":"2024-03-26T03:49:59.160356+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:49:59.160356+0000","last_undegraded":"2024-03-26T03:49:59.160356+0000","last_fullsized":"2024-03-26T03:49:59.160356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254130+0000","last_change":"2024-03-26T03:48:48.029400+0000","last_active":"2024-03-26T03:49:59.254130+0000","last_peered":"2024-03-26T03:49:59.254130+0000","last_clean":"2024-03-26T03:49:59.254130+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:49:59.254130+0000","last_undegraded":"2024-03-26T03:49:59.254130+0000","last_fullsized":"2024-03-26T03:49:59.254130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160716+0000","last_change":"2024-03-26T03:48:44.015539+0000","last_active":"2024-03-26T03:49:59.160716+0000","last_peered":"2024-03-26T03:49:59.160716+0000","last_clean":"2024-03-26T03:49:59.160716+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:49:59.160716+0000","last_undegraded":"2024-03-26T03:49:59.160716+0000","last_fullsized":"2024-03-26T03:49:59.160716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.181972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330260+0000","last_change":"2024-03-26T03:48:46.210667+0000","last_active":"2024-03-26T03:49:59.330260+0000","last_peered":"2024-03-26T03:49:59.330260+0000","last_clean":"2024-03-26T03:49:59.330260+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:49:59.330260+0000","last_undegraded":"2024-03-26T03:49:59.330260+0000","last_fullsized":"2024-03-26T03:49:59.330260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.284559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160367+0000","last_change":"2024-03-26T03:48:50.033575+0000","last_active":"2024-03-26T03:49:59.160367+0000","last_peered":"2024-03-26T03:49:59.160367+0000","last_clean":"2024-03-26T03:49:59.160367+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:49:59.160367+0000","last_undegraded":"2024-03-26T03:49:59.160367+0000","last_fullsized":"2024-03-26T03:49:59.160367+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.438017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254250+0000","last_change":"2024-03-26T03:48:48.032124+0000","last_active":"2024-03-26T03:49:59.254250+0000","last_peered":"2024-03-26T03:49:59.254250+0000","last_clean":"2024-03-26T03:49:59.254250+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:49:59.254250+0000","last_undegraded":"2024-03-26T03:49:59.254250+0000","last_fullsized":"2024-03-26T03:49:59.254250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329621+0000","last_change":"2024-03-26T03:49:56.749968+0000","last_active":"2024-03-26T03:49:59.329621+0000","last_peered":"2024-03-26T03:49:59.329621+0000","last_clean":"2024-03-26T03:49:59.329621+0000","last_became_active":"2024-03-26T03:48:44.018391+0000","last_became_peered":"2024-03-26T03:48:44.018391+0000","last_unstale":"2024-03-26T03:49:59.329621+0000","last_undegraded":"2024-03-26T03:49:59.329621+0000","last_fullsized":"2024-03-26T03:49:59.329621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.749811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.749811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.793876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161560+0000","last_change":"2024-03-26T03:48:46.209877+0000","last_active":"2024-03-26T03:49:59.161560+0000","last_peered":"2024-03-26T03:49:59.161560+0000","last_clean":"2024-03-26T03:49:59.161560+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:49:59.161560+0000","last_undegraded":"2024-03-26T03:49:59.161560+0000","last_fullsized":"2024-03-26T03:49:59.161560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.267980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329662+0000","last_change":"2024-03-26T03:48:50.263320+0000","last_active":"2024-03-26T03:49:59.329662+0000","last_peered":"2024-03-26T03:49:59.329662+0000","last_clean":"2024-03-26T03:49:59.329662+0000","last_became_active":"2024-03-26T03:48:50.262776+0000","last_became_peered":"2024-03-26T03:48:50.262776+0000","last_unstale":"2024-03-26T03:49:59.329662+0000","last_undegraded":"2024-03-26T03:49:59.329662+0000","last_fullsized":"2024-03-26T03:49:59.329662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.455736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330181+0000","last_change":"2024-03-26T03:48:48.029617+0000","last_active":"2024-03-26T03:49:59.330181+0000","last_peered":"2024-03-26T03:49:59.330181+0000","last_clean":"2024-03-26T03:49:59.330181+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:49:59.330181+0000","last_undegraded":"2024-03-26T03:49:59.330181+0000","last_fullsized":"2024-03-26T03:49:59.330181+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328963+0000","last_change":"2024-03-26T03:49:52.565054+0000","last_active":"2024-03-26T03:49:59.328963+0000","last_peered":"2024-03-26T03:49:59.328963+0000","last_clean":"2024-03-26T03:49:59.328963+0000","last_became_active":"2024-03-26T03:48:44.014783+0000","last_became_peered":"2024-03-26T03:48:44.014783+0000","last_unstale":"2024-03-26T03:49:59.328963+0000","last_undegraded":"2024-03-26T03:49:59.328963+0000","last_fullsized":"2024-03-26T03:49:59.328963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.564972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.484718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.026079+0000","last_change":"2024-03-26T03:50:02.026119+0000","last_active":"2024-03-26T03:50:02.026079+0000","last_peered":"2024-03-26T03:50:02.026079+0000","last_clean":"2024-03-26T03:50:02.026079+0000","last_became_active":"2024-03-26T03:48:46.210541+0000","last_became_peered":"2024-03-26T03:48:46.210541+0000","last_unstale":"2024-03-26T03:50:02.026079+0000","last_undegraded":"2024-03-26T03:50:02.026079+0000","last_fullsized":"2024-03-26T03:50:02.026079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:50:02.025987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.025987+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.118172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329391+0000","last_change":"2024-03-26T03:49:52.162584+0000","last_active":"2024-03-26T03:49:59.329391+0000","last_peered":"2024-03-26T03:49:59.329391+0000","last_clean":"2024-03-26T03:49:59.329391+0000","last_became_active":"2024-03-26T03:48:50.260705+0000","last_became_peered":"2024-03-26T03:48:50.260705+0000","last_unstale":"2024-03-26T03:49:59.329391+0000","last_undegraded":"2024-03-26T03:49:59.329391+0000","last_fullsized":"2024-03-26T03:49:59.329391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.162478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.162478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:09.131410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330603+0000","last_change":"2024-03-26T03:48:44.019372+0000","last_active":"2024-03-26T03:49:59.330603+0000","last_peered":"2024-03-26T03:49:59.330603+0000","last_clean":"2024-03-26T03:49:59.330603+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:49:59.330603+0000","last_undegraded":"2024-03-26T03:49:59.330603+0000","last_fullsized":"2024-03-26T03:49:59.330603+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.992376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328697+0000","last_change":"2024-03-26T03:48:46.210551+0000","last_active":"2024-03-26T03:49:59.328697+0000","last_peered":"2024-03-26T03:49:59.328697+0000","last_clean":"2024-03-26T03:49:59.328697+0000","last_became_active":"2024-03-26T03:48:46.210207+0000","last_became_peered":"2024-03-26T03:48:46.210207+0000","last_unstale":"2024-03-26T03:49:59.328697+0000","last_undegraded":"2024-03-26T03:49:59.328697+0000","last_fullsized":"2024-03-26T03:49:59.328697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.203264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329877+0000","last_change":"2024-03-26T03:49:54.753786+0000","last_active":"2024-03-26T03:49:59.329877+0000","last_peered":"2024-03-26T03:49:59.329877+0000","last_clean":"2024-03-26T03:49:59.329877+0000","last_became_active":"2024-03-26T03:48:48.027440+0000","last_became_peered":"2024-03-26T03:48:48.027440+0000","last_unstale":"2024-03-26T03:49:59.329877+0000","last_undegraded":"2024-03-26T03:49:59.329877+0000","last_fullsized":"2024-03-26T03:49:59.329877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:54.753680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.753680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.334605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253582+0000","last_change":"2024-03-26T03:48:50.043541+0000","last_active":"2024-03-26T03:49:59.253582+0000","last_peered":"2024-03-26T03:49:59.253582+0000","last_clean":"2024-03-26T03:49:59.253582+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:49:59.253582+0000","last_undegraded":"2024-03-26T03:49:59.253582+0000","last_fullsized":"2024-03-26T03:49:59.253582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.195338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330140+0000","last_change":"2024-03-26T03:48:44.017669+0000","last_active":"2024-03-26T03:49:59.330140+0000","last_peered":"2024-03-26T03:49:59.330140+0000","last_clean":"2024-03-26T03:49:59.330140+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:49:59.330140+0000","last_undegraded":"2024-03-26T03:49:59.330140+0000","last_fullsized":"2024-03-26T03:49:59.330140+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.577875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159657+0000","last_change":"2024-03-26T03:48:46.021263+0000","last_active":"2024-03-26T03:49:59.159657+0000","last_peered":"2024-03-26T03:49:59.159657+0000","last_clean":"2024-03-26T03:49:59.159657+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:49:59.159657+0000","last_undegraded":"2024-03-26T03:49:59.159657+0000","last_fullsized":"2024-03-26T03:49:59.159657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159607+0000","last_change":"2024-03-26T03:48:48.019097+0000","last_active":"2024-03-26T03:49:59.159607+0000","last_peered":"2024-03-26T03:49:59.159607+0000","last_clean":"2024-03-26T03:49:59.159607+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:49:59.159607+0000","last_undegraded":"2024-03-26T03:49:59.159607+0000","last_fullsized":"2024-03-26T03:49:59.159607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.081182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328805+0000","last_change":"2024-03-26T03:49:58.807686+0000","last_active":"2024-03-26T03:49:59.328805+0000","last_peered":"2024-03-26T03:49:59.328805+0000","last_clean":"2024-03-26T03:49:59.328805+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:49:59.328805+0000","last_undegraded":"2024-03-26T03:49:59.328805+0000","last_fullsized":"2024-03-26T03:49:59.328805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.807606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.807606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.096069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160010+0000","last_change":"2024-03-26T03:48:48.018417+0000","last_active":"2024-03-26T03:49:59.160010+0000","last_peered":"2024-03-26T03:49:59.160010+0000","last_clean":"2024-03-26T03:49:59.160010+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:49:59.160010+0000","last_undegraded":"2024-03-26T03:49:59.160010+0000","last_fullsized":"2024-03-26T03:49:59.160010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.957546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330604+0000","last_change":"2024-03-26T03:48:44.013581+0000","last_active":"2024-03-26T03:49:59.330604+0000","last_peered":"2024-03-26T03:49:59.330604+0000","last_clean":"2024-03-26T03:49:59.330604+0000","last_became_active":"2024-03-26T03:48:44.013295+0000","last_became_peered":"2024-03-26T03:48:44.013295+0000","last_unstale":"2024-03-26T03:49:59.330604+0000","last_undegraded":"2024-03-26T03:49:59.330604+0000","last_fullsized":"2024-03-26T03:49:59.330604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:04.845697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330072+0000","last_change":"2024-03-26T03:48:46.209925+0000","last_active":"2024-03-26T03:49:59.330072+0000","last_peered":"2024-03-26T03:49:59.330072+0000","last_clean":"2024-03-26T03:49:59.330072+0000","last_became_active":"2024-03-26T03:48:46.209516+0000","last_became_peered":"2024-03-26T03:48:46.209516+0000","last_unstale":"2024-03-26T03:49:59.330072+0000","last_undegraded":"2024-03-26T03:49:59.330072+0000","last_fullsized":"2024-03-26T03:49:59.330072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.460867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159748+0000","last_change":"2024-03-26T03:48:50.040712+0000","last_active":"2024-03-26T03:49:59.159748+0000","last_peered":"2024-03-26T03:49:59.159748+0000","last_clean":"2024-03-26T03:49:59.159748+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:49:59.159748+0000","last_undegraded":"2024-03-26T03:49:59.159748+0000","last_fullsized":"2024-03-26T03:49:59.159748+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.592380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.255830+0000","last_change":"2024-03-26T03:49:59.255850+0000","last_active":"2024-03-26T03:49:59.255830+0000","last_peered":"2024-03-26T03:49:59.255830+0000","last_clean":"2024-03-26T03:49:59.255830+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:49:59.255830+0000","last_undegraded":"2024-03-26T03:49:59.255830+0000","last_fullsized":"2024-03-26T03:49:59.255830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.255770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:16.644194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329906+0000","last_change":"2024-03-26T03:49:57.098936+0000","last_active":"2024-03-26T03:49:59.329906+0000","last_peered":"2024-03-26T03:49:59.329906+0000","last_clean":"2024-03-26T03:49:59.329906+0000","last_became_active":"2024-03-26T03:48:44.016241+0000","last_became_peered":"2024-03-26T03:48:44.016241+0000","last_unstale":"2024-03-26T03:49:59.329906+0000","last_undegraded":"2024-03-26T03:49:59.329906+0000","last_fullsized":"2024-03-26T03:49:59.329906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.098828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.273950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161398+0000","last_change":"2024-03-26T03:48:46.023925+0000","last_active":"2024-03-26T03:49:59.161398+0000","last_peered":"2024-03-26T03:49:59.161398+0000","last_clean":"2024-03-26T03:49:59.161398+0000","last_became_active":"2024-03-26T03:48:46.023348+0000","last_became_peered":"2024-03-26T03:48:46.023348+0000","last_unstale":"2024-03-26T03:49:59.161398+0000","last_undegraded":"2024-03-26T03:49:59.161398+0000","last_fullsized":"2024-03-26T03:49:59.161398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253710+0000","last_change":"2024-03-26T03:48:50.044572+0000","last_active":"2024-03-26T03:49:59.253710+0000","last_peered":"2024-03-26T03:49:59.253710+0000","last_clean":"2024-03-26T03:49:59.253710+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:49:59.253710+0000","last_undegraded":"2024-03-26T03:49:59.253710+0000","last_fullsized":"2024-03-26T03:49:59.253710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.392408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329204+0000","last_change":"2024-03-26T03:48:47.012857+0000","last_active":"2024-03-26T03:49:59.329204+0000","last_peered":"2024-03-26T03:49:59.329204+0000","last_clean":"2024-03-26T03:49:59.329204+0000","last_became_active":"2024-03-26T03:48:43.008452+0000","last_became_peered":"2024-03-26T03:48:43.008452+0000","last_unstale":"2024-03-26T03:49:59.329204+0000","last_undegraded":"2024-03-26T03:49:59.329204+0000","last_fullsized":"2024-03-26T03:49:59.329204+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:48:41.993340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.308748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159473+0000","last_change":"2024-03-26T03:48:44.012922+0000","last_active":"2024-03-26T03:49:59.159473+0000","last_peered":"2024-03-26T03:49:59.159473+0000","last_clean":"2024-03-26T03:49:59.159473+0000","last_became_active":"2024-03-26T03:48:44.012147+0000","last_became_peered":"2024-03-26T03:48:44.012147+0000","last_unstale":"2024-03-26T03:49:59.159473+0000","last_undegraded":"2024-03-26T03:49:59.159473+0000","last_fullsized":"2024-03-26T03:49:59.159473+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.410313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":125,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329182+0000","last_change":"2024-03-26T03:48:46.027084+0000","last_active":"2024-03-26T03:49:59.329182+0000","last_peered":"2024-03-26T03:49:59.329182+0000","last_clean":"2024-03-26T03:49:59.329182+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:49:59.329182+0000","last_undegraded":"2024-03-26T03:49:59.329182+0000","last_fullsized":"2024-03-26T03:49:59.329182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161180+0000","last_change":"2024-03-26T03:48:48.024108+0000","last_active":"2024-03-26T03:49:59.161180+0000","last_peered":"2024-03-26T03:49:59.161180+0000","last_clean":"2024-03-26T03:49:59.161180+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:49:59.161180+0000","last_undegraded":"2024-03-26T03:49:59.161180+0000","last_fullsized":"2024-03-26T03:49:59.161180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330703+0000","last_change":"2024-03-26T03:48:50.262602+0000","last_active":"2024-03-26T03:49:59.330703+0000","last_peered":"2024-03-26T03:49:59.330703+0000","last_clean":"2024-03-26T03:49:59.330703+0000","last_became_active":"2024-03-26T03:48:50.262294+0000","last_became_peered":"2024-03-26T03:48:50.262294+0000","last_unstale":"2024-03-26T03:49:59.330703+0000","last_undegraded":"2024-03-26T03:49:59.330703+0000","last_fullsized":"2024-03-26T03:49:59.330703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.628908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328621+0000","last_change":"2024-03-26T03:48:44.015006+0000","last_active":"2024-03-26T03:49:59.328621+0000","last_peered":"2024-03-26T03:49:59.328621+0000","last_clean":"2024-03-26T03:49:59.328621+0000","last_became_active":"2024-03-26T03:48:44.014239+0000","last_became_peered":"2024-03-26T03:48:44.014239+0000","last_unstale":"2024-03-26T03:49:59.328621+0000","last_undegraded":"2024-03-26T03:49:59.328621+0000","last_fullsized":"2024-03-26T03:49:59.328621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159761+0000","last_change":"2024-03-26T03:49:48.295417+0000","last_active":"2024-03-26T03:49:59.159761+0000","last_peered":"2024-03-26T03:49:59.159761+0000","last_clean":"2024-03-26T03:49:59.159761+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:49:59.159761+0000","last_undegraded":"2024-03-26T03:49:59.159761+0000","last_fullsized":"2024-03-26T03:49:59.159761+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.295312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.295312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.045160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329370+0000","last_change":"2024-03-26T03:48:46.019207+0000","last_active":"2024-03-26T03:49:59.329370+0000","last_peered":"2024-03-26T03:49:59.329370+0000","last_clean":"2024-03-26T03:49:59.329370+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:49:59.329370+0000","last_undegraded":"2024-03-26T03:49:59.329370+0000","last_fullsized":"2024-03-26T03:49:59.329370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.796465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160754+0000","last_change":"2024-03-26T03:48:48.020655+0000","last_active":"2024-03-26T03:49:59.160754+0000","last_peered":"2024-03-26T03:49:59.160754+0000","last_clean":"2024-03-26T03:49:59.160754+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:49:59.160754+0000","last_undegraded":"2024-03-26T03:49:59.160754+0000","last_fullsized":"2024-03-26T03:49:59.160754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:15.127534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330371+0000","last_change":"2024-03-26T03:48:50.039906+0000","last_active":"2024-03-26T03:49:59.330371+0000","last_peered":"2024-03-26T03:49:59.330371+0000","last_clean":"2024-03-26T03:49:59.330371+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:49:59.330371+0000","last_undegraded":"2024-03-26T03:49:59.330371+0000","last_fullsized":"2024-03-26T03:49:59.330371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.037330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":301,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329014+0000","last_change":"2024-03-26T03:48:51.651891+0000","last_active":"2024-03-26T03:49:59.329014+0000","last_peered":"2024-03-26T03:49:59.329014+0000","last_clean":"2024-03-26T03:49:59.329014+0000","last_became_active":"2024-03-26T03:48:34.401623+0000","last_became_peered":"2024-03-26T03:48:34.401623+0000","last_unstale":"2024-03-26T03:49:59.329014+0000","last_undegraded":"2024-03-26T03:49:59.329014+0000","last_fullsized":"2024-03-26T03:49:59.329014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T03:48:51.651774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:48:51.651774+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:20.226983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253530+0000","last_change":"2024-03-26T03:48:44.017929+0000","last_active":"2024-03-26T03:49:59.253530+0000","last_peered":"2024-03-26T03:49:59.253530+0000","last_clean":"2024-03-26T03:49:59.253530+0000","last_became_active":"2024-03-26T03:48:44.017637+0000","last_became_peered":"2024-03-26T03:48:44.017637+0000","last_unstale":"2024-03-26T03:49:59.253530+0000","last_undegraded":"2024-03-26T03:49:59.253530+0000","last_fullsized":"2024-03-26T03:49:59.253530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.329800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329894+0000","last_change":"2024-03-26T03:48:46.210606+0000","last_active":"2024-03-26T03:49:59.329894+0000","last_peered":"2024-03-26T03:49:59.329894+0000","last_clean":"2024-03-26T03:49:59.329894+0000","last_became_active":"2024-03-26T03:48:46.210282+0000","last_became_peered":"2024-03-26T03:48:46.210282+0000","last_unstale":"2024-03-26T03:49:59.329894+0000","last_undegraded":"2024-03-26T03:49:59.329894+0000","last_fullsized":"2024-03-26T03:49:59.329894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:00.772718+0000","last_change":"2024-03-26T03:50:00.772766+0000","last_active":"2024-03-26T03:50:00.772718+0000","last_peered":"2024-03-26T03:50:00.772718+0000","last_clean":"2024-03-26T03:50:00.772718+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:50:00.772718+0000","last_undegraded":"2024-03-26T03:50:00.772718+0000","last_fullsized":"2024-03-26T03:50:00.772718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.772625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.974412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328900+0000","last_change":"2024-03-26T03:48:50.043668+0000","last_active":"2024-03-26T03:49:59.328900+0000","last_peered":"2024-03-26T03:49:59.328900+0000","last_clean":"2024-03-26T03:49:59.328900+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:49:59.328900+0000","last_undegraded":"2024-03-26T03:49:59.328900+0000","last_fullsized":"2024-03-26T03:49:59.328900+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:03.526812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160640+0000","last_change":"2024-03-26T03:48:44.011514+0000","last_active":"2024-03-26T03:49:59.160640+0000","last_peered":"2024-03-26T03:49:59.160640+0000","last_clean":"2024-03-26T03:49:59.160640+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:49:59.160640+0000","last_undegraded":"2024-03-26T03:49:59.160640+0000","last_fullsized":"2024-03-26T03:49:59.160640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.917981+0000","last_change":"2024-03-26T03:49:45.618585+0000","last_active":"2024-03-26T03:50:02.917981+0000","last_peered":"2024-03-26T03:50:02.917981+0000","last_clean":"2024-03-26T03:50:02.917981+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:50:02.917981+0000","last_undegraded":"2024-03-26T03:50:02.917981+0000","last_fullsized":"2024-03-26T03:50:02.917981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:49:45.618470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:45.618470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.990274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161390+0000","last_change":"2024-03-26T03:48:46.026122+0000","last_active":"2024-03-26T03:49:59.161390+0000","last_peered":"2024-03-26T03:49:59.161390+0000","last_clean":"2024-03-26T03:49:59.161390+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:49:59.161390+0000","last_undegraded":"2024-03-26T03:49:59.161390+0000","last_fullsized":"2024-03-26T03:49:59.161390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.218762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330733+0000","last_change":"2024-03-26T03:48:48.022621+0000","last_active":"2024-03-26T03:49:59.330733+0000","last_peered":"2024-03-26T03:49:59.330733+0000","last_clean":"2024-03-26T03:49:59.330733+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:49:59.330733+0000","last_undegraded":"2024-03-26T03:49:59.330733+0000","last_fullsized":"2024-03-26T03:49:59.330733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.069511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328518+0000","last_change":"2024-03-26T03:48:50.260878+0000","last_active":"2024-03-26T03:49:59.328518+0000","last_peered":"2024-03-26T03:49:59.328518+0000","last_clean":"2024-03-26T03:49:59.328518+0000","last_became_active":"2024-03-26T03:48:50.260476+0000","last_became_peered":"2024-03-26T03:48:50.260476+0000","last_unstale":"2024-03-26T03:49:59.328518+0000","last_undegraded":"2024-03-26T03:49:59.328518+0000","last_fullsized":"2024-03-26T03:49:59.328518+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.412999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:00.032849+0000","last_change":"2024-03-26T03:50:00.032886+0000","last_active":"2024-03-26T03:50:00.032849+0000","last_peered":"2024-03-26T03:50:00.032849+0000","last_clean":"2024-03-26T03:50:00.032849+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:50:00.032849+0000","last_undegraded":"2024-03-26T03:50:00.032849+0000","last_fullsized":"2024-03-26T03:50:00.032849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.032763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.032763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:06.902858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:03.113076+0000","last_change":"2024-03-26T03:50:03.113111+0000","last_active":"2024-03-26T03:50:03.113076+0000","last_peered":"2024-03-26T03:50:03.113076+0000","last_clean":"2024-03-26T03:50:03.113076+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:50:03.113076+0000","last_undegraded":"2024-03-26T03:50:03.113076+0000","last_fullsized":"2024-03-26T03:50:03.113076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:03.112983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.112983+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.126914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330164+0000","last_change":"2024-03-26T03:49:54.591020+0000","last_active":"2024-03-26T03:49:59.330164+0000","last_peered":"2024-03-26T03:49:59.330164+0000","last_clean":"2024-03-26T03:49:59.330164+0000","last_became_active":"2024-03-26T03:48:48.021249+0000","last_became_peered":"2024-03-26T03:48:48.021249+0000","last_unstale":"2024-03-26T03:49:59.330164+0000","last_undegraded":"2024-03-26T03:49:59.330164+0000","last_fullsized":"2024-03-26T03:49:59.330164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:54.590915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.590915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.886951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159829+0000","last_change":"2024-03-26T03:48:50.045296+0000","last_active":"2024-03-26T03:49:59.159829+0000","last_peered":"2024-03-26T03:49:59.159829+0000","last_clean":"2024-03-26T03:49:59.159829+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:49:59.159829+0000","last_undegraded":"2024-03-26T03:49:59.159829+0000","last_fullsized":"2024-03-26T03:49:59.159829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:16.928066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330124+0000","last_change":"2024-03-26T03:48:44.014959+0000","last_active":"2024-03-26T03:49:59.330124+0000","last_peered":"2024-03-26T03:49:59.330124+0000","last_clean":"2024-03-26T03:49:59.330124+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:49:59.330124+0000","last_undegraded":"2024-03-26T03:49:59.330124+0000","last_fullsized":"2024-03-26T03:49:59.330124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.457323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328889+0000","last_change":"2024-03-26T03:49:46.927333+0000","last_active":"2024-03-26T03:49:59.328889+0000","last_peered":"2024-03-26T03:49:59.328889+0000","last_clean":"2024-03-26T03:49:59.328889+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:49:59.328889+0000","last_undegraded":"2024-03-26T03:49:59.328889+0000","last_fullsized":"2024-03-26T03:49:59.328889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:49:46.927230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:46.927230+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:54.304888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160961+0000","last_change":"2024-03-26T03:48:48.019177+0000","last_active":"2024-03-26T03:49:59.160961+0000","last_peered":"2024-03-26T03:49:59.160961+0000","last_clean":"2024-03-26T03:49:59.160961+0000","last_became_active":"2024-03-26T03:48:48.018852+0000","last_became_peered":"2024-03-26T03:48:48.018852+0000","last_unstale":"2024-03-26T03:49:59.160961+0000","last_undegraded":"2024-03-26T03:49:59.160961+0000","last_fullsized":"2024-03-26T03:49:59.160961+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.916370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330891+0000","last_change":"2024-03-26T03:49:57.800214+0000","last_active":"2024-03-26T03:49:59.330891+0000","last_peered":"2024-03-26T03:49:59.330891+0000","last_clean":"2024-03-26T03:49:59.330891+0000","last_became_active":"2024-03-26T03:48:50.262788+0000","last_became_peered":"2024-03-26T03:48:50.262788+0000","last_unstale":"2024-03-26T03:49:59.330891+0000","last_undegraded":"2024-03-26T03:49:59.330891+0000","last_fullsized":"2024-03-26T03:49:59.330891+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.800113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.800113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:00.814334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.092005+0000","last_change":"2024-03-26T03:50:02.092039+0000","last_active":"2024-03-26T03:50:02.092005+0000","last_peered":"2024-03-26T03:50:02.092005+0000","last_clean":"2024-03-26T03:50:02.092005+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:50:02.092005+0000","last_undegraded":"2024-03-26T03:50:02.092005+0000","last_fullsized":"2024-03-26T03:50:02.092005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:02.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.091909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.088925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159522+0000","last_change":"2024-03-26T03:48:46.017035+0000","last_active":"2024-03-26T03:49:59.159522+0000","last_peered":"2024-03-26T03:49:59.159522+0000","last_clean":"2024-03-26T03:49:59.159522+0000","last_became_active":"2024-03-26T03:48:46.016634+0000","last_became_peered":"2024-03-26T03:48:46.016634+0000","last_unstale":"2024-03-26T03:49:59.159522+0000","last_undegraded":"2024-03-26T03:49:59.159522+0000","last_fullsized":"2024-03-26T03:49:59.159522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253669+0000","last_change":"2024-03-26T03:48:48.028663+0000","last_active":"2024-03-26T03:49:59.253669+0000","last_peered":"2024-03-26T03:49:59.253669+0000","last_clean":"2024-03-26T03:49:59.253669+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:49:59.253669+0000","last_undegraded":"2024-03-26T03:49:59.253669+0000","last_fullsized":"2024-03-26T03:49:59.253669+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159432+0000","last_change":"2024-03-26T03:48:50.043708+0000","last_active":"2024-03-26T03:49:59.159432+0000","last_peered":"2024-03-26T03:49:59.159432+0000","last_clean":"2024-03-26T03:49:59.159432+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:49:59.159432+0000","last_undegraded":"2024-03-26T03:49:59.159432+0000","last_fullsized":"2024-03-26T03:49:59.159432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254153+0000","last_change":"2024-03-26T03:48:44.013652+0000","last_active":"2024-03-26T03:49:59.254153+0000","last_peered":"2024-03-26T03:49:59.254153+0000","last_clean":"2024-03-26T03:49:59.254153+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:49:59.254153+0000","last_undegraded":"2024-03-26T03:49:59.254153+0000","last_fullsized":"2024-03-26T03:49:59.254153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":123,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253377+0000","last_change":"2024-03-26T03:49:51.111438+0000","last_active":"2024-03-26T03:49:59.253377+0000","last_peered":"2024-03-26T03:49:59.253377+0000","last_clean":"2024-03-26T03:49:59.253377+0000","last_became_active":"2024-03-26T03:48:46.209319+0000","last_became_peered":"2024-03-26T03:48:46.209319+0000","last_unstale":"2024-03-26T03:49:59.253377+0000","last_undegraded":"2024-03-26T03:49:59.253377+0000","last_fullsized":"2024-03-26T03:49:59.253377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.111316+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.974577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159262+0000","last_change":"2024-03-26T03:48:48.029521+0000","last_active":"2024-03-26T03:49:59.159262+0000","last_peered":"2024-03-26T03:49:59.159262+0000","last_clean":"2024-03-26T03:49:59.159262+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:49:59.159262+0000","last_undegraded":"2024-03-26T03:49:59.159262+0000","last_fullsized":"2024-03-26T03:49:59.159262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329859+0000","last_change":"2024-03-26T03:48:50.034819+0000","last_active":"2024-03-26T03:49:59.329859+0000","last_peered":"2024-03-26T03:49:59.329859+0000","last_clean":"2024-03-26T03:49:59.329859+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:49:59.329859+0000","last_undegraded":"2024-03-26T03:49:59.329859+0000","last_fullsized":"2024-03-26T03:49:59.329859+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.402354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160400+0000","last_change":"2024-03-26T03:48:48.020914+0000","last_active":"2024-03-26T03:49:59.160400+0000","last_peered":"2024-03-26T03:49:59.160400+0000","last_clean":"2024-03-26T03:49:59.160400+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:49:59.160400+0000","last_undegraded":"2024-03-26T03:49:59.160400+0000","last_fullsized":"2024-03-26T03:49:59.160400+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253433+0000","last_change":"2024-03-26T03:48:44.017015+0000","last_active":"2024-03-26T03:49:59.253433+0000","last_peered":"2024-03-26T03:49:59.253433+0000","last_clean":"2024-03-26T03:49:59.253433+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:49:59.253433+0000","last_undegraded":"2024-03-26T03:49:59.253433+0000","last_fullsized":"2024-03-26T03:49:59.253433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:08.476078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329362+0000","last_change":"2024-03-26T03:48:46.026556+0000","last_active":"2024-03-26T03:49:59.329362+0000","last_peered":"2024-03-26T03:49:59.329362+0000","last_clean":"2024-03-26T03:49:59.329362+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:49:59.329362+0000","last_undegraded":"2024-03-26T03:49:59.329362+0000","last_fullsized":"2024-03-26T03:49:59.329362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253539+0000","last_change":"2024-03-26T03:48:50.261659+0000","last_active":"2024-03-26T03:49:59.253539+0000","last_peered":"2024-03-26T03:49:59.253539+0000","last_clean":"2024-03-26T03:49:59.253539+0000","last_became_active":"2024-03-26T03:48:50.260697+0000","last_became_peered":"2024-03-26T03:48:50.260697+0000","last_unstale":"2024-03-26T03:49:59.253539+0000","last_undegraded":"2024-03-26T03:49:59.253539+0000","last_fullsized":"2024-03-26T03:49:59.253539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:03.983242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.189050+0000","last_change":"2024-03-26T03:50:02.189077+0000","last_active":"2024-03-26T03:50:02.189050+0000","last_peered":"2024-03-26T03:50:02.189050+0000","last_clean":"2024-03-26T03:50:02.189050+0000","last_became_active":"2024-03-26T03:48:48.019448+0000","last_became_peered":"2024-03-26T03:48:48.019448+0000","last_unstale":"2024-03-26T03:50:02.189050+0000","last_undegraded":"2024-03-26T03:50:02.189050+0000","last_fullsized":"2024-03-26T03:50:02.189050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.188975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.969923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159264+0000","last_change":"2024-03-26T03:49:48.865134+0000","last_active":"2024-03-26T03:49:59.159264+0000","last_peered":"2024-03-26T03:49:59.159264+0000","last_clean":"2024-03-26T03:49:59.159264+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:49:59.159264+0000","last_undegraded":"2024-03-26T03:49:59.159264+0000","last_fullsized":"2024-03-26T03:49:59.159264+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.864997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.477744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160163+0000","last_change":"2024-03-26T03:48:46.019214+0000","last_active":"2024-03-26T03:49:59.160163+0000","last_peered":"2024-03-26T03:49:59.160163+0000","last_clean":"2024-03-26T03:49:59.160163+0000","last_became_active":"2024-03-26T03:48:46.018601+0000","last_became_peered":"2024-03-26T03:48:46.018601+0000","last_unstale":"2024-03-26T03:49:59.160163+0000","last_undegraded":"2024-03-26T03:49:59.160163+0000","last_fullsized":"2024-03-26T03:49:59.160163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160965+0000","last_change":"2024-03-26T03:48:50.044134+0000","last_active":"2024-03-26T03:49:59.160965+0000","last_peered":"2024-03-26T03:49:59.160965+0000","last_clean":"2024-03-26T03:49:59.160965+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:49:59.160965+0000","last_undegraded":"2024-03-26T03:49:59.160965+0000","last_fullsized":"2024-03-26T03:49:59.160965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.818024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329349+0000","last_change":"2024-03-26T03:48:48.026964+0000","last_active":"2024-03-26T03:49:59.329349+0000","last_peered":"2024-03-26T03:49:59.329349+0000","last_clean":"2024-03-26T03:49:59.329349+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:49:59.329349+0000","last_undegraded":"2024-03-26T03:49:59.329349+0000","last_fullsized":"2024-03-26T03:49:59.329349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330207+0000","last_change":"2024-03-26T03:49:51.516987+0000","last_active":"2024-03-26T03:49:59.330207+0000","last_peered":"2024-03-26T03:49:59.330207+0000","last_clean":"2024-03-26T03:49:59.330207+0000","last_became_active":"2024-03-26T03:48:44.012660+0000","last_became_peered":"2024-03-26T03:48:44.012660+0000","last_unstale":"2024-03-26T03:49:59.330207+0000","last_undegraded":"2024-03-26T03:49:59.330207+0000","last_fullsized":"2024-03-26T03:49:59.330207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:51.516891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.516891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:57.433207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:03.880793+0000","last_change":"2024-03-26T03:50:03.880855+0000","last_active":"2024-03-26T03:50:03.880793+0000","last_peered":"2024-03-26T03:50:03.880793+0000","last_clean":"2024-03-26T03:50:03.880793+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:50:03.880793+0000","last_undegraded":"2024-03-26T03:50:03.880793+0000","last_fullsized":"2024-03-26T03:50:03.880793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:03.880711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.880711+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.791446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.877459+0000","last_change":"2024-03-26T03:50:02.877487+0000","last_active":"2024-03-26T03:50:02.877459+0000","last_peered":"2024-03-26T03:50:02.877459+0000","last_clean":"2024-03-26T03:50:02.877459+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:50:02.877459+0000","last_undegraded":"2024-03-26T03:50:02.877459+0000","last_fullsized":"2024-03-26T03:50:02.877459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.877373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.877373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.008619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.300700+0000","last_change":"2024-03-26T03:50:02.300700+0000","last_active":"2024-03-26T03:50:02.300700+0000","last_peered":"2024-03-26T03:50:02.300700+0000","last_clean":"2024-03-26T03:50:02.300700+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:50:02.300700+0000","last_undegraded":"2024-03-26T03:50:02.300700+0000","last_fullsized":"2024-03-26T03:50:02.300700+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:02.300594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.300594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.021817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253820+0000","last_change":"2024-03-26T03:48:44.015913+0000","last_active":"2024-03-26T03:49:59.253820+0000","last_peered":"2024-03-26T03:49:59.253820+0000","last_clean":"2024-03-26T03:49:59.253820+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:49:59.253820+0000","last_undegraded":"2024-03-26T03:49:59.253820+0000","last_fullsized":"2024-03-26T03:49:59.253820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.708831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159773+0000","last_change":"2024-03-26T03:48:46.023356+0000","last_active":"2024-03-26T03:49:59.159773+0000","last_peered":"2024-03-26T03:49:59.159773+0000","last_clean":"2024-03-26T03:49:59.159773+0000","last_became_active":"2024-03-26T03:48:46.022937+0000","last_became_peered":"2024-03-26T03:48:46.022937+0000","last_unstale":"2024-03-26T03:49:59.159773+0000","last_undegraded":"2024-03-26T03:49:59.159773+0000","last_fullsized":"2024-03-26T03:49:59.159773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.912372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":599,"ondisk_log_size":599,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070484,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1396736,"data_stored":935613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1363968,"data_stored":933201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":344618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32540,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712612,"statfs":{"total":95995035648,"available":95961714688,"internally_reserved":0,"allocated":1396736,"data_stored":935342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31928,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713224,"statfs":{"total":95995035648,"available":95962341376,"internally_reserved":0,"allocated":770048,"data_stored":342961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":345071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":343451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659165,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":774144,"data_stored":343159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:50:06.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:50:06.509 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T03:50:06.535 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T03:50:06.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T03:50:06.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:06.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:06.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T03:50:06.536 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:06 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T03:50:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T03:50:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T03:50:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T03:50:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:06 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T03:50:07.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T03:50:07.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T03:50:07.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T03:50:07.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T03:50:07.627 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T03:50:07.628 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T03:50:07.628 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T03:50:07.628 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:07 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T03:50:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:07 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T03:50:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:07 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T03:50:07.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:07 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T03:50:07.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:07 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T03:50:08.367 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:50:08.369 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 4.1d scrub starts 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 4.1d scrub ok 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 4.1d scrub starts 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 4.1d scrub ok 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:08.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:08 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 4.1d scrub starts 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 4.1d scrub ok 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:08 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:09.101 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-03-26T03:50:07.164035+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1295,"num_read_kb":1109,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":807,"ondisk_log_size":807,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244896,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716320,"statfs":{"total":767960285184,"available":767709511680,"internally_reserved":0,"allocated":8155136,"data_stored":4523416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003494"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159349+0000","last_change":"2024-03-26T03:48:50.035293+0000","last_active":"2024-03-26T03:49:59.159349+0000","last_peered":"2024-03-26T03:49:59.159349+0000","last_clean":"2024-03-26T03:49:59.159349+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:49:59.159349+0000","last_undegraded":"2024-03-26T03:49:59.159349+0000","last_fullsized":"2024-03-26T03:49:59.159349+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.235116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328558+0000","last_change":"2024-03-26T03:48:48.025490+0000","last_active":"2024-03-26T03:49:59.328558+0000","last_peered":"2024-03-26T03:49:59.328558+0000","last_clean":"2024-03-26T03:49:59.328558+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:49:59.328558+0000","last_undegraded":"2024-03-26T03:49:59.328558+0000","last_fullsized":"2024-03-26T03:49:59.328558+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.103757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161626+0000","last_change":"2024-03-26T03:49:49.005275+0000","last_active":"2024-03-26T03:49:59.161626+0000","last_peered":"2024-03-26T03:49:59.161626+0000","last_clean":"2024-03-26T03:49:59.161626+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:49:59.161626+0000","last_undegraded":"2024-03-26T03:49:59.161626+0000","last_fullsized":"2024-03-26T03:49:59.161626+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:49.005129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:49.005129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.159629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159200+0000","last_change":"2024-03-26T03:48:46.018459+0000","last_active":"2024-03-26T03:49:59.159200+0000","last_peered":"2024-03-26T03:49:59.159200+0000","last_clean":"2024-03-26T03:49:59.159200+0000","last_became_active":"2024-03-26T03:48:46.018079+0000","last_became_peered":"2024-03-26T03:48:46.018079+0000","last_unstale":"2024-03-26T03:49:59.159200+0000","last_undegraded":"2024-03-26T03:49:59.159200+0000","last_fullsized":"2024-03-26T03:49:59.159200+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.894463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253610+0000","last_change":"2024-03-26T03:48:50.262953+0000","last_active":"2024-03-26T03:49:59.253610+0000","last_peered":"2024-03-26T03:49:59.253610+0000","last_clean":"2024-03-26T03:49:59.253610+0000","last_became_active":"2024-03-26T03:48:50.262548+0000","last_became_peered":"2024-03-26T03:48:50.262548+0000","last_unstale":"2024-03-26T03:49:59.253610+0000","last_undegraded":"2024-03-26T03:49:59.253610+0000","last_fullsized":"2024-03-26T03:49:59.253610+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159206+0000","last_change":"2024-03-26T03:48:48.025511+0000","last_active":"2024-03-26T03:49:59.159206+0000","last_peered":"2024-03-26T03:49:59.159206+0000","last_clean":"2024-03-26T03:49:59.159206+0000","last_became_active":"2024-03-26T03:48:48.024923+0000","last_became_peered":"2024-03-26T03:48:48.024923+0000","last_unstale":"2024-03-26T03:49:59.159206+0000","last_undegraded":"2024-03-26T03:49:59.159206+0000","last_fullsized":"2024-03-26T03:49:59.159206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.464377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253931+0000","last_change":"2024-03-26T03:48:44.017642+0000","last_active":"2024-03-26T03:49:59.253931+0000","last_peered":"2024-03-26T03:49:59.253931+0000","last_clean":"2024-03-26T03:49:59.253931+0000","last_became_active":"2024-03-26T03:48:44.017341+0000","last_became_peered":"2024-03-26T03:48:44.017341+0000","last_unstale":"2024-03-26T03:49:59.253931+0000","last_undegraded":"2024-03-26T03:49:59.253931+0000","last_fullsized":"2024-03-26T03:49:59.253931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.584504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":122,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253773+0000","last_change":"2024-03-26T03:48:46.020802+0000","last_active":"2024-03-26T03:49:59.253773+0000","last_peered":"2024-03-26T03:49:59.253773+0000","last_clean":"2024-03-26T03:49:59.253773+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:49:59.253773+0000","last_undegraded":"2024-03-26T03:49:59.253773+0000","last_fullsized":"2024-03-26T03:49:59.253773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160055+0000","last_change":"2024-03-26T03:48:50.037848+0000","last_active":"2024-03-26T03:49:59.160055+0000","last_peered":"2024-03-26T03:49:59.160055+0000","last_clean":"2024-03-26T03:49:59.160055+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:49:59.160055+0000","last_undegraded":"2024-03-26T03:49:59.160055+0000","last_fullsized":"2024-03-26T03:49:59.160055+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.299751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329686+0000","last_change":"2024-03-26T03:49:59.135484+0000","last_active":"2024-03-26T03:49:59.329686+0000","last_peered":"2024-03-26T03:49:59.329686+0000","last_clean":"2024-03-26T03:49:59.329686+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:49:59.329686+0000","last_undegraded":"2024-03-26T03:49:59.329686+0000","last_fullsized":"2024-03-26T03:49:59.329686+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.135403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.135403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.439280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329947+0000","last_change":"2024-03-26T03:48:44.015557+0000","last_active":"2024-03-26T03:49:59.329947+0000","last_peered":"2024-03-26T03:49:59.329947+0000","last_clean":"2024-03-26T03:49:59.329947+0000","last_became_active":"2024-03-26T03:48:44.015277+0000","last_became_peered":"2024-03-26T03:48:44.015277+0000","last_unstale":"2024-03-26T03:49:59.329947+0000","last_undegraded":"2024-03-26T03:49:59.329947+0000","last_fullsized":"2024-03-26T03:49:59.329947+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159518+0000","last_change":"2024-03-26T03:49:54.881008+0000","last_active":"2024-03-26T03:49:59.159518+0000","last_peered":"2024-03-26T03:49:59.159518+0000","last_clean":"2024-03-26T03:49:59.159518+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:49:59.159518+0000","last_undegraded":"2024-03-26T03:49:59.159518+0000","last_fullsized":"2024-03-26T03:49:59.159518+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:49:54.880892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.880892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.130490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329449+0000","last_change":"2024-03-26T03:48:50.039963+0000","last_active":"2024-03-26T03:49:59.329449+0000","last_peered":"2024-03-26T03:49:59.329449+0000","last_clean":"2024-03-26T03:49:59.329449+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:49:59.329449+0000","last_undegraded":"2024-03-26T03:49:59.329449+0000","last_fullsized":"2024-03-26T03:49:59.329449+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.621578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328906+0000","last_change":"2024-03-26T03:48:48.032017+0000","last_active":"2024-03-26T03:49:59.328906+0000","last_peered":"2024-03-26T03:49:59.328906+0000","last_clean":"2024-03-26T03:49:59.328906+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:49:59.328906+0000","last_undegraded":"2024-03-26T03:49:59.328906+0000","last_fullsized":"2024-03-26T03:49:59.328906+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.912589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329810+0000","last_change":"2024-03-26T03:48:44.013770+0000","last_active":"2024-03-26T03:49:59.329810+0000","last_peered":"2024-03-26T03:49:59.329810+0000","last_clean":"2024-03-26T03:49:59.329810+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:49:59.329810+0000","last_undegraded":"2024-03-26T03:49:59.329810+0000","last_fullsized":"2024-03-26T03:49:59.329810+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.115985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'11","reported_seq":71,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159577+0000","last_change":"2024-03-26T03:48:46.023941+0000","last_active":"2024-03-26T03:49:59.159577+0000","last_peered":"2024-03-26T03:49:59.159577+0000","last_clean":"2024-03-26T03:49:59.159577+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:49:59.159577+0000","last_undegraded":"2024-03-26T03:49:59.159577+0000","last_fullsized":"2024-03-26T03:49:59.159577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.340361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160262+0000","last_change":"2024-03-26T03:48:50.261495+0000","last_active":"2024-03-26T03:49:59.160262+0000","last_peered":"2024-03-26T03:49:59.160262+0000","last_clean":"2024-03-26T03:49:59.160262+0000","last_became_active":"2024-03-26T03:48:50.260266+0000","last_became_peered":"2024-03-26T03:48:50.260266+0000","last_unstale":"2024-03-26T03:49:59.160262+0000","last_undegraded":"2024-03-26T03:49:59.160262+0000","last_fullsized":"2024-03-26T03:49:59.160262+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.926198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329321+0000","last_change":"2024-03-26T03:48:48.025562+0000","last_active":"2024-03-26T03:49:59.329321+0000","last_peered":"2024-03-26T03:49:59.329321+0000","last_clean":"2024-03-26T03:49:59.329321+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:49:59.329321+0000","last_undegraded":"2024-03-26T03:49:59.329321+0000","last_fullsized":"2024-03-26T03:49:59.329321+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.783166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161082+0000","last_change":"2024-03-26T03:49:58.053896+0000","last_active":"2024-03-26T03:49:59.161082+0000","last_peered":"2024-03-26T03:49:59.161082+0000","last_clean":"2024-03-26T03:49:59.161082+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:49:59.161082+0000","last_undegraded":"2024-03-26T03:49:59.161082+0000","last_fullsized":"2024-03-26T03:49:59.161082+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.053811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.053811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.133041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160189+0000","last_change":"2024-03-26T03:49:53.266632+0000","last_active":"2024-03-26T03:49:59.160189+0000","last_peered":"2024-03-26T03:49:59.160189+0000","last_clean":"2024-03-26T03:49:59.160189+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:49:59.160189+0000","last_undegraded":"2024-03-26T03:49:59.160189+0000","last_fullsized":"2024-03-26T03:49:59.160189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:49:53.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:53.266536+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.703552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254274+0000","last_change":"2024-03-26T03:48:50.261258+0000","last_active":"2024-03-26T03:49:59.254274+0000","last_peered":"2024-03-26T03:49:59.254274+0000","last_clean":"2024-03-26T03:49:59.254274+0000","last_became_active":"2024-03-26T03:48:50.260688+0000","last_became_peered":"2024-03-26T03:48:50.260688+0000","last_unstale":"2024-03-26T03:49:59.254274+0000","last_undegraded":"2024-03-26T03:49:59.254274+0000","last_fullsized":"2024-03-26T03:49:59.254274+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159768+0000","last_change":"2024-03-26T03:48:48.030756+0000","last_active":"2024-03-26T03:49:59.159768+0000","last_peered":"2024-03-26T03:49:59.159768+0000","last_clean":"2024-03-26T03:49:59.159768+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:49:59.159768+0000","last_undegraded":"2024-03-26T03:49:59.159768+0000","last_fullsized":"2024-03-26T03:49:59.159768+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:08.338651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159541+0000","last_change":"2024-03-26T03:49:58.872361+0000","last_active":"2024-03-26T03:49:59.159541+0000","last_peered":"2024-03-26T03:49:59.159541+0000","last_clean":"2024-03-26T03:49:59.159541+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:49:59.159541+0000","last_undegraded":"2024-03-26T03:49:59.159541+0000","last_fullsized":"2024-03-26T03:49:59.159541+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.872258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.872258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.889376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":84,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253731+0000","last_change":"2024-03-26T03:48:46.027100+0000","last_active":"2024-03-26T03:49:59.253731+0000","last_peered":"2024-03-26T03:49:59.253731+0000","last_clean":"2024-03-26T03:49:59.253731+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:49:59.253731+0000","last_undegraded":"2024-03-26T03:49:59.253731+0000","last_fullsized":"2024-03-26T03:49:59.253731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":51,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.740617+0000","last_change":"2024-03-26T03:50:02.740650+0000","last_active":"2024-03-26T03:50:02.740617+0000","last_peered":"2024-03-26T03:50:02.740617+0000","last_clean":"2024-03-26T03:50:02.740617+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:50:02.740617+0000","last_undegraded":"2024-03-26T03:50:02.740617+0000","last_fullsized":"2024-03-26T03:50:02.740617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.740545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.965068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330438+0000","last_change":"2024-03-26T03:48:48.022793+0000","last_active":"2024-03-26T03:49:59.330438+0000","last_peered":"2024-03-26T03:49:59.330438+0000","last_clean":"2024-03-26T03:49:59.330438+0000","last_became_active":"2024-03-26T03:48:48.022480+0000","last_became_peered":"2024-03-26T03:48:48.022480+0000","last_unstale":"2024-03-26T03:49:59.330438+0000","last_undegraded":"2024-03-26T03:49:59.330438+0000","last_fullsized":"2024-03-26T03:49:59.330438+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:14.690568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328667+0000","last_change":"2024-03-26T03:48:44.018169+0000","last_active":"2024-03-26T03:49:59.328667+0000","last_peered":"2024-03-26T03:49:59.328667+0000","last_clean":"2024-03-26T03:49:59.328667+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:49:59.328667+0000","last_undegraded":"2024-03-26T03:49:59.328667+0000","last_fullsized":"2024-03-26T03:49:59.328667+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.658563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":76,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161712+0000","last_change":"2024-03-26T03:48:46.210691+0000","last_active":"2024-03-26T03:49:59.161712+0000","last_peered":"2024-03-26T03:49:59.161712+0000","last_clean":"2024-03-26T03:49:59.161712+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:49:59.161712+0000","last_undegraded":"2024-03-26T03:49:59.161712+0000","last_fullsized":"2024-03-26T03:49:59.161712+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.585779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329047+0000","last_change":"2024-03-26T03:48:50.035147+0000","last_active":"2024-03-26T03:49:59.329047+0000","last_peered":"2024-03-26T03:49:59.329047+0000","last_clean":"2024-03-26T03:49:59.329047+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:49:59.329047+0000","last_undegraded":"2024-03-26T03:49:59.329047+0000","last_fullsized":"2024-03-26T03:49:59.329047+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.410878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161704+0000","last_change":"2024-03-26T03:48:48.020953+0000","last_active":"2024-03-26T03:49:59.161704+0000","last_peered":"2024-03-26T03:49:59.161704+0000","last_clean":"2024-03-26T03:49:59.161704+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:49:59.161704+0000","last_undegraded":"2024-03-26T03:49:59.161704+0000","last_fullsized":"2024-03-26T03:49:59.161704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253266+0000","last_change":"2024-03-26T03:48:44.016177+0000","last_active":"2024-03-26T03:49:59.253266+0000","last_peered":"2024-03-26T03:49:59.253266+0000","last_clean":"2024-03-26T03:49:59.253266+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:49:59.253266+0000","last_undegraded":"2024-03-26T03:49:59.253266+0000","last_fullsized":"2024-03-26T03:49:59.253266+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.493384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330349+0000","last_change":"2024-03-26T03:49:58.491993+0000","last_active":"2024-03-26T03:49:59.330349+0000","last_peered":"2024-03-26T03:49:59.330349+0000","last_clean":"2024-03-26T03:49:59.330349+0000","last_became_active":"2024-03-26T03:48:46.023417+0000","last_became_peered":"2024-03-26T03:48:46.023417+0000","last_unstale":"2024-03-26T03:49:59.330349+0000","last_undegraded":"2024-03-26T03:49:59.330349+0000","last_fullsized":"2024-03-26T03:49:59.330349+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:49:58.491875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.491875+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.158055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328787+0000","last_change":"2024-03-26T03:48:50.029982+0000","last_active":"2024-03-26T03:49:59.328787+0000","last_peered":"2024-03-26T03:49:59.328787+0000","last_clean":"2024-03-26T03:49:59.328787+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:49:59.328787+0000","last_undegraded":"2024-03-26T03:49:59.328787+0000","last_fullsized":"2024-03-26T03:49:59.328787+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.788600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328760+0000","last_change":"2024-03-26T03:48:48.024958+0000","last_active":"2024-03-26T03:49:59.328760+0000","last_peered":"2024-03-26T03:49:59.328760+0000","last_clean":"2024-03-26T03:49:59.328760+0000","last_became_active":"2024-03-26T03:48:48.022532+0000","last_became_peered":"2024-03-26T03:48:48.022532+0000","last_unstale":"2024-03-26T03:49:59.328760+0000","last_undegraded":"2024-03-26T03:49:59.328760+0000","last_fullsized":"2024-03-26T03:49:59.328760+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.189825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161609+0000","last_change":"2024-03-26T03:48:44.012132+0000","last_active":"2024-03-26T03:49:59.161609+0000","last_peered":"2024-03-26T03:49:59.161609+0000","last_clean":"2024-03-26T03:49:59.161609+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:49:59.161609+0000","last_undegraded":"2024-03-26T03:49:59.161609+0000","last_fullsized":"2024-03-26T03:49:59.161609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":55,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253280+0000","last_change":"2024-03-26T03:48:46.027982+0000","last_active":"2024-03-26T03:49:59.253280+0000","last_peered":"2024-03-26T03:49:59.253280+0000","last_clean":"2024-03-26T03:49:59.253280+0000","last_became_active":"2024-03-26T03:48:46.027671+0000","last_became_peered":"2024-03-26T03:48:46.027671+0000","last_unstale":"2024-03-26T03:49:59.253280+0000","last_undegraded":"2024-03-26T03:49:59.253280+0000","last_fullsized":"2024-03-26T03:49:59.253280+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253482+0000","last_change":"2024-03-26T03:48:50.261731+0000","last_active":"2024-03-26T03:49:59.253482+0000","last_peered":"2024-03-26T03:49:59.253482+0000","last_clean":"2024-03-26T03:49:59.253482+0000","last_became_active":"2024-03-26T03:48:50.261396+0000","last_became_peered":"2024-03-26T03:48:50.261396+0000","last_unstale":"2024-03-26T03:49:59.253482+0000","last_undegraded":"2024-03-26T03:49:59.253482+0000","last_fullsized":"2024-03-26T03:49:59.253482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329302+0000","last_change":"2024-03-26T03:48:48.029980+0000","last_active":"2024-03-26T03:49:59.329302+0000","last_peered":"2024-03-26T03:49:59.329302+0000","last_clean":"2024-03-26T03:49:59.329302+0000","last_became_active":"2024-03-26T03:48:48.029501+0000","last_became_peered":"2024-03-26T03:48:48.029501+0000","last_unstale":"2024-03-26T03:49:59.329302+0000","last_undegraded":"2024-03-26T03:49:59.329302+0000","last_fullsized":"2024-03-26T03:49:59.329302+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.171082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330124+0000","last_change":"2024-03-26T03:48:44.019133+0000","last_active":"2024-03-26T03:49:59.330124+0000","last_peered":"2024-03-26T03:49:59.330124+0000","last_clean":"2024-03-26T03:49:59.330124+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:49:59.330124+0000","last_undegraded":"2024-03-26T03:49:59.330124+0000","last_fullsized":"2024-03-26T03:49:59.330124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'17","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253661+0000","last_change":"2024-03-26T03:48:46.209729+0000","last_active":"2024-03-26T03:49:59.253661+0000","last_peered":"2024-03-26T03:49:59.253661+0000","last_clean":"2024-03-26T03:49:59.253661+0000","last_became_active":"2024-03-26T03:48:46.209381+0000","last_became_peered":"2024-03-26T03:48:46.209381+0000","last_unstale":"2024-03-26T03:49:59.253661+0000","last_undegraded":"2024-03-26T03:49:59.253661+0000","last_fullsized":"2024-03-26T03:49:59.253661+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159720+0000","last_change":"2024-03-26T03:48:50.037564+0000","last_active":"2024-03-26T03:49:59.159720+0000","last_peered":"2024-03-26T03:49:59.159720+0000","last_clean":"2024-03-26T03:49:59.159720+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:49:59.159720+0000","last_undegraded":"2024-03-26T03:49:59.159720+0000","last_fullsized":"2024-03-26T03:49:59.159720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.630721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253770+0000","last_change":"2024-03-26T03:49:56.980611+0000","last_active":"2024-03-26T03:49:59.253770+0000","last_peered":"2024-03-26T03:49:59.253770+0000","last_clean":"2024-03-26T03:49:59.253770+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:49:59.253770+0000","last_undegraded":"2024-03-26T03:49:59.253770+0000","last_fullsized":"2024-03-26T03:49:59.253770+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.980524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.980524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.615228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329754+0000","last_change":"2024-03-26T03:49:50.773333+0000","last_active":"2024-03-26T03:49:59.329754+0000","last_peered":"2024-03-26T03:49:59.329754+0000","last_clean":"2024-03-26T03:49:59.329754+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:49:59.329754+0000","last_undegraded":"2024-03-26T03:49:59.329754+0000","last_fullsized":"2024-03-26T03:49:59.329754+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:50.773202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:50.773202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.850607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328926+0000","last_change":"2024-03-26T03:49:51.011344+0000","last_active":"2024-03-26T03:49:59.328926+0000","last_peered":"2024-03-26T03:49:59.328926+0000","last_clean":"2024-03-26T03:49:59.328926+0000","last_became_active":"2024-03-26T03:48:46.017045+0000","last_became_peered":"2024-03-26T03:48:46.017045+0000","last_unstale":"2024-03-26T03:49:59.328926+0000","last_undegraded":"2024-03-26T03:49:59.328926+0000","last_fullsized":"2024-03-26T03:49:59.328926+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:49:51.011226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.011226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:04.692152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.986797+0000","last_change":"2024-03-26T03:50:02.986818+0000","last_active":"2024-03-26T03:50:02.986797+0000","last_peered":"2024-03-26T03:50:02.986797+0000","last_clean":"2024-03-26T03:50:02.986797+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:50:02.986797+0000","last_undegraded":"2024-03-26T03:50:02.986797+0000","last_fullsized":"2024-03-26T03:50:02.986797+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.986736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.986736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.565698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328830+0000","last_change":"2024-03-26T03:48:48.030582+0000","last_active":"2024-03-26T03:49:59.328830+0000","last_peered":"2024-03-26T03:49:59.328830+0000","last_clean":"2024-03-26T03:49:59.328830+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:49:59.328830+0000","last_undegraded":"2024-03-26T03:49:59.328830+0000","last_fullsized":"2024-03-26T03:49:59.328830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.060061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159891+0000","last_change":"2024-03-26T03:49:56.925827+0000","last_active":"2024-03-26T03:49:59.159891+0000","last_peered":"2024-03-26T03:49:59.159891+0000","last_clean":"2024-03-26T03:49:59.159891+0000","last_became_active":"2024-03-26T03:48:44.013906+0000","last_became_peered":"2024-03-26T03:48:44.013906+0000","last_unstale":"2024-03-26T03:49:59.159891+0000","last_undegraded":"2024-03-26T03:49:59.159891+0000","last_fullsized":"2024-03-26T03:49:59.159891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.925725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.925725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":68,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159677+0000","last_change":"2024-03-26T03:48:46.019374+0000","last_active":"2024-03-26T03:49:59.159677+0000","last_peered":"2024-03-26T03:49:59.159677+0000","last_clean":"2024-03-26T03:49:59.159677+0000","last_became_active":"2024-03-26T03:48:46.019043+0000","last_became_peered":"2024-03-26T03:48:46.019043+0000","last_unstale":"2024-03-26T03:49:59.159677+0000","last_undegraded":"2024-03-26T03:49:59.159677+0000","last_fullsized":"2024-03-26T03:49:59.159677+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.864202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161210+0000","last_change":"2024-03-26T03:48:50.046189+0000","last_active":"2024-03-26T03:49:59.161210+0000","last_peered":"2024-03-26T03:49:59.161210+0000","last_clean":"2024-03-26T03:49:59.161210+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:49:59.161210+0000","last_undegraded":"2024-03-26T03:49:59.161210+0000","last_fullsized":"2024-03-26T03:49:59.161210+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.027196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330076+0000","last_change":"2024-03-26T03:49:55.131855+0000","last_active":"2024-03-26T03:49:59.330076+0000","last_peered":"2024-03-26T03:49:59.330076+0000","last_clean":"2024-03-26T03:49:59.330076+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:49:59.330076+0000","last_undegraded":"2024-03-26T03:49:59.330076+0000","last_fullsized":"2024-03-26T03:49:59.330076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:55.131726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:55.131726+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.670201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330602+0000","last_change":"2024-03-26T03:49:48.735316+0000","last_active":"2024-03-26T03:49:59.330602+0000","last_peered":"2024-03-26T03:49:59.330602+0000","last_clean":"2024-03-26T03:49:59.330602+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:49:59.330602+0000","last_undegraded":"2024-03-26T03:49:59.330602+0000","last_fullsized":"2024-03-26T03:49:59.330602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.735213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.735213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:52.231017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253828+0000","last_change":"2024-03-26T03:48:46.209727+0000","last_active":"2024-03-26T03:49:59.253828+0000","last_peered":"2024-03-26T03:49:59.253828+0000","last_clean":"2024-03-26T03:49:59.253828+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:49:59.253828+0000","last_undegraded":"2024-03-26T03:49:59.253828+0000","last_fullsized":"2024-03-26T03:49:59.253828+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.096233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160356+0000","last_change":"2024-03-26T03:48:50.038644+0000","last_active":"2024-03-26T03:49:59.160356+0000","last_peered":"2024-03-26T03:49:59.160356+0000","last_clean":"2024-03-26T03:49:59.160356+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:49:59.160356+0000","last_undegraded":"2024-03-26T03:49:59.160356+0000","last_fullsized":"2024-03-26T03:49:59.160356+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254130+0000","last_change":"2024-03-26T03:48:48.029400+0000","last_active":"2024-03-26T03:49:59.254130+0000","last_peered":"2024-03-26T03:49:59.254130+0000","last_clean":"2024-03-26T03:49:59.254130+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:49:59.254130+0000","last_undegraded":"2024-03-26T03:49:59.254130+0000","last_fullsized":"2024-03-26T03:49:59.254130+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.644125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160716+0000","last_change":"2024-03-26T03:48:44.015539+0000","last_active":"2024-03-26T03:49:59.160716+0000","last_peered":"2024-03-26T03:49:59.160716+0000","last_clean":"2024-03-26T03:49:59.160716+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:49:59.160716+0000","last_undegraded":"2024-03-26T03:49:59.160716+0000","last_fullsized":"2024-03-26T03:49:59.160716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.181972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330260+0000","last_change":"2024-03-26T03:48:46.210667+0000","last_active":"2024-03-26T03:49:59.330260+0000","last_peered":"2024-03-26T03:49:59.330260+0000","last_clean":"2024-03-26T03:49:59.330260+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:49:59.330260+0000","last_undegraded":"2024-03-26T03:49:59.330260+0000","last_fullsized":"2024-03-26T03:49:59.330260+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.284559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"73'1","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160367+0000","last_change":"2024-03-26T03:48:50.033575+0000","last_active":"2024-03-26T03:49:59.160367+0000","last_peered":"2024-03-26T03:49:59.160367+0000","last_clean":"2024-03-26T03:49:59.160367+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:49:59.160367+0000","last_undegraded":"2024-03-26T03:49:59.160367+0000","last_fullsized":"2024-03-26T03:49:59.160367+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.438017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254250+0000","last_change":"2024-03-26T03:48:48.032124+0000","last_active":"2024-03-26T03:49:59.254250+0000","last_peered":"2024-03-26T03:49:59.254250+0000","last_clean":"2024-03-26T03:49:59.254250+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:49:59.254250+0000","last_undegraded":"2024-03-26T03:49:59.254250+0000","last_fullsized":"2024-03-26T03:49:59.254250+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329621+0000","last_change":"2024-03-26T03:49:56.749968+0000","last_active":"2024-03-26T03:49:59.329621+0000","last_peered":"2024-03-26T03:49:59.329621+0000","last_clean":"2024-03-26T03:49:59.329621+0000","last_became_active":"2024-03-26T03:48:44.018391+0000","last_became_peered":"2024-03-26T03:48:44.018391+0000","last_unstale":"2024-03-26T03:49:59.329621+0000","last_undegraded":"2024-03-26T03:49:59.329621+0000","last_fullsized":"2024-03-26T03:49:59.329621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.749811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.749811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.793876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":89,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161560+0000","last_change":"2024-03-26T03:48:46.209877+0000","last_active":"2024-03-26T03:49:59.161560+0000","last_peered":"2024-03-26T03:49:59.161560+0000","last_clean":"2024-03-26T03:49:59.161560+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:49:59.161560+0000","last_undegraded":"2024-03-26T03:49:59.161560+0000","last_fullsized":"2024-03-26T03:49:59.161560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.267980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329662+0000","last_change":"2024-03-26T03:48:50.263320+0000","last_active":"2024-03-26T03:49:59.329662+0000","last_peered":"2024-03-26T03:49:59.329662+0000","last_clean":"2024-03-26T03:49:59.329662+0000","last_became_active":"2024-03-26T03:48:50.262776+0000","last_became_peered":"2024-03-26T03:48:50.262776+0000","last_unstale":"2024-03-26T03:49:59.329662+0000","last_undegraded":"2024-03-26T03:49:59.329662+0000","last_fullsized":"2024-03-26T03:49:59.329662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.455736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330181+0000","last_change":"2024-03-26T03:48:48.029617+0000","last_active":"2024-03-26T03:49:59.330181+0000","last_peered":"2024-03-26T03:49:59.330181+0000","last_clean":"2024-03-26T03:49:59.330181+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:49:59.330181+0000","last_undegraded":"2024-03-26T03:49:59.330181+0000","last_fullsized":"2024-03-26T03:49:59.330181+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328963+0000","last_change":"2024-03-26T03:49:52.565054+0000","last_active":"2024-03-26T03:49:59.328963+0000","last_peered":"2024-03-26T03:49:59.328963+0000","last_clean":"2024-03-26T03:49:59.328963+0000","last_became_active":"2024-03-26T03:48:44.014783+0000","last_became_peered":"2024-03-26T03:48:44.014783+0000","last_unstale":"2024-03-26T03:49:59.328963+0000","last_undegraded":"2024-03-26T03:49:59.328963+0000","last_fullsized":"2024-03-26T03:49:59.328963+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.564972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.484718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":79,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.026079+0000","last_change":"2024-03-26T03:50:02.026119+0000","last_active":"2024-03-26T03:50:02.026079+0000","last_peered":"2024-03-26T03:50:02.026079+0000","last_clean":"2024-03-26T03:50:02.026079+0000","last_became_active":"2024-03-26T03:48:46.210541+0000","last_became_peered":"2024-03-26T03:48:46.210541+0000","last_unstale":"2024-03-26T03:50:02.026079+0000","last_undegraded":"2024-03-26T03:50:02.026079+0000","last_fullsized":"2024-03-26T03:50:02.026079+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:50:02.025987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.025987+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.118172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329391+0000","last_change":"2024-03-26T03:49:52.162584+0000","last_active":"2024-03-26T03:49:59.329391+0000","last_peered":"2024-03-26T03:49:59.329391+0000","last_clean":"2024-03-26T03:49:59.329391+0000","last_became_active":"2024-03-26T03:48:50.260705+0000","last_became_peered":"2024-03-26T03:48:50.260705+0000","last_unstale":"2024-03-26T03:49:59.329391+0000","last_undegraded":"2024-03-26T03:49:59.329391+0000","last_fullsized":"2024-03-26T03:49:59.329391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.162478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.162478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:09.131410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":65,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330603+0000","last_change":"2024-03-26T03:48:44.019372+0000","last_active":"2024-03-26T03:49:59.330603+0000","last_peered":"2024-03-26T03:49:59.330603+0000","last_clean":"2024-03-26T03:49:59.330603+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:49:59.330603+0000","last_undegraded":"2024-03-26T03:49:59.330603+0000","last_fullsized":"2024-03-26T03:49:59.330603+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.992376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":98,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328697+0000","last_change":"2024-03-26T03:48:46.210551+0000","last_active":"2024-03-26T03:49:59.328697+0000","last_peered":"2024-03-26T03:49:59.328697+0000","last_clean":"2024-03-26T03:49:59.328697+0000","last_became_active":"2024-03-26T03:48:46.210207+0000","last_became_peered":"2024-03-26T03:48:46.210207+0000","last_unstale":"2024-03-26T03:49:59.328697+0000","last_undegraded":"2024-03-26T03:49:59.328697+0000","last_fullsized":"2024-03-26T03:49:59.328697+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.203264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329877+0000","last_change":"2024-03-26T03:49:54.753786+0000","last_active":"2024-03-26T03:49:59.329877+0000","last_peered":"2024-03-26T03:49:59.329877+0000","last_clean":"2024-03-26T03:49:59.329877+0000","last_became_active":"2024-03-26T03:48:48.027440+0000","last_became_peered":"2024-03-26T03:48:48.027440+0000","last_unstale":"2024-03-26T03:49:59.329877+0000","last_undegraded":"2024-03-26T03:49:59.329877+0000","last_fullsized":"2024-03-26T03:49:59.329877+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:54.753680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.753680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.334605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253582+0000","last_change":"2024-03-26T03:48:50.043541+0000","last_active":"2024-03-26T03:49:59.253582+0000","last_peered":"2024-03-26T03:49:59.253582+0000","last_clean":"2024-03-26T03:49:59.253582+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:49:59.253582+0000","last_undegraded":"2024-03-26T03:49:59.253582+0000","last_fullsized":"2024-03-26T03:49:59.253582+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.195338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330140+0000","last_change":"2024-03-26T03:48:44.017669+0000","last_active":"2024-03-26T03:49:59.330140+0000","last_peered":"2024-03-26T03:49:59.330140+0000","last_clean":"2024-03-26T03:49:59.330140+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:49:59.330140+0000","last_undegraded":"2024-03-26T03:49:59.330140+0000","last_fullsized":"2024-03-26T03:49:59.330140+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.577875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":96,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159657+0000","last_change":"2024-03-26T03:48:46.021263+0000","last_active":"2024-03-26T03:49:59.159657+0000","last_peered":"2024-03-26T03:49:59.159657+0000","last_clean":"2024-03-26T03:49:59.159657+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:49:59.159657+0000","last_undegraded":"2024-03-26T03:49:59.159657+0000","last_fullsized":"2024-03-26T03:49:59.159657+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159607+0000","last_change":"2024-03-26T03:48:48.019097+0000","last_active":"2024-03-26T03:49:59.159607+0000","last_peered":"2024-03-26T03:49:59.159607+0000","last_clean":"2024-03-26T03:49:59.159607+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:49:59.159607+0000","last_undegraded":"2024-03-26T03:49:59.159607+0000","last_fullsized":"2024-03-26T03:49:59.159607+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.081182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328805+0000","last_change":"2024-03-26T03:49:58.807686+0000","last_active":"2024-03-26T03:49:59.328805+0000","last_peered":"2024-03-26T03:49:59.328805+0000","last_clean":"2024-03-26T03:49:59.328805+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:49:59.328805+0000","last_undegraded":"2024-03-26T03:49:59.328805+0000","last_fullsized":"2024-03-26T03:49:59.328805+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.807606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.807606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.096069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160010+0000","last_change":"2024-03-26T03:48:48.018417+0000","last_active":"2024-03-26T03:49:59.160010+0000","last_peered":"2024-03-26T03:49:59.160010+0000","last_clean":"2024-03-26T03:49:59.160010+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:49:59.160010+0000","last_undegraded":"2024-03-26T03:49:59.160010+0000","last_fullsized":"2024-03-26T03:49:59.160010+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.957546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330604+0000","last_change":"2024-03-26T03:48:44.013581+0000","last_active":"2024-03-26T03:49:59.330604+0000","last_peered":"2024-03-26T03:49:59.330604+0000","last_clean":"2024-03-26T03:49:59.330604+0000","last_became_active":"2024-03-26T03:48:44.013295+0000","last_became_peered":"2024-03-26T03:48:44.013295+0000","last_unstale":"2024-03-26T03:49:59.330604+0000","last_undegraded":"2024-03-26T03:49:59.330604+0000","last_fullsized":"2024-03-26T03:49:59.330604+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:04.845697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330072+0000","last_change":"2024-03-26T03:48:46.209925+0000","last_active":"2024-03-26T03:49:59.330072+0000","last_peered":"2024-03-26T03:49:59.330072+0000","last_clean":"2024-03-26T03:49:59.330072+0000","last_became_active":"2024-03-26T03:48:46.209516+0000","last_became_peered":"2024-03-26T03:48:46.209516+0000","last_unstale":"2024-03-26T03:49:59.330072+0000","last_undegraded":"2024-03-26T03:49:59.330072+0000","last_fullsized":"2024-03-26T03:49:59.330072+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:02.460867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159748+0000","last_change":"2024-03-26T03:48:50.040712+0000","last_active":"2024-03-26T03:49:59.159748+0000","last_peered":"2024-03-26T03:49:59.159748+0000","last_clean":"2024-03-26T03:49:59.159748+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:49:59.159748+0000","last_undegraded":"2024-03-26T03:49:59.159748+0000","last_fullsized":"2024-03-26T03:49:59.159748+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.592380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.255830+0000","last_change":"2024-03-26T03:49:59.255850+0000","last_active":"2024-03-26T03:49:59.255830+0000","last_peered":"2024-03-26T03:49:59.255830+0000","last_clean":"2024-03-26T03:49:59.255830+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:49:59.255830+0000","last_undegraded":"2024-03-26T03:49:59.255830+0000","last_fullsized":"2024-03-26T03:49:59.255830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.255770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:16.644194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329906+0000","last_change":"2024-03-26T03:49:57.098936+0000","last_active":"2024-03-26T03:49:59.329906+0000","last_peered":"2024-03-26T03:49:59.329906+0000","last_clean":"2024-03-26T03:49:59.329906+0000","last_became_active":"2024-03-26T03:48:44.016241+0000","last_became_peered":"2024-03-26T03:48:44.016241+0000","last_unstale":"2024-03-26T03:49:59.329906+0000","last_undegraded":"2024-03-26T03:49:59.329906+0000","last_fullsized":"2024-03-26T03:49:59.329906+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.098828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.273950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161398+0000","last_change":"2024-03-26T03:48:46.023925+0000","last_active":"2024-03-26T03:49:59.161398+0000","last_peered":"2024-03-26T03:49:59.161398+0000","last_clean":"2024-03-26T03:49:59.161398+0000","last_became_active":"2024-03-26T03:48:46.023348+0000","last_became_peered":"2024-03-26T03:48:46.023348+0000","last_unstale":"2024-03-26T03:49:59.161398+0000","last_undegraded":"2024-03-26T03:49:59.161398+0000","last_fullsized":"2024-03-26T03:49:59.161398+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253710+0000","last_change":"2024-03-26T03:48:50.044572+0000","last_active":"2024-03-26T03:49:59.253710+0000","last_peered":"2024-03-26T03:49:59.253710+0000","last_clean":"2024-03-26T03:49:59.253710+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:49:59.253710+0000","last_undegraded":"2024-03-26T03:49:59.253710+0000","last_fullsized":"2024-03-26T03:49:59.253710+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.392408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329204+0000","last_change":"2024-03-26T03:48:47.012857+0000","last_active":"2024-03-26T03:49:59.329204+0000","last_peered":"2024-03-26T03:49:59.329204+0000","last_clean":"2024-03-26T03:49:59.329204+0000","last_became_active":"2024-03-26T03:48:43.008452+0000","last_became_peered":"2024-03-26T03:48:43.008452+0000","last_unstale":"2024-03-26T03:49:59.329204+0000","last_undegraded":"2024-03-26T03:49:59.329204+0000","last_fullsized":"2024-03-26T03:49:59.329204+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:48:41.993340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.308748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159473+0000","last_change":"2024-03-26T03:48:44.012922+0000","last_active":"2024-03-26T03:49:59.159473+0000","last_peered":"2024-03-26T03:49:59.159473+0000","last_clean":"2024-03-26T03:49:59.159473+0000","last_became_active":"2024-03-26T03:48:44.012147+0000","last_became_peered":"2024-03-26T03:48:44.012147+0000","last_unstale":"2024-03-26T03:49:59.159473+0000","last_undegraded":"2024-03-26T03:49:59.159473+0000","last_fullsized":"2024-03-26T03:49:59.159473+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.410313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":125,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329182+0000","last_change":"2024-03-26T03:48:46.027084+0000","last_active":"2024-03-26T03:49:59.329182+0000","last_peered":"2024-03-26T03:49:59.329182+0000","last_clean":"2024-03-26T03:49:59.329182+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:49:59.329182+0000","last_undegraded":"2024-03-26T03:49:59.329182+0000","last_fullsized":"2024-03-26T03:49:59.329182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161180+0000","last_change":"2024-03-26T03:48:48.024108+0000","last_active":"2024-03-26T03:49:59.161180+0000","last_peered":"2024-03-26T03:49:59.161180+0000","last_clean":"2024-03-26T03:49:59.161180+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:49:59.161180+0000","last_undegraded":"2024-03-26T03:49:59.161180+0000","last_fullsized":"2024-03-26T03:49:59.161180+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330703+0000","last_change":"2024-03-26T03:48:50.262602+0000","last_active":"2024-03-26T03:49:59.330703+0000","last_peered":"2024-03-26T03:49:59.330703+0000","last_clean":"2024-03-26T03:49:59.330703+0000","last_became_active":"2024-03-26T03:48:50.262294+0000","last_became_peered":"2024-03-26T03:48:50.262294+0000","last_unstale":"2024-03-26T03:49:59.330703+0000","last_undegraded":"2024-03-26T03:49:59.330703+0000","last_fullsized":"2024-03-26T03:49:59.330703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:06.628908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":52,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328621+0000","last_change":"2024-03-26T03:48:44.015006+0000","last_active":"2024-03-26T03:49:59.328621+0000","last_peered":"2024-03-26T03:49:59.328621+0000","last_clean":"2024-03-26T03:49:59.328621+0000","last_became_active":"2024-03-26T03:48:44.014239+0000","last_became_peered":"2024-03-26T03:48:44.014239+0000","last_unstale":"2024-03-26T03:49:59.328621+0000","last_undegraded":"2024-03-26T03:49:59.328621+0000","last_fullsized":"2024-03-26T03:49:59.328621+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159761+0000","last_change":"2024-03-26T03:49:48.295417+0000","last_active":"2024-03-26T03:49:59.159761+0000","last_peered":"2024-03-26T03:49:59.159761+0000","last_clean":"2024-03-26T03:49:59.159761+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:49:59.159761+0000","last_undegraded":"2024-03-26T03:49:59.159761+0000","last_fullsized":"2024-03-26T03:49:59.159761+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.295312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.295312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.045160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329370+0000","last_change":"2024-03-26T03:48:46.019207+0000","last_active":"2024-03-26T03:49:59.329370+0000","last_peered":"2024-03-26T03:49:59.329370+0000","last_clean":"2024-03-26T03:49:59.329370+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:49:59.329370+0000","last_undegraded":"2024-03-26T03:49:59.329370+0000","last_fullsized":"2024-03-26T03:49:59.329370+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.796465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160754+0000","last_change":"2024-03-26T03:48:48.020655+0000","last_active":"2024-03-26T03:49:59.160754+0000","last_peered":"2024-03-26T03:49:59.160754+0000","last_clean":"2024-03-26T03:49:59.160754+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:49:59.160754+0000","last_undegraded":"2024-03-26T03:49:59.160754+0000","last_fullsized":"2024-03-26T03:49:59.160754+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:15.127534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330371+0000","last_change":"2024-03-26T03:48:50.039906+0000","last_active":"2024-03-26T03:49:59.330371+0000","last_peered":"2024-03-26T03:49:59.330371+0000","last_clean":"2024-03-26T03:49:59.330371+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:49:59.330371+0000","last_undegraded":"2024-03-26T03:49:59.330371+0000","last_fullsized":"2024-03-26T03:49:59.330371+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.037330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":301,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329014+0000","last_change":"2024-03-26T03:48:51.651891+0000","last_active":"2024-03-26T03:49:59.329014+0000","last_peered":"2024-03-26T03:49:59.329014+0000","last_clean":"2024-03-26T03:49:59.329014+0000","last_became_active":"2024-03-26T03:48:34.401623+0000","last_became_peered":"2024-03-26T03:48:34.401623+0000","last_unstale":"2024-03-26T03:49:59.329014+0000","last_undegraded":"2024-03-26T03:49:59.329014+0000","last_fullsized":"2024-03-26T03:49:59.329014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T03:48:51.651774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:48:51.651774+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:20.226983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253530+0000","last_change":"2024-03-26T03:48:44.017929+0000","last_active":"2024-03-26T03:49:59.253530+0000","last_peered":"2024-03-26T03:49:59.253530+0000","last_clean":"2024-03-26T03:49:59.253530+0000","last_became_active":"2024-03-26T03:48:44.017637+0000","last_became_peered":"2024-03-26T03:48:44.017637+0000","last_unstale":"2024-03-26T03:49:59.253530+0000","last_undegraded":"2024-03-26T03:49:59.253530+0000","last_fullsized":"2024-03-26T03:49:59.253530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.329800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":88,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329894+0000","last_change":"2024-03-26T03:48:46.210606+0000","last_active":"2024-03-26T03:49:59.329894+0000","last_peered":"2024-03-26T03:49:59.329894+0000","last_clean":"2024-03-26T03:49:59.329894+0000","last_became_active":"2024-03-26T03:48:46.210282+0000","last_became_peered":"2024-03-26T03:48:46.210282+0000","last_unstale":"2024-03-26T03:49:59.329894+0000","last_undegraded":"2024-03-26T03:49:59.329894+0000","last_fullsized":"2024-03-26T03:49:59.329894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:00.772718+0000","last_change":"2024-03-26T03:50:00.772766+0000","last_active":"2024-03-26T03:50:00.772718+0000","last_peered":"2024-03-26T03:50:00.772718+0000","last_clean":"2024-03-26T03:50:00.772718+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:50:00.772718+0000","last_undegraded":"2024-03-26T03:50:00.772718+0000","last_fullsized":"2024-03-26T03:50:00.772718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.772625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.974412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328900+0000","last_change":"2024-03-26T03:48:50.043668+0000","last_active":"2024-03-26T03:49:59.328900+0000","last_peered":"2024-03-26T03:49:59.328900+0000","last_clean":"2024-03-26T03:49:59.328900+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:49:59.328900+0000","last_undegraded":"2024-03-26T03:49:59.328900+0000","last_fullsized":"2024-03-26T03:49:59.328900+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:03.526812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160640+0000","last_change":"2024-03-26T03:48:44.011514+0000","last_active":"2024-03-26T03:49:59.160640+0000","last_peered":"2024-03-26T03:49:59.160640+0000","last_clean":"2024-03-26T03:49:59.160640+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:49:59.160640+0000","last_undegraded":"2024-03-26T03:49:59.160640+0000","last_fullsized":"2024-03-26T03:49:59.160640+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":146,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.917981+0000","last_change":"2024-03-26T03:49:45.618585+0000","last_active":"2024-03-26T03:50:02.917981+0000","last_peered":"2024-03-26T03:50:02.917981+0000","last_clean":"2024-03-26T03:50:02.917981+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:50:02.917981+0000","last_undegraded":"2024-03-26T03:50:02.917981+0000","last_fullsized":"2024-03-26T03:50:02.917981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:49:45.618470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:45.618470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.990274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":67,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.161390+0000","last_change":"2024-03-26T03:48:46.026122+0000","last_active":"2024-03-26T03:49:59.161390+0000","last_peered":"2024-03-26T03:49:59.161390+0000","last_clean":"2024-03-26T03:49:59.161390+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:49:59.161390+0000","last_undegraded":"2024-03-26T03:49:59.161390+0000","last_fullsized":"2024-03-26T03:49:59.161390+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:49:59.218762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330733+0000","last_change":"2024-03-26T03:48:48.022621+0000","last_active":"2024-03-26T03:49:59.330733+0000","last_peered":"2024-03-26T03:49:59.330733+0000","last_clean":"2024-03-26T03:49:59.330733+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:49:59.330733+0000","last_undegraded":"2024-03-26T03:49:59.330733+0000","last_fullsized":"2024-03-26T03:49:59.330733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.069511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":40,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328518+0000","last_change":"2024-03-26T03:48:50.260878+0000","last_active":"2024-03-26T03:49:59.328518+0000","last_peered":"2024-03-26T03:49:59.328518+0000","last_clean":"2024-03-26T03:49:59.328518+0000","last_became_active":"2024-03-26T03:48:50.260476+0000","last_became_peered":"2024-03-26T03:48:50.260476+0000","last_unstale":"2024-03-26T03:49:59.328518+0000","last_undegraded":"2024-03-26T03:49:59.328518+0000","last_fullsized":"2024-03-26T03:49:59.328518+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:00.412999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":60,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:00.032849+0000","last_change":"2024-03-26T03:50:00.032886+0000","last_active":"2024-03-26T03:50:00.032849+0000","last_peered":"2024-03-26T03:50:00.032849+0000","last_clean":"2024-03-26T03:50:00.032849+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:50:00.032849+0000","last_undegraded":"2024-03-26T03:50:00.032849+0000","last_fullsized":"2024-03-26T03:50:00.032849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.032763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.032763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:06.902858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":105,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:03.113076+0000","last_change":"2024-03-26T03:50:03.113111+0000","last_active":"2024-03-26T03:50:03.113076+0000","last_peered":"2024-03-26T03:50:03.113076+0000","last_clean":"2024-03-26T03:50:03.113076+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:50:03.113076+0000","last_undegraded":"2024-03-26T03:50:03.113076+0000","last_fullsized":"2024-03-26T03:50:03.113076+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:03.112983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.112983+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.126914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330164+0000","last_change":"2024-03-26T03:49:54.591020+0000","last_active":"2024-03-26T03:49:59.330164+0000","last_peered":"2024-03-26T03:49:59.330164+0000","last_clean":"2024-03-26T03:49:59.330164+0000","last_became_active":"2024-03-26T03:48:48.021249+0000","last_became_peered":"2024-03-26T03:48:48.021249+0000","last_unstale":"2024-03-26T03:49:59.330164+0000","last_undegraded":"2024-03-26T03:49:59.330164+0000","last_fullsized":"2024-03-26T03:49:59.330164+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:54.590915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.590915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.886951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159829+0000","last_change":"2024-03-26T03:48:50.045296+0000","last_active":"2024-03-26T03:49:59.159829+0000","last_peered":"2024-03-26T03:49:59.159829+0000","last_clean":"2024-03-26T03:49:59.159829+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:49:59.159829+0000","last_undegraded":"2024-03-26T03:49:59.159829+0000","last_fullsized":"2024-03-26T03:49:59.159829+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:16.928066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330124+0000","last_change":"2024-03-26T03:48:44.014959+0000","last_active":"2024-03-26T03:49:59.330124+0000","last_peered":"2024-03-26T03:49:59.330124+0000","last_clean":"2024-03-26T03:49:59.330124+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:49:59.330124+0000","last_undegraded":"2024-03-26T03:49:59.330124+0000","last_fullsized":"2024-03-26T03:49:59.330124+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.457323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":83,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.328889+0000","last_change":"2024-03-26T03:49:46.927333+0000","last_active":"2024-03-26T03:49:59.328889+0000","last_peered":"2024-03-26T03:49:59.328889+0000","last_clean":"2024-03-26T03:49:59.328889+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:49:59.328889+0000","last_undegraded":"2024-03-26T03:49:59.328889+0000","last_fullsized":"2024-03-26T03:49:59.328889+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:49:46.927230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:46.927230+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:54.304888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160961+0000","last_change":"2024-03-26T03:48:48.019177+0000","last_active":"2024-03-26T03:49:59.160961+0000","last_peered":"2024-03-26T03:49:59.160961+0000","last_clean":"2024-03-26T03:49:59.160961+0000","last_became_active":"2024-03-26T03:48:48.018852+0000","last_became_peered":"2024-03-26T03:48:48.018852+0000","last_unstale":"2024-03-26T03:49:59.160961+0000","last_undegraded":"2024-03-26T03:49:59.160961+0000","last_fullsized":"2024-03-26T03:49:59.160961+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.916370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330891+0000","last_change":"2024-03-26T03:49:57.800214+0000","last_active":"2024-03-26T03:49:59.330891+0000","last_peered":"2024-03-26T03:49:59.330891+0000","last_clean":"2024-03-26T03:49:59.330891+0000","last_became_active":"2024-03-26T03:48:50.262788+0000","last_became_peered":"2024-03-26T03:48:50.262788+0000","last_unstale":"2024-03-26T03:49:59.330891+0000","last_undegraded":"2024-03-26T03:49:59.330891+0000","last_fullsized":"2024-03-26T03:49:59.330891+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.800113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.800113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:00.814334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":62,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.092005+0000","last_change":"2024-03-26T03:50:02.092039+0000","last_active":"2024-03-26T03:50:02.092005+0000","last_peered":"2024-03-26T03:50:02.092005+0000","last_clean":"2024-03-26T03:50:02.092005+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:50:02.092005+0000","last_undegraded":"2024-03-26T03:50:02.092005+0000","last_fullsized":"2024-03-26T03:50:02.092005+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:02.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.091909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.088925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159522+0000","last_change":"2024-03-26T03:48:46.017035+0000","last_active":"2024-03-26T03:49:59.159522+0000","last_peered":"2024-03-26T03:49:59.159522+0000","last_clean":"2024-03-26T03:49:59.159522+0000","last_became_active":"2024-03-26T03:48:46.016634+0000","last_became_peered":"2024-03-26T03:48:46.016634+0000","last_unstale":"2024-03-26T03:49:59.159522+0000","last_undegraded":"2024-03-26T03:49:59.159522+0000","last_fullsized":"2024-03-26T03:49:59.159522+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253669+0000","last_change":"2024-03-26T03:48:48.028663+0000","last_active":"2024-03-26T03:49:59.253669+0000","last_peered":"2024-03-26T03:49:59.253669+0000","last_clean":"2024-03-26T03:49:59.253669+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:49:59.253669+0000","last_undegraded":"2024-03-26T03:49:59.253669+0000","last_fullsized":"2024-03-26T03:49:59.253669+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159432+0000","last_change":"2024-03-26T03:48:50.043708+0000","last_active":"2024-03-26T03:49:59.159432+0000","last_peered":"2024-03-26T03:49:59.159432+0000","last_clean":"2024-03-26T03:49:59.159432+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:49:59.159432+0000","last_undegraded":"2024-03-26T03:49:59.159432+0000","last_fullsized":"2024-03-26T03:49:59.159432+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.254153+0000","last_change":"2024-03-26T03:48:44.013652+0000","last_active":"2024-03-26T03:49:59.254153+0000","last_peered":"2024-03-26T03:49:59.254153+0000","last_clean":"2024-03-26T03:49:59.254153+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:49:59.254153+0000","last_undegraded":"2024-03-26T03:49:59.254153+0000","last_fullsized":"2024-03-26T03:49:59.254153+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":123,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253377+0000","last_change":"2024-03-26T03:49:51.111438+0000","last_active":"2024-03-26T03:49:59.253377+0000","last_peered":"2024-03-26T03:49:59.253377+0000","last_clean":"2024-03-26T03:49:59.253377+0000","last_became_active":"2024-03-26T03:48:46.209319+0000","last_became_peered":"2024-03-26T03:48:46.209319+0000","last_unstale":"2024-03-26T03:49:59.253377+0000","last_undegraded":"2024-03-26T03:49:59.253377+0000","last_fullsized":"2024-03-26T03:49:59.253377+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.111316+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.974577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":45,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159262+0000","last_change":"2024-03-26T03:48:48.029521+0000","last_active":"2024-03-26T03:49:59.159262+0000","last_peered":"2024-03-26T03:49:59.159262+0000","last_clean":"2024-03-26T03:49:59.159262+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:49:59.159262+0000","last_undegraded":"2024-03-26T03:49:59.159262+0000","last_fullsized":"2024-03-26T03:49:59.159262+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329859+0000","last_change":"2024-03-26T03:48:50.034819+0000","last_active":"2024-03-26T03:49:59.329859+0000","last_peered":"2024-03-26T03:49:59.329859+0000","last_clean":"2024-03-26T03:49:59.329859+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:49:59.329859+0000","last_undegraded":"2024-03-26T03:49:59.329859+0000","last_fullsized":"2024-03-26T03:49:59.329859+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:07.402354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":70,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160400+0000","last_change":"2024-03-26T03:48:48.020914+0000","last_active":"2024-03-26T03:49:59.160400+0000","last_peered":"2024-03-26T03:49:59.160400+0000","last_clean":"2024-03-26T03:49:59.160400+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:49:59.160400+0000","last_undegraded":"2024-03-26T03:49:59.160400+0000","last_fullsized":"2024-03-26T03:49:59.160400+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253433+0000","last_change":"2024-03-26T03:48:44.017015+0000","last_active":"2024-03-26T03:49:59.253433+0000","last_peered":"2024-03-26T03:49:59.253433+0000","last_clean":"2024-03-26T03:49:59.253433+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:49:59.253433+0000","last_undegraded":"2024-03-26T03:49:59.253433+0000","last_fullsized":"2024-03-26T03:49:59.253433+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:08.476078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":73,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329362+0000","last_change":"2024-03-26T03:48:46.026556+0000","last_active":"2024-03-26T03:49:59.329362+0000","last_peered":"2024-03-26T03:49:59.329362+0000","last_clean":"2024-03-26T03:49:59.329362+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:49:59.329362+0000","last_undegraded":"2024-03-26T03:49:59.329362+0000","last_fullsized":"2024-03-26T03:49:59.329362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253539+0000","last_change":"2024-03-26T03:48:50.261659+0000","last_active":"2024-03-26T03:49:59.253539+0000","last_peered":"2024-03-26T03:49:59.253539+0000","last_clean":"2024-03-26T03:49:59.253539+0000","last_became_active":"2024-03-26T03:48:50.260697+0000","last_became_peered":"2024-03-26T03:48:50.260697+0000","last_unstale":"2024-03-26T03:49:59.253539+0000","last_undegraded":"2024-03-26T03:49:59.253539+0000","last_fullsized":"2024-03-26T03:49:59.253539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:03.983242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.189050+0000","last_change":"2024-03-26T03:50:02.189077+0000","last_active":"2024-03-26T03:50:02.189050+0000","last_peered":"2024-03-26T03:50:02.189050+0000","last_clean":"2024-03-26T03:50:02.189050+0000","last_became_active":"2024-03-26T03:48:48.019448+0000","last_became_peered":"2024-03-26T03:48:48.019448+0000","last_unstale":"2024-03-26T03:50:02.189050+0000","last_undegraded":"2024-03-26T03:50:02.189050+0000","last_fullsized":"2024-03-26T03:50:02.189050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.188975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.969923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159264+0000","last_change":"2024-03-26T03:49:48.865134+0000","last_active":"2024-03-26T03:49:59.159264+0000","last_peered":"2024-03-26T03:49:59.159264+0000","last_clean":"2024-03-26T03:49:59.159264+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:49:59.159264+0000","last_undegraded":"2024-03-26T03:49:59.159264+0000","last_fullsized":"2024-03-26T03:49:59.159264+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.864997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.477744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160163+0000","last_change":"2024-03-26T03:48:46.019214+0000","last_active":"2024-03-26T03:49:59.160163+0000","last_peered":"2024-03-26T03:49:59.160163+0000","last_clean":"2024-03-26T03:49:59.160163+0000","last_became_active":"2024-03-26T03:48:46.018601+0000","last_became_peered":"2024-03-26T03:48:46.018601+0000","last_unstale":"2024-03-26T03:49:59.160163+0000","last_undegraded":"2024-03-26T03:49:59.160163+0000","last_fullsized":"2024-03-26T03:49:59.160163+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":41,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.160965+0000","last_change":"2024-03-26T03:48:50.044134+0000","last_active":"2024-03-26T03:49:59.160965+0000","last_peered":"2024-03-26T03:49:59.160965+0000","last_clean":"2024-03-26T03:49:59.160965+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:49:59.160965+0000","last_undegraded":"2024-03-26T03:49:59.160965+0000","last_fullsized":"2024-03-26T03:49:59.160965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:01.818024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.329349+0000","last_change":"2024-03-26T03:48:48.026964+0000","last_active":"2024-03-26T03:49:59.329349+0000","last_peered":"2024-03-26T03:49:59.329349+0000","last_clean":"2024-03-26T03:49:59.329349+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:49:59.329349+0000","last_undegraded":"2024-03-26T03:49:59.329349+0000","last_fullsized":"2024-03-26T03:49:59.329349+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":61,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.330207+0000","last_change":"2024-03-26T03:49:51.516987+0000","last_active":"2024-03-26T03:49:59.330207+0000","last_peered":"2024-03-26T03:49:59.330207+0000","last_clean":"2024-03-26T03:49:59.330207+0000","last_became_active":"2024-03-26T03:48:44.012660+0000","last_became_peered":"2024-03-26T03:48:44.012660+0000","last_unstale":"2024-03-26T03:49:59.330207+0000","last_undegraded":"2024-03-26T03:49:59.330207+0000","last_fullsized":"2024-03-26T03:49:59.330207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:51.516891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.516891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:57.433207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:03.880793+0000","last_change":"2024-03-26T03:50:03.880855+0000","last_active":"2024-03-26T03:50:03.880793+0000","last_peered":"2024-03-26T03:50:03.880793+0000","last_clean":"2024-03-26T03:50:03.880793+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:50:03.880793+0000","last_undegraded":"2024-03-26T03:50:03.880793+0000","last_fullsized":"2024-03-26T03:50:03.880793+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:03.880711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.880711+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.791446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.877459+0000","last_change":"2024-03-26T03:50:02.877487+0000","last_active":"2024-03-26T03:50:02.877459+0000","last_peered":"2024-03-26T03:50:02.877459+0000","last_clean":"2024-03-26T03:50:02.877459+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:50:02.877459+0000","last_undegraded":"2024-03-26T03:50:02.877459+0000","last_fullsized":"2024-03-26T03:50:02.877459+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.877373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.877373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.008619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":78,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:50:02.300700+0000","last_change":"2024-03-26T03:50:02.300700+0000","last_active":"2024-03-26T03:50:02.300700+0000","last_peered":"2024-03-26T03:50:02.300700+0000","last_clean":"2024-03-26T03:50:02.300700+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:50:02.300700+0000","last_undegraded":"2024-03-26T03:50:02.300700+0000","last_fullsized":"2024-03-26T03:50:02.300700+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:02.300594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.300594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.021817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":53,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.253820+0000","last_change":"2024-03-26T03:48:44.015913+0000","last_active":"2024-03-26T03:49:59.253820+0000","last_peered":"2024-03-26T03:49:59.253820+0000","last_clean":"2024-03-26T03:49:59.253820+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:49:59.253820+0000","last_undegraded":"2024-03-26T03:49:59.253820+0000","last_fullsized":"2024-03-26T03:49:59.253820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.708831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":64,"reported_epoch":87,"state":"active+clean","last_fresh":"2024-03-26T03:49:59.159773+0000","last_change":"2024-03-26T03:48:46.023356+0000","last_active":"2024-03-26T03:49:59.159773+0000","last_peered":"2024-03-26T03:49:59.159773+0000","last_clean":"2024-03-26T03:49:59.159773+0000","last_became_active":"2024-03-26T03:48:46.022937+0000","last_became_peered":"2024-03-26T03:48:46.022937+0000","last_unstale":"2024-03-26T03:49:59.159773+0000","last_undegraded":"2024-03-26T03:49:59.159773+0000","last_fullsized":"2024-03-26T03:49:59.159773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.912372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":599,"ondisk_log_size":599,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070484,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28380,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716772,"statfs":{"total":95995035648,"available":95965974528,"internally_reserved":0,"allocated":1396736,"data_stored":935613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32512,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712640,"statfs":{"total":95995035648,"available":95961743360,"internally_reserved":0,"allocated":1363968,"data_stored":933201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":344618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32540,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712612,"statfs":{"total":95995035648,"available":95961714688,"internally_reserved":0,"allocated":1396736,"data_stored":935342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31928,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713224,"statfs":{"total":95995035648,"available":95962341376,"internally_reserved":0,"allocated":770048,"data_stored":342961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":815104,"data_stored":345071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27800,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717352,"statfs":{"total":95995035648,"available":95966568448,"internally_reserved":0,"allocated":802816,"data_stored":343451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659165,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27772,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717380,"statfs":{"total":95995035648,"available":95966597120,"internally_reserved":0,"allocated":774144,"data_stored":343159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:50:09.106 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T03:50:09.106 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T03:50:09.106 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T03:50:09.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T03:50:09.357 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[37501]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T03:50:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:09 smithi006 ceph-mon[29446]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:09.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:09 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T03:50:09.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:09 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T03:50:09.665 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:09 smithi086 ceph-mon[32965]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:50:10.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[37501]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:50:10.527 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:10 smithi006 ceph-mon[29446]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:50:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T03:50:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T03:50:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T03:50:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T03:50:10.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:10.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:50:10.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:50:10.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:10 smithi086 ceph-mon[32965]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T03:50:11.276 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:50:11.276 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T03:50:11.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:11 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T03:50:11.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:11 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T03:50:11.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:11 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T03:50:11.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:11 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T03:50:11.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1166639633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1166639633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T03:50:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1166639633' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T03:50:11.953 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T03:50:11.954 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T03:50:11.954 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-26T03:50:11.965 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T03:50:11.965 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-adk-testing-2024-03-25-1628'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'MON_DOWN', 'mons down, quorum', 'is down \\(out of quorum\\)', 'OSDMAP_FLAGS', 'PG_DEGRADED', 'OSD_DOWN', 'POOL_APP_NOT_ENABLED', 'CACHE_POOL_NO_HIT_SET', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-adk-testing-2024-03-25-1628', 'sha1': 'e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003'}} 2024-03-26T03:50:11.965 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-26T03:50:11.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-26T03:50:12.216 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:12.563 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:12 smithi086 ceph-mon[32965]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:12.563 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:12 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T03:50:12.564 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:12 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[37501]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[29446]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T03:50:12.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:12 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:50:13.566 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:50:13.567 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:50:13.586 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:50:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:13 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:50:13.838 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:13 smithi086 systemd[1]: Starting Ceph grafana.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:13.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:13.575Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003705246s 2024-03-26T03:50:14.312 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE Bus STOPPING 2024-03-26T03:50:14.312 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:50:14.312 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE Bus STOPPED 2024-03-26T03:50:14.312 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE Bus STARTING 2024-03-26T03:50:14.312 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE Serving on http://:::9283 2024-03-26T03:50:14.313 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:14] ENGINE Bus STARTED 2024-03-26T03:50:14.325 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:14.325 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:13 smithi086 podman[64106]: 2024-03-26 03:50:13.882689618 +0000 UTC m=+0.018564059 image pull 2bacad6d85d802cd176c146528adcc0e47a8a83e94bc4a8b7ba1a9b9ef2c65e7 quay.io/ceph/grafana:9.4.12 2024-03-26T03:50:14.325 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:14 smithi086 podman[64106]: 2024-03-26 03:50:14.002909833 +0000 UTC m=+0.138784263 container create bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T03:50:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' 2024-03-26T03:50:14.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:14 smithi086 ceph-mon[32965]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:50:14.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:50:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:50:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T03:50:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T03:50:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[37501]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T03:50:14.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' 2024-03-26T03:50:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:14 smithi006 ceph-mon[29446]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T03:50:14.692 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-26T03:50:14.692 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T03:50:14.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:50:14.937 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T03:50:15.493 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T03:50:15.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T03:50:15.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T03:50:15.494 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:15.494 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 podman[64106]: 2024-03-26 03:50:15.242695166 +0000 UTC m=+1.378569594 container init bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:50:15.494 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 podman[64106]: 2024-03-26 03:50:15.246540674 +0000 UTC m=+1.382415102 container start bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T03:50:15.494 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 bash[64106]: bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 2024-03-26T03:50:15.494 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 systemd[1]: Started Ceph grafana.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401709917Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401894086Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.40190879Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401914059Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401918157Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401922107Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401925707Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401929548Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401933153Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401937609Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401941699Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.40194628Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401950356Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T03:50:15.495 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401954641Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401958839Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401962679Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401966488Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.401970412Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=settings t=2024-03-26T03:50:15.40197492Z level=info msg="App mode production" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=sqlstore t=2024-03-26T03:50:15.402028647Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=sqlstore t=2024-03-26T03:50:15.402045688Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.403317664Z level=info msg="Starting DB migrations" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.403410861Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.403962725Z level=info msg="Executing migration" id="create user table" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.404384169Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.40521773Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.406096556Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.406946608Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T03:50:15.496 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.407827266Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.410824526Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.413664841Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.414150784Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.417076455Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.419860366Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.422669305Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.425463462Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T03:50:15.497 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.428242858Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.431101698Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.433898369Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.436807378Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.439725503Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.442487809Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.445287345Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.447967896Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.450895057Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.45367365Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.456635691Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.461761267Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.464347821Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T03:50:15.498 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.464753284Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.467529372Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.470298895Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.47296553Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.473762786Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.476515033Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.47932114Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.482004078Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.482395499Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.485050071Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.487852328Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.490477083Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.493271584Z level=info msg="Executing migration" id="create star table" 2024-03-26T03:50:15.499 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.49579448Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T03:50:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T03:50:15.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.498554348Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.499437739Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.50028622Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.501105129Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.501960386Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.50281193Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.503598025Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T03:50:15.744 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.504425174Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.504724295Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.505307716Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.506161631Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.507009619Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.507904394Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.50868582Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.509503362Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.510327019Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.511446752Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.512313869Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T03:50:15.745 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.513106007Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.51393231Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.514817433Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.515673299Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.516221138Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.517060062Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.517918837Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.51876433Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.521704057Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.524564858Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.527349923Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T03:50:15.747 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.530126815Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.532966787Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.535651331Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.538478733Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.541312791Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.544003616Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.546931166Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.549694427Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.55268005Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.555494019Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.558405511Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.560974821Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.563704584Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.566519312Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T03:50:15.748 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.569249117Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.571961912Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.574570485Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.577334272Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.579945209Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.582501834Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.585166139Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.587807252Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.590592016Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.593280419Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.59618091Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.598900574Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.601609118Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.604320014Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T03:50:15.749 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.606999305Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.609652461Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.612484387Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.615109903Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.615953872Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.616781789Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.617629228Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.618484493Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.619242191Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.620107098Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.620929492Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.621790541Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T03:50:15.750 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.622640594Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.62350752Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.624270872Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.625164254Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.625984966Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.626873805Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.62771963Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.628539926Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.629378672Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.630234115Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.631074179Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.631926432Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.632353838Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.633687149Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T03:50:15.751 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.634528453Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.635372419Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.636160875Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.637081686Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.637928059Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.638766673Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.639142041Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.639991907Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.640875861Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.641778429Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.642404758Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.643250308Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.643643534Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.644102982Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T03:50:15.752 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.646832222Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.647271197Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.650101328Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.652760449Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.655731538Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.658550713Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.661311749Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.664006817Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.666868022Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.667270023Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.670254102Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.673014695Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.675743338Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.676180548Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T03:50:15.753 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.678972553Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.68179293Z level=info msg="Executing migration" id="create session table" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.684552559Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.684871337Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.687530483Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.690177994Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.692872783Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.69317213Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.695727921Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.696062542Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.698530482Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.701334799Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.70172487Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.704498423Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.707093072Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T03:50:15.754 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.709895816Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.710526837Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.713209734Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.713794753Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.716011761Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.716808209Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.717248879Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.718080762Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.718875136Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.719715068Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.720167622Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.720973776Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.722648669Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T03:50:15.755 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.72348395Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.723910973Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.724730288Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.725609513Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.726430974Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.727177195Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.727950442Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.728761015Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.729579195Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.73003246Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.730896647Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.731745356Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.732624635Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.733083355Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T03:50:15.756 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.733835797Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.734608806Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.735423338Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.736162213Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.736980028Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.73779844Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.738635708Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.73913162Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.739978154Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.740872546Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T03:50:15.757 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.741710887Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.742135967Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.742985944Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.743826666Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.744691647Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.745502207Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.746363959Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T03:50:15.758 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.747208393Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.748094415Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.748432236Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.749374937Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.750203112Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.751145839Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.751589174Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.754445874Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.757387486Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.760156179Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.760535471Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T03:50:15.996 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.763217784Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.766065246Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.768764611Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.76911245Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.77192433Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.774871823Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.777772951Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T03:50:15.997 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.778209911Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.780968568Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.783744652Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.78664074Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.786957857Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.789788265Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.792469889Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.795247972Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.795668412Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.798468737Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.801182601Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.803883792Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.804192349Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T03:50:15.998 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.806954761Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.809664083Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.812437367Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.815222211Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.817866003Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.820625477Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.823279012Z level=info msg="Executing migration" id="create team table" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.823664778Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.826363144Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.829032092Z level=info msg="Executing migration" id="create team member table" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.82990493Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.830316536Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.831196072Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.832008974Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T03:50:15.999 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.832857709Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.833692492Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.834564661Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.835416111Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.836197527Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.836631915Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.837493363Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.838294197Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.839065348Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.839918363Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.840767551Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.841653726Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.842467301Z level=info msg="Executing migration" id="create tag table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.843358671Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.844153483Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T03:50:16.000 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.845008388Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.845865637Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.846728323Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.849097598Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.849946876Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.850795247Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.851661858Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.852524201Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.853351823Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.854332887Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.855148538Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.856004395Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.856865454Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.857738824Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.858675334Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T03:50:16.001 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.859556985Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T03:50:16.002 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.862234927Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T03:50:16.002 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.865128328Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T03:50:16.002 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.867847242Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T03:50:16.002 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.870781843Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.873667408Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.876502802Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.87938366Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.882134079Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.884978991Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.887855549Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.89072459Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.893628323Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.896456073Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.899315363Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T03:50:16.003 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.902288983Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.905147847Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.908012342Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.910859623Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.913577871Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.916372824Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.919008297Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.921794072Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.924533894Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.927195526Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.929945783Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T03:50:16.004 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.932679546Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T03:50:16.005 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.935432839Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T03:50:16.005 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.93818252Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T03:50:16.005 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.940966477Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T03:50:16.005 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.943645458Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.946415475Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.949041657Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.951746803Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.954329552Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.954944773Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T03:50:16.006 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.955759249Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.960205725Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.96458395Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.965438398Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.966318849Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.967314404Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.968111683Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.968956957Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.969809888Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.97065983Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.971474435Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.972471293Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.973429549Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T03:50:16.007 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.974379108Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.975224972Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.976104638Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.977070239Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.977998192Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.978895779Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.979920398Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.980828044Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.981713266Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.982554739Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.983434566Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.984325964Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.9871595Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.989881039Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T03:50:16.008 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.992532123Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T03:50:16.009 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.995283405Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T03:50:16.009 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:15.99803794Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T03:50:16.247 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.001040884Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.003646033Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.006618221Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.009318021Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.012235835Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.014991965Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T03:50:16.248 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.017925045Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.020789959Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.023637321Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.02661024Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.02939604Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.032173761Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.034925417Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.037629712Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.040296347Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.04302981Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.045689501Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T03:50:16.249 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.048158135Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T03:50:16.250 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.048247562Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.050831607Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.053443634Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.056210391Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.056350721Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.05646602Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.058970313Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.061617157Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.064357646Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.067021657Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.06980798Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.072625286Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.075269475Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.077960106Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T03:50:16.251 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.080671681Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.081510506Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.087058511Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.08812403Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.088934694Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.094636595Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.100283455Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.101098049Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.101945077Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.102795637Z level=info msg="Executing migration" id="create permission table" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.103652595Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.104467375Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.105355515Z level=info msg="Executing migration" id="create role table" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.106224936Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.107298555Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T03:50:16.252 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.108372928Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.109191986Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.110048155Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.110905761Z level=info msg="Executing migration" id="create team role table" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.111757653Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.112606281Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.113464584Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.114352237Z level=info msg="Executing migration" id="create user role table" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.115224607Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.11604792Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.11693857Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.117787907Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.118675801Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.119550907Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T03:50:16.253 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.120439862Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.121588144Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.122503799Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.123360608Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.124206826Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.125073763Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.125939795Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.128783817Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.131529542Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T03:50:16.254 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.134288248Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.137175456Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.13992727Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.142609261Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.145392367Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.148248648Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.150893432Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.153826466Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.156492698Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.159474538Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.16229079Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.165204409Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.168194067Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.170982067Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.173731188Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T03:50:16.255 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.176308029Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.179018285Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.181752154Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.181846956Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.184461452Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.184548305Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.18706534Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.189630135Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.192277272Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.19502126Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.197705019Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.200350609Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.202937922Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.20562471Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.208303728Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T03:50:16.256 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.210990075Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.213865732Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.216545677Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.2203745Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.222946674Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.225696541Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.22840476Z level=info msg="Executing migration" id="add share column" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.231101708Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.233923571Z level=info msg="Executing migration" id="create file table" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.236591314Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.239416717Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.242092899Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.244795042Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.247489073Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T03:50:16.257 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.25022529Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T03:50:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: Deploying daemon node-exporter.a on smithi006 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: 6.14 deep-scrub starts 2024-03-26T03:50:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-mon[32965]: 6.14 deep-scrub ok 2024-03-26T03:50:16.588 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.251097608Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.251953468Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.252810653Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.254602972Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.255464166Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.256249066Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.257077514Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.257857512Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.258643191Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.259792857Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.260965603Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.261835923Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T03:50:16.589 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.274776637Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.275554767Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.276373094Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.280651965Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.281470526Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.282350151Z level=info msg="Executing migration" id="create folder table" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.283147596Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.284022169Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.284879005Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.285762689Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=migrator t=2024-03-26T03:50:16.286608461Z level=info msg="migrations completed" performed=485 skipped=0 duration=883.232036ms 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=sqlstore t=2024-03-26T03:50:16.28701713Z level=info msg="Created default organization" 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=plugin.finder t=2024-03-26T03:50:16.319616411Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=secrets t=2024-03-26T03:50:16.319717222Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T03:50:16.590 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=query_data t=2024-03-26T03:50:16.321998085Z level=info msg="Query Service initialization" 2024-03-26T03:50:16.591 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=live.push_http t=2024-03-26T03:50:16.326137393Z level=info msg="Live Push Gateway initialization" 2024-03-26T03:50:16.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: Deploying daemon node-exporter.a on smithi006 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: 6.14 deep-scrub starts 2024-03-26T03:50:16.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[37501]: 6.14 deep-scrub ok 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:16.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: Deploying daemon node-exporter.a on smithi006 2024-03-26T03:50:16.854 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: 6.14 deep-scrub starts 2024-03-26T03:50:16.854 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:16 smithi006 ceph-mon[29446]: 6.14 deep-scrub ok 2024-03-26T03:50:16.854 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:16 smithi006 systemd[1]: Starting Ceph node-exporter.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:16.890 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:50:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:50:17.140 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:16 smithi006 bash[72485]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=infra.usagestats.collector t=2024-03-26T03:50:16.942144525Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=provisioning.datasources t=2024-03-26T03:50:16.943157097Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=provisioning.datasources t=2024-03-26T03:50:16.954155915Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=provisioning.alerting t=2024-03-26T03:50:16.964838629Z level=info msg="starting to provision alerting" 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=provisioning.alerting t=2024-03-26T03:50:16.964862012Z level=info msg="finished to provision alerting" 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=grafanaStorageLogger t=2024-03-26T03:50:16.965011953Z level=info msg="storage starting" 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=ngalert.state.manager t=2024-03-26T03:50:16.965031088Z level=info msg="Warming state cache for startup" 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=http.server t=2024-03-26T03:50:16.967293114Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T03:50:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=ngalert.state.manager t=2024-03-26T03:50:16.970779092Z level=info msg="State cache has been initialized" states=0 duration=5.747562ms 2024-03-26T03:50:17.339 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=ticker t=2024-03-26T03:50:16.970837532Z level=info msg=starting first_tick=2024-03-26T03:50:20Z 2024-03-26T03:50:17.339 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 03:50:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=ngalert.multiorg.alertmanager t=2024-03-26T03:50:16.970836971Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T03:50:17.399 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:17 smithi006 bash[72485]: Getting image source signatures 2024-03-26T03:50:17.400 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:17 smithi006 bash[72485]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T03:50:17.400 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:17 smithi006 bash[72485]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T03:50:17.400 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:17 smithi006 bash[72485]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T03:50:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:17 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T03:50:17.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:17 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T03:50:17.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:17 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T03:50:17.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:17 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T03:50:17.868 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:17 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T03:50:18.105 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-26T03:50:15.165636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1305,"num_read_kb":1119,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":807,"ondisk_log_size":807,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244992,"kb_used_data":8060,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716224,"statfs":{"total":767960285184,"available":767709413376,"internally_reserved":0,"allocated":8253440,"data_stored":4574792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001600"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369632+0000","last_change":"2024-03-26T03:48:50.035293+0000","last_active":"2024-03-26T03:50:09.369632+0000","last_peered":"2024-03-26T03:50:09.369632+0000","last_clean":"2024-03-26T03:50:09.369632+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:50:09.369632+0000","last_undegraded":"2024-03-26T03:50:09.369632+0000","last_fullsized":"2024-03-26T03:50:09.369632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.235116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369728+0000","last_change":"2024-03-26T03:48:48.025490+0000","last_active":"2024-03-26T03:50:09.369728+0000","last_peered":"2024-03-26T03:50:09.369728+0000","last_clean":"2024-03-26T03:50:09.369728+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:50:09.369728+0000","last_undegraded":"2024-03-26T03:50:09.369728+0000","last_fullsized":"2024-03-26T03:50:09.369728+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730374+0000","last_change":"2024-03-26T03:49:49.005275+0000","last_active":"2024-03-26T03:50:09.730374+0000","last_peered":"2024-03-26T03:50:09.730374+0000","last_clean":"2024-03-26T03:50:09.730374+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:50:09.730374+0000","last_undegraded":"2024-03-26T03:50:09.730374+0000","last_fullsized":"2024-03-26T03:50:09.730374+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:49.005129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:49.005129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.159629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369189+0000","last_change":"2024-03-26T03:48:46.018459+0000","last_active":"2024-03-26T03:50:09.369189+0000","last_peered":"2024-03-26T03:50:09.369189+0000","last_clean":"2024-03-26T03:50:09.369189+0000","last_became_active":"2024-03-26T03:48:46.018079+0000","last_became_peered":"2024-03-26T03:48:46.018079+0000","last_unstale":"2024-03-26T03:50:09.369189+0000","last_undegraded":"2024-03-26T03:50:09.369189+0000","last_fullsized":"2024-03-26T03:50:09.369189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.102479+0000","last_change":"2024-03-26T03:50:13.102521+0000","last_active":"2024-03-26T03:50:13.102479+0000","last_peered":"2024-03-26T03:50:13.102479+0000","last_clean":"2024-03-26T03:50:13.102479+0000","last_became_active":"2024-03-26T03:48:50.262548+0000","last_became_peered":"2024-03-26T03:48:50.262548+0000","last_unstale":"2024-03-26T03:50:13.102479+0000","last_undegraded":"2024-03-26T03:50:13.102479+0000","last_fullsized":"2024-03-26T03:50:13.102479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.102383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.102383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.725379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369189+0000","last_change":"2024-03-26T03:48:48.025511+0000","last_active":"2024-03-26T03:50:09.369189+0000","last_peered":"2024-03-26T03:50:09.369189+0000","last_clean":"2024-03-26T03:50:09.369189+0000","last_became_active":"2024-03-26T03:48:48.024923+0000","last_became_peered":"2024-03-26T03:48:48.024923+0000","last_unstale":"2024-03-26T03:50:09.369189+0000","last_undegraded":"2024-03-26T03:50:09.369189+0000","last_fullsized":"2024-03-26T03:50:09.369189+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.464377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660931+0000","last_change":"2024-03-26T03:48:44.017642+0000","last_active":"2024-03-26T03:50:09.660931+0000","last_peered":"2024-03-26T03:50:09.660931+0000","last_clean":"2024-03-26T03:50:09.660931+0000","last_became_active":"2024-03-26T03:48:44.017341+0000","last_became_peered":"2024-03-26T03:48:44.017341+0000","last_unstale":"2024-03-26T03:50:09.660931+0000","last_undegraded":"2024-03-26T03:50:09.660931+0000","last_fullsized":"2024-03-26T03:50:09.660931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":124,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660597+0000","last_change":"2024-03-26T03:48:46.020802+0000","last_active":"2024-03-26T03:50:09.660597+0000","last_peered":"2024-03-26T03:50:09.660597+0000","last_clean":"2024-03-26T03:50:09.660597+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:50:09.660597+0000","last_undegraded":"2024-03-26T03:50:09.660597+0000","last_fullsized":"2024-03-26T03:50:09.660597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730007+0000","last_change":"2024-03-26T03:48:50.037848+0000","last_active":"2024-03-26T03:50:09.730007+0000","last_peered":"2024-03-26T03:50:09.730007+0000","last_clean":"2024-03-26T03:50:09.730007+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:50:09.730007+0000","last_undegraded":"2024-03-26T03:50:09.730007+0000","last_fullsized":"2024-03-26T03:50:09.730007+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.299751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730999+0000","last_change":"2024-03-26T03:49:59.135484+0000","last_active":"2024-03-26T03:50:09.730999+0000","last_peered":"2024-03-26T03:50:09.730999+0000","last_clean":"2024-03-26T03:50:09.730999+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:50:09.730999+0000","last_undegraded":"2024-03-26T03:50:09.730999+0000","last_fullsized":"2024-03-26T03:50:09.730999+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.135403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.135403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.439280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731113+0000","last_change":"2024-03-26T03:50:09.087479+0000","last_active":"2024-03-26T03:50:09.731113+0000","last_peered":"2024-03-26T03:50:09.731113+0000","last_clean":"2024-03-26T03:50:09.731113+0000","last_became_active":"2024-03-26T03:48:44.015277+0000","last_became_peered":"2024-03-26T03:48:44.015277+0000","last_unstale":"2024-03-26T03:50:09.731113+0000","last_undegraded":"2024-03-26T03:50:09.731113+0000","last_fullsized":"2024-03-26T03:50:09.731113+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:09.087395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.087395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.974902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369765+0000","last_change":"2024-03-26T03:49:54.881008+0000","last_active":"2024-03-26T03:50:09.369765+0000","last_peered":"2024-03-26T03:50:09.369765+0000","last_clean":"2024-03-26T03:50:09.369765+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:50:09.369765+0000","last_undegraded":"2024-03-26T03:50:09.369765+0000","last_fullsized":"2024-03-26T03:50:09.369765+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:49:54.880892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.880892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.130490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730447+0000","last_change":"2024-03-26T03:48:50.039963+0000","last_active":"2024-03-26T03:50:09.730447+0000","last_peered":"2024-03-26T03:50:09.730447+0000","last_clean":"2024-03-26T03:50:09.730447+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:50:09.730447+0000","last_undegraded":"2024-03-26T03:50:09.730447+0000","last_fullsized":"2024-03-26T03:50:09.730447+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.621578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730960+0000","last_change":"2024-03-26T03:48:48.032017+0000","last_active":"2024-03-26T03:50:09.730960+0000","last_peered":"2024-03-26T03:50:09.730960+0000","last_clean":"2024-03-26T03:50:09.730960+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:50:09.730960+0000","last_undegraded":"2024-03-26T03:50:09.730960+0000","last_fullsized":"2024-03-26T03:50:09.730960+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.912589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730498+0000","last_change":"2024-03-26T03:48:44.013770+0000","last_active":"2024-03-26T03:50:09.730498+0000","last_peered":"2024-03-26T03:50:09.730498+0000","last_clean":"2024-03-26T03:50:09.730498+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:50:09.730498+0000","last_undegraded":"2024-03-26T03:50:09.730498+0000","last_fullsized":"2024-03-26T03:50:09.730498+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.115985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'11","reported_seq":73,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369927+0000","last_change":"2024-03-26T03:48:46.023941+0000","last_active":"2024-03-26T03:50:09.369927+0000","last_peered":"2024-03-26T03:50:09.369927+0000","last_clean":"2024-03-26T03:50:09.369927+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:50:09.369927+0000","last_undegraded":"2024-03-26T03:50:09.369927+0000","last_fullsized":"2024-03-26T03:50:09.369927+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370997+0000","last_change":"2024-03-26T03:48:50.261495+0000","last_active":"2024-03-26T03:50:09.370997+0000","last_peered":"2024-03-26T03:50:09.370997+0000","last_clean":"2024-03-26T03:50:09.370997+0000","last_became_active":"2024-03-26T03:48:50.260266+0000","last_became_peered":"2024-03-26T03:48:50.260266+0000","last_unstale":"2024-03-26T03:50:09.370997+0000","last_undegraded":"2024-03-26T03:50:09.370997+0000","last_fullsized":"2024-03-26T03:50:09.370997+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.926198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370360+0000","last_change":"2024-03-26T03:48:48.025562+0000","last_active":"2024-03-26T03:50:09.370360+0000","last_peered":"2024-03-26T03:50:09.370360+0000","last_clean":"2024-03-26T03:50:09.370360+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:50:09.370360+0000","last_undegraded":"2024-03-26T03:50:09.370360+0000","last_fullsized":"2024-03-26T03:50:09.370360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.783166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730250+0000","last_change":"2024-03-26T03:49:58.053896+0000","last_active":"2024-03-26T03:50:09.730250+0000","last_peered":"2024-03-26T03:50:09.730250+0000","last_clean":"2024-03-26T03:50:09.730250+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:50:09.730250+0000","last_undegraded":"2024-03-26T03:50:09.730250+0000","last_fullsized":"2024-03-26T03:50:09.730250+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.053811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.053811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.133041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":84,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.368960+0000","last_change":"2024-03-26T03:49:53.266632+0000","last_active":"2024-03-26T03:50:09.368960+0000","last_peered":"2024-03-26T03:50:09.368960+0000","last_clean":"2024-03-26T03:50:09.368960+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:50:09.368960+0000","last_undegraded":"2024-03-26T03:50:09.368960+0000","last_fullsized":"2024-03-26T03:50:09.368960+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:49:53.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:53.266536+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.703552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661062+0000","last_change":"2024-03-26T03:48:50.261258+0000","last_active":"2024-03-26T03:50:09.661062+0000","last_peered":"2024-03-26T03:50:09.661062+0000","last_clean":"2024-03-26T03:50:09.661062+0000","last_became_active":"2024-03-26T03:48:50.260688+0000","last_became_peered":"2024-03-26T03:48:50.260688+0000","last_unstale":"2024-03-26T03:50:09.661062+0000","last_undegraded":"2024-03-26T03:50:09.661062+0000","last_fullsized":"2024-03-26T03:50:09.661062+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370475+0000","last_change":"2024-03-26T03:48:48.030756+0000","last_active":"2024-03-26T03:50:09.370475+0000","last_peered":"2024-03-26T03:50:09.370475+0000","last_clean":"2024-03-26T03:50:09.370475+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:50:09.370475+0000","last_undegraded":"2024-03-26T03:50:09.370475+0000","last_fullsized":"2024-03-26T03:50:09.370475+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369784+0000","last_change":"2024-03-26T03:49:58.872361+0000","last_active":"2024-03-26T03:50:09.369784+0000","last_peered":"2024-03-26T03:50:09.369784+0000","last_clean":"2024-03-26T03:50:09.369784+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:50:09.369784+0000","last_undegraded":"2024-03-26T03:50:09.369784+0000","last_fullsized":"2024-03-26T03:50:09.369784+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.872258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.872258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.889376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660700+0000","last_change":"2024-03-26T03:50:07.047577+0000","last_active":"2024-03-26T03:50:09.660700+0000","last_peered":"2024-03-26T03:50:09.660700+0000","last_clean":"2024-03-26T03:50:09.660700+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:50:09.660700+0000","last_undegraded":"2024-03-26T03:50:09.660700+0000","last_fullsized":"2024-03-26T03:50:09.660700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:50:07.047453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.047453+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.870720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":53,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731662+0000","last_change":"2024-03-26T03:50:02.740650+0000","last_active":"2024-03-26T03:50:09.731662+0000","last_peered":"2024-03-26T03:50:09.731662+0000","last_clean":"2024-03-26T03:50:09.731662+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:50:09.731662+0000","last_undegraded":"2024-03-26T03:50:09.731662+0000","last_fullsized":"2024-03-26T03:50:09.731662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.740545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.965068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":80,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731921+0000","last_change":"2024-03-26T03:48:48.022793+0000","last_active":"2024-03-26T03:50:09.731921+0000","last_peered":"2024-03-26T03:50:09.731921+0000","last_clean":"2024-03-26T03:50:09.731921+0000","last_became_active":"2024-03-26T03:48:48.022480+0000","last_became_peered":"2024-03-26T03:48:48.022480+0000","last_unstale":"2024-03-26T03:50:09.731921+0000","last_undegraded":"2024-03-26T03:50:09.731921+0000","last_fullsized":"2024-03-26T03:50:09.731921+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:14.690568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369820+0000","last_change":"2024-03-26T03:48:44.018169+0000","last_active":"2024-03-26T03:50:09.369820+0000","last_peered":"2024-03-26T03:50:09.369820+0000","last_clean":"2024-03-26T03:50:09.369820+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:50:09.369820+0000","last_undegraded":"2024-03-26T03:50:09.369820+0000","last_fullsized":"2024-03-26T03:50:09.369820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.658563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":78,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730308+0000","last_change":"2024-03-26T03:48:46.210691+0000","last_active":"2024-03-26T03:50:09.730308+0000","last_peered":"2024-03-26T03:50:09.730308+0000","last_clean":"2024-03-26T03:50:09.730308+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:50:09.730308+0000","last_undegraded":"2024-03-26T03:50:09.730308+0000","last_fullsized":"2024-03-26T03:50:09.730308+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369796+0000","last_change":"2024-03-26T03:48:50.035147+0000","last_active":"2024-03-26T03:50:09.369796+0000","last_peered":"2024-03-26T03:50:09.369796+0000","last_clean":"2024-03-26T03:50:09.369796+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:50:09.369796+0000","last_undegraded":"2024-03-26T03:50:09.369796+0000","last_fullsized":"2024-03-26T03:50:09.369796+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730444+0000","last_change":"2024-03-26T03:50:07.149609+0000","last_active":"2024-03-26T03:50:09.730444+0000","last_peered":"2024-03-26T03:50:09.730444+0000","last_clean":"2024-03-26T03:50:09.730444+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:50:09.730444+0000","last_undegraded":"2024-03-26T03:50:09.730444+0000","last_fullsized":"2024-03-26T03:50:09.730444+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.149533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.149533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.646920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660155+0000","last_change":"2024-03-26T03:48:44.016177+0000","last_active":"2024-03-26T03:50:09.660155+0000","last_peered":"2024-03-26T03:50:09.660155+0000","last_clean":"2024-03-26T03:50:09.660155+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:50:09.660155+0000","last_undegraded":"2024-03-26T03:50:09.660155+0000","last_fullsized":"2024-03-26T03:50:09.660155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.493384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731804+0000","last_change":"2024-03-26T03:49:58.491993+0000","last_active":"2024-03-26T03:50:09.731804+0000","last_peered":"2024-03-26T03:50:09.731804+0000","last_clean":"2024-03-26T03:50:09.731804+0000","last_became_active":"2024-03-26T03:48:46.023417+0000","last_became_peered":"2024-03-26T03:48:46.023417+0000","last_unstale":"2024-03-26T03:50:09.731804+0000","last_undegraded":"2024-03-26T03:50:09.731804+0000","last_fullsized":"2024-03-26T03:50:09.731804+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:49:58.491875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.491875+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.158055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730872+0000","last_change":"2024-03-26T03:48:50.029982+0000","last_active":"2024-03-26T03:50:09.730872+0000","last_peered":"2024-03-26T03:50:09.730872+0000","last_clean":"2024-03-26T03:50:09.730872+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:50:09.730872+0000","last_undegraded":"2024-03-26T03:50:09.730872+0000","last_fullsized":"2024-03-26T03:50:09.730872+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.788600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.565053+0000","last_change":"2024-03-26T03:50:13.565074+0000","last_active":"2024-03-26T03:50:13.565053+0000","last_peered":"2024-03-26T03:50:13.565053+0000","last_clean":"2024-03-26T03:50:13.565053+0000","last_became_active":"2024-03-26T03:48:48.022532+0000","last_became_peered":"2024-03-26T03:48:48.022532+0000","last_unstale":"2024-03-26T03:50:13.565053+0000","last_undegraded":"2024-03-26T03:50:13.565053+0000","last_fullsized":"2024-03-26T03:50:13.565053+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.564988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.564988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.338769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.220574+0000","last_change":"2024-03-26T03:50:13.220621+0000","last_active":"2024-03-26T03:50:13.220574+0000","last_peered":"2024-03-26T03:50:13.220574+0000","last_clean":"2024-03-26T03:50:13.220574+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:50:13.220574+0000","last_undegraded":"2024-03-26T03:50:13.220574+0000","last_fullsized":"2024-03-26T03:50:13.220574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.220474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.220474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.320102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660239+0000","last_change":"2024-03-26T03:48:46.027982+0000","last_active":"2024-03-26T03:50:09.660239+0000","last_peered":"2024-03-26T03:50:09.660239+0000","last_clean":"2024-03-26T03:50:09.660239+0000","last_became_active":"2024-03-26T03:48:46.027671+0000","last_became_peered":"2024-03-26T03:48:46.027671+0000","last_unstale":"2024-03-26T03:50:09.660239+0000","last_undegraded":"2024-03-26T03:50:09.660239+0000","last_fullsized":"2024-03-26T03:50:09.660239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660316+0000","last_change":"2024-03-26T03:48:50.261731+0000","last_active":"2024-03-26T03:50:09.660316+0000","last_peered":"2024-03-26T03:50:09.660316+0000","last_clean":"2024-03-26T03:50:09.660316+0000","last_became_active":"2024-03-26T03:48:50.261396+0000","last_became_peered":"2024-03-26T03:48:50.261396+0000","last_unstale":"2024-03-26T03:50:09.660316+0000","last_undegraded":"2024-03-26T03:50:09.660316+0000","last_fullsized":"2024-03-26T03:50:09.660316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732212+0000","last_change":"2024-03-26T03:48:48.029980+0000","last_active":"2024-03-26T03:50:09.732212+0000","last_peered":"2024-03-26T03:50:09.732212+0000","last_clean":"2024-03-26T03:50:09.732212+0000","last_became_active":"2024-03-26T03:48:48.029501+0000","last_became_peered":"2024-03-26T03:48:48.029501+0000","last_unstale":"2024-03-26T03:50:09.732212+0000","last_undegraded":"2024-03-26T03:50:09.732212+0000","last_fullsized":"2024-03-26T03:50:09.732212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.171082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731610+0000","last_change":"2024-03-26T03:50:08.135619+0000","last_active":"2024-03-26T03:50:09.731610+0000","last_peered":"2024-03-26T03:50:09.731610+0000","last_clean":"2024-03-26T03:50:09.731610+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:50:09.731610+0000","last_undegraded":"2024-03-26T03:50:09.731610+0000","last_fullsized":"2024-03-26T03:50:09.731610+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.135537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.135537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.675656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'17","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660437+0000","last_change":"2024-03-26T03:48:46.209729+0000","last_active":"2024-03-26T03:50:09.660437+0000","last_peered":"2024-03-26T03:50:09.660437+0000","last_clean":"2024-03-26T03:50:09.660437+0000","last_became_active":"2024-03-26T03:48:46.209381+0000","last_became_peered":"2024-03-26T03:48:46.209381+0000","last_unstale":"2024-03-26T03:50:09.660437+0000","last_undegraded":"2024-03-26T03:50:09.660437+0000","last_fullsized":"2024-03-26T03:50:09.660437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.729713+0000","last_change":"2024-03-26T03:48:50.037564+0000","last_active":"2024-03-26T03:50:09.729713+0000","last_peered":"2024-03-26T03:50:09.729713+0000","last_clean":"2024-03-26T03:50:09.729713+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:50:09.729713+0000","last_undegraded":"2024-03-26T03:50:09.729713+0000","last_fullsized":"2024-03-26T03:50:09.729713+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660573+0000","last_change":"2024-03-26T03:49:56.980611+0000","last_active":"2024-03-26T03:50:09.660573+0000","last_peered":"2024-03-26T03:50:09.660573+0000","last_clean":"2024-03-26T03:50:09.660573+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:50:09.660573+0000","last_undegraded":"2024-03-26T03:50:09.660573+0000","last_fullsized":"2024-03-26T03:50:09.660573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.980524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.980524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.615228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732302+0000","last_change":"2024-03-26T03:49:50.773333+0000","last_active":"2024-03-26T03:50:09.732302+0000","last_peered":"2024-03-26T03:50:09.732302+0000","last_clean":"2024-03-26T03:50:09.732302+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:50:09.732302+0000","last_undegraded":"2024-03-26T03:50:09.732302+0000","last_fullsized":"2024-03-26T03:50:09.732302+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:50.773202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:50.773202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.850607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370008+0000","last_change":"2024-03-26T03:49:51.011344+0000","last_active":"2024-03-26T03:50:09.370008+0000","last_peered":"2024-03-26T03:50:09.370008+0000","last_clean":"2024-03-26T03:50:09.370008+0000","last_became_active":"2024-03-26T03:48:46.017045+0000","last_became_peered":"2024-03-26T03:48:46.017045+0000","last_unstale":"2024-03-26T03:50:09.370008+0000","last_undegraded":"2024-03-26T03:50:09.370008+0000","last_fullsized":"2024-03-26T03:50:09.370008+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:49:51.011226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.011226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:04.692152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660216+0000","last_change":"2024-03-26T03:50:02.986818+0000","last_active":"2024-03-26T03:50:09.660216+0000","last_peered":"2024-03-26T03:50:09.660216+0000","last_clean":"2024-03-26T03:50:09.660216+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:50:09.660216+0000","last_undegraded":"2024-03-26T03:50:09.660216+0000","last_fullsized":"2024-03-26T03:50:09.660216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.986736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.986736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.565698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369901+0000","last_change":"2024-03-26T03:48:48.030582+0000","last_active":"2024-03-26T03:50:09.369901+0000","last_peered":"2024-03-26T03:50:09.369901+0000","last_clean":"2024-03-26T03:50:09.369901+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:50:09.369901+0000","last_undegraded":"2024-03-26T03:50:09.369901+0000","last_fullsized":"2024-03-26T03:50:09.369901+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370658+0000","last_change":"2024-03-26T03:49:56.925827+0000","last_active":"2024-03-26T03:50:09.370658+0000","last_peered":"2024-03-26T03:50:09.370658+0000","last_clean":"2024-03-26T03:50:09.370658+0000","last_became_active":"2024-03-26T03:48:44.013906+0000","last_became_peered":"2024-03-26T03:48:44.013906+0000","last_unstale":"2024-03-26T03:50:09.370658+0000","last_undegraded":"2024-03-26T03:50:09.370658+0000","last_fullsized":"2024-03-26T03:50:09.370658+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.925725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.925725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":70,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370125+0000","last_change":"2024-03-26T03:48:46.019374+0000","last_active":"2024-03-26T03:50:09.370125+0000","last_peered":"2024-03-26T03:50:09.370125+0000","last_clean":"2024-03-26T03:50:09.370125+0000","last_became_active":"2024-03-26T03:48:46.019043+0000","last_became_peered":"2024-03-26T03:48:46.019043+0000","last_unstale":"2024-03-26T03:50:09.370125+0000","last_undegraded":"2024-03-26T03:50:09.370125+0000","last_fullsized":"2024-03-26T03:50:09.370125+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.864202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730142+0000","last_change":"2024-03-26T03:48:50.046189+0000","last_active":"2024-03-26T03:50:09.730142+0000","last_peered":"2024-03-26T03:50:09.730142+0000","last_clean":"2024-03-26T03:50:09.730142+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:50:09.730142+0000","last_undegraded":"2024-03-26T03:50:09.730142+0000","last_fullsized":"2024-03-26T03:50:09.730142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.027196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731173+0000","last_change":"2024-03-26T03:49:55.131855+0000","last_active":"2024-03-26T03:50:09.731173+0000","last_peered":"2024-03-26T03:50:09.731173+0000","last_clean":"2024-03-26T03:50:09.731173+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:50:09.731173+0000","last_undegraded":"2024-03-26T03:50:09.731173+0000","last_fullsized":"2024-03-26T03:50:09.731173+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:55.131726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:55.131726+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.670201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731224+0000","last_change":"2024-03-26T03:49:48.735316+0000","last_active":"2024-03-26T03:50:09.731224+0000","last_peered":"2024-03-26T03:50:09.731224+0000","last_clean":"2024-03-26T03:50:09.731224+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:50:09.731224+0000","last_undegraded":"2024-03-26T03:50:09.731224+0000","last_fullsized":"2024-03-26T03:50:09.731224+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.735213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.735213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:52.231017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660648+0000","last_change":"2024-03-26T03:48:46.209727+0000","last_active":"2024-03-26T03:50:09.660648+0000","last_peered":"2024-03-26T03:50:09.660648+0000","last_clean":"2024-03-26T03:50:09.660648+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:50:09.660648+0000","last_undegraded":"2024-03-26T03:50:09.660648+0000","last_fullsized":"2024-03-26T03:50:09.660648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.096233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:12.892310+0000","last_change":"2024-03-26T03:50:12.892351+0000","last_active":"2024-03-26T03:50:12.892310+0000","last_peered":"2024-03-26T03:50:12.892310+0000","last_clean":"2024-03-26T03:50:12.892310+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:50:12.892310+0000","last_undegraded":"2024-03-26T03:50:12.892310+0000","last_fullsized":"2024-03-26T03:50:12.892310+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.892200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.892200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.447366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660989+0000","last_change":"2024-03-26T03:48:48.029400+0000","last_active":"2024-03-26T03:50:09.660989+0000","last_peered":"2024-03-26T03:50:09.660989+0000","last_clean":"2024-03-26T03:50:09.660989+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:50:09.660989+0000","last_undegraded":"2024-03-26T03:50:09.660989+0000","last_fullsized":"2024-03-26T03:50:09.660989+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"66'2","reported_seq":67,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.729894+0000","last_change":"2024-03-26T03:48:44.015539+0000","last_active":"2024-03-26T03:50:09.729894+0000","last_peered":"2024-03-26T03:50:09.729894+0000","last_clean":"2024-03-26T03:50:09.729894+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:50:09.729894+0000","last_undegraded":"2024-03-26T03:50:09.729894+0000","last_fullsized":"2024-03-26T03:50:09.729894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'15","reported_seq":79,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731292+0000","last_change":"2024-03-26T03:48:46.210667+0000","last_active":"2024-03-26T03:50:09.731292+0000","last_peered":"2024-03-26T03:50:09.731292+0000","last_clean":"2024-03-26T03:50:09.731292+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:50:09.731292+0000","last_undegraded":"2024-03-26T03:50:09.731292+0000","last_fullsized":"2024-03-26T03:50:09.731292+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"73'1","reported_seq":44,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369107+0000","last_change":"2024-03-26T03:48:50.033575+0000","last_active":"2024-03-26T03:50:09.369107+0000","last_peered":"2024-03-26T03:50:09.369107+0000","last_clean":"2024-03-26T03:50:09.369107+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:50:09.369107+0000","last_undegraded":"2024-03-26T03:50:09.369107+0000","last_fullsized":"2024-03-26T03:50:09.369107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.438017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661037+0000","last_change":"2024-03-26T03:48:48.032124+0000","last_active":"2024-03-26T03:50:09.661037+0000","last_peered":"2024-03-26T03:50:09.661037+0000","last_clean":"2024-03-26T03:50:09.661037+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:50:09.661037+0000","last_undegraded":"2024-03-26T03:50:09.661037+0000","last_fullsized":"2024-03-26T03:50:09.661037+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730552+0000","last_change":"2024-03-26T03:49:56.749968+0000","last_active":"2024-03-26T03:50:09.730552+0000","last_peered":"2024-03-26T03:50:09.730552+0000","last_clean":"2024-03-26T03:50:09.730552+0000","last_became_active":"2024-03-26T03:48:44.018391+0000","last_became_peered":"2024-03-26T03:48:44.018391+0000","last_unstale":"2024-03-26T03:50:09.730552+0000","last_undegraded":"2024-03-26T03:50:09.730552+0000","last_fullsized":"2024-03-26T03:50:09.730552+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.749811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.749811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.793876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730386+0000","last_change":"2024-03-26T03:48:46.209877+0000","last_active":"2024-03-26T03:50:09.730386+0000","last_peered":"2024-03-26T03:50:09.730386+0000","last_clean":"2024-03-26T03:50:09.730386+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:50:09.730386+0000","last_undegraded":"2024-03-26T03:50:09.730386+0000","last_fullsized":"2024-03-26T03:50:09.730386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732302+0000","last_change":"2024-03-26T03:48:50.263320+0000","last_active":"2024-03-26T03:50:09.732302+0000","last_peered":"2024-03-26T03:50:09.732302+0000","last_clean":"2024-03-26T03:50:09.732302+0000","last_became_active":"2024-03-26T03:48:50.262776+0000","last_became_peered":"2024-03-26T03:48:50.262776+0000","last_unstale":"2024-03-26T03:50:09.732302+0000","last_undegraded":"2024-03-26T03:50:09.732302+0000","last_fullsized":"2024-03-26T03:50:09.732302+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.455736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.077469+0000","last_change":"2024-03-26T03:50:14.077503+0000","last_active":"2024-03-26T03:50:14.077469+0000","last_peered":"2024-03-26T03:50:14.077469+0000","last_clean":"2024-03-26T03:50:14.077469+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:50:14.077469+0000","last_undegraded":"2024-03-26T03:50:14.077469+0000","last_fullsized":"2024-03-26T03:50:14.077469+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:14.077398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:14.077398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.279309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730344+0000","last_change":"2024-03-26T03:49:52.565054+0000","last_active":"2024-03-26T03:50:09.730344+0000","last_peered":"2024-03-26T03:50:09.730344+0000","last_clean":"2024-03-26T03:50:09.730344+0000","last_became_active":"2024-03-26T03:48:44.014783+0000","last_became_peered":"2024-03-26T03:48:44.014783+0000","last_unstale":"2024-03-26T03:50:09.730344+0000","last_undegraded":"2024-03-26T03:50:09.730344+0000","last_fullsized":"2024-03-26T03:50:09.730344+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.564972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.484718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661119+0000","last_change":"2024-03-26T03:50:02.026119+0000","last_active":"2024-03-26T03:50:09.661119+0000","last_peered":"2024-03-26T03:50:09.661119+0000","last_clean":"2024-03-26T03:50:09.661119+0000","last_became_active":"2024-03-26T03:48:46.210541+0000","last_became_peered":"2024-03-26T03:48:46.210541+0000","last_unstale":"2024-03-26T03:50:09.661119+0000","last_undegraded":"2024-03-26T03:50:09.661119+0000","last_fullsized":"2024-03-26T03:50:09.661119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:50:02.025987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.025987+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.118172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730995+0000","last_change":"2024-03-26T03:49:52.162584+0000","last_active":"2024-03-26T03:50:09.730995+0000","last_peered":"2024-03-26T03:50:09.730995+0000","last_clean":"2024-03-26T03:50:09.730995+0000","last_became_active":"2024-03-26T03:48:50.260705+0000","last_became_peered":"2024-03-26T03:48:50.260705+0000","last_unstale":"2024-03-26T03:50:09.730995+0000","last_undegraded":"2024-03-26T03:50:09.730995+0000","last_fullsized":"2024-03-26T03:50:09.730995+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.162478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.162478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:09.131410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":75,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732442+0000","last_change":"2024-03-26T03:50:04.799748+0000","last_active":"2024-03-26T03:50:09.732442+0000","last_peered":"2024-03-26T03:50:09.732442+0000","last_clean":"2024-03-26T03:50:09.732442+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:50:09.732442+0000","last_undegraded":"2024-03-26T03:50:09.732442+0000","last_fullsized":"2024-03-26T03:50:09.732442+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:50:04.799655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:04.799655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.356778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730266+0000","last_change":"2024-03-26T03:48:46.210551+0000","last_active":"2024-03-26T03:50:09.730266+0000","last_peered":"2024-03-26T03:50:09.730266+0000","last_clean":"2024-03-26T03:50:09.730266+0000","last_became_active":"2024-03-26T03:48:46.210207+0000","last_became_peered":"2024-03-26T03:48:46.210207+0000","last_unstale":"2024-03-26T03:50:09.730266+0000","last_undegraded":"2024-03-26T03:50:09.730266+0000","last_fullsized":"2024-03-26T03:50:09.730266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.203264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730809+0000","last_change":"2024-03-26T03:49:54.753786+0000","last_active":"2024-03-26T03:50:09.730809+0000","last_peered":"2024-03-26T03:50:09.730809+0000","last_clean":"2024-03-26T03:50:09.730809+0000","last_became_active":"2024-03-26T03:48:48.027440+0000","last_became_peered":"2024-03-26T03:48:48.027440+0000","last_unstale":"2024-03-26T03:50:09.730809+0000","last_undegraded":"2024-03-26T03:50:09.730809+0000","last_fullsized":"2024-03-26T03:50:09.730809+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:54.753680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.753680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.334605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660362+0000","last_change":"2024-03-26T03:48:50.043541+0000","last_active":"2024-03-26T03:50:09.660362+0000","last_peered":"2024-03-26T03:50:09.660362+0000","last_clean":"2024-03-26T03:50:09.660362+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:50:09.660362+0000","last_undegraded":"2024-03-26T03:50:09.660362+0000","last_fullsized":"2024-03-26T03:50:09.660362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732207+0000","last_change":"2024-03-26T03:48:44.017669+0000","last_active":"2024-03-26T03:50:09.732207+0000","last_peered":"2024-03-26T03:50:09.732207+0000","last_clean":"2024-03-26T03:50:09.732207+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:50:09.732207+0000","last_undegraded":"2024-03-26T03:50:09.732207+0000","last_fullsized":"2024-03-26T03:50:09.732207+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":106,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.913412+0000","last_change":"2024-03-26T03:50:13.913444+0000","last_active":"2024-03-26T03:50:13.913412+0000","last_peered":"2024-03-26T03:50:13.913412+0000","last_clean":"2024-03-26T03:50:13.913412+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:50:13.913412+0000","last_undegraded":"2024-03-26T03:50:13.913412+0000","last_fullsized":"2024-03-26T03:50:13.913412+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:50:13.913348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.913348+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:33.729626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.104293+0000","last_change":"2024-03-26T03:50:13.104332+0000","last_active":"2024-03-26T03:50:13.104293+0000","last_peered":"2024-03-26T03:50:13.104293+0000","last_clean":"2024-03-26T03:50:13.104293+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:50:13.104293+0000","last_undegraded":"2024-03-26T03:50:13.104293+0000","last_fullsized":"2024-03-26T03:50:13.104293+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.104219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.104219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.804503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731992+0000","last_change":"2024-03-26T03:49:58.807686+0000","last_active":"2024-03-26T03:50:09.731992+0000","last_peered":"2024-03-26T03:50:09.731992+0000","last_clean":"2024-03-26T03:50:09.731992+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:50:09.731992+0000","last_undegraded":"2024-03-26T03:50:09.731992+0000","last_fullsized":"2024-03-26T03:50:09.731992+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.807606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.807606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.096069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.368876+0000","last_change":"2024-03-26T03:48:48.018417+0000","last_active":"2024-03-26T03:50:09.368876+0000","last_peered":"2024-03-26T03:50:09.368876+0000","last_clean":"2024-03-26T03:50:09.368876+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:50:09.368876+0000","last_undegraded":"2024-03-26T03:50:09.368876+0000","last_fullsized":"2024-03-26T03:50:09.368876+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.957546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732473+0000","last_change":"2024-03-26T03:50:08.854957+0000","last_active":"2024-03-26T03:50:09.732473+0000","last_peered":"2024-03-26T03:50:09.732473+0000","last_clean":"2024-03-26T03:50:09.732473+0000","last_became_active":"2024-03-26T03:48:44.013295+0000","last_became_peered":"2024-03-26T03:48:44.013295+0000","last_unstale":"2024-03-26T03:50:09.732473+0000","last_undegraded":"2024-03-26T03:50:09.732473+0000","last_fullsized":"2024-03-26T03:50:09.732473+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.854874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.854874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:37.587577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730868+0000","last_change":"2024-03-26T03:48:46.209925+0000","last_active":"2024-03-26T03:50:09.730868+0000","last_peered":"2024-03-26T03:50:09.730868+0000","last_clean":"2024-03-26T03:50:09.730868+0000","last_became_active":"2024-03-26T03:48:46.209516+0000","last_became_peered":"2024-03-26T03:48:46.209516+0000","last_unstale":"2024-03-26T03:50:09.730868+0000","last_undegraded":"2024-03-26T03:50:09.730868+0000","last_fullsized":"2024-03-26T03:50:09.730868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.729772+0000","last_change":"2024-03-26T03:48:50.040712+0000","last_active":"2024-03-26T03:50:09.729772+0000","last_peered":"2024-03-26T03:50:09.729772+0000","last_clean":"2024-03-26T03:50:09.729772+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:50:09.729772+0000","last_undegraded":"2024-03-26T03:50:09.729772+0000","last_fullsized":"2024-03-26T03:50:09.729772+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.368588+0000","last_change":"2024-03-26T03:49:59.255850+0000","last_active":"2024-03-26T03:50:09.368588+0000","last_peered":"2024-03-26T03:50:09.368588+0000","last_clean":"2024-03-26T03:50:09.368588+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:50:09.368588+0000","last_undegraded":"2024-03-26T03:50:09.368588+0000","last_fullsized":"2024-03-26T03:50:09.368588+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.255770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:16.644194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730582+0000","last_change":"2024-03-26T03:49:57.098936+0000","last_active":"2024-03-26T03:50:09.730582+0000","last_peered":"2024-03-26T03:50:09.730582+0000","last_clean":"2024-03-26T03:50:09.730582+0000","last_became_active":"2024-03-26T03:48:44.016241+0000","last_became_peered":"2024-03-26T03:48:44.016241+0000","last_unstale":"2024-03-26T03:50:09.730582+0000","last_undegraded":"2024-03-26T03:50:09.730582+0000","last_fullsized":"2024-03-26T03:50:09.730582+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.098828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.273950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:11.191288+0000","last_change":"2024-03-26T03:50:11.191322+0000","last_active":"2024-03-26T03:50:11.191288+0000","last_peered":"2024-03-26T03:50:11.191288+0000","last_clean":"2024-03-26T03:50:11.191288+0000","last_became_active":"2024-03-26T03:48:46.023348+0000","last_became_peered":"2024-03-26T03:48:46.023348+0000","last_unstale":"2024-03-26T03:50:11.191288+0000","last_undegraded":"2024-03-26T03:50:11.191288+0000","last_fullsized":"2024-03-26T03:50:11.191288+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:11.191227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.191227+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.173879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660784+0000","last_change":"2024-03-26T03:48:50.044572+0000","last_active":"2024-03-26T03:50:09.660784+0000","last_peered":"2024-03-26T03:50:09.660784+0000","last_clean":"2024-03-26T03:50:09.660784+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:50:09.660784+0000","last_undegraded":"2024-03-26T03:50:09.660784+0000","last_fullsized":"2024-03-26T03:50:09.660784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730748+0000","last_change":"2024-03-26T03:48:47.012857+0000","last_active":"2024-03-26T03:50:09.730748+0000","last_peered":"2024-03-26T03:50:09.730748+0000","last_clean":"2024-03-26T03:50:09.730748+0000","last_became_active":"2024-03-26T03:48:43.008452+0000","last_became_peered":"2024-03-26T03:48:43.008452+0000","last_unstale":"2024-03-26T03:50:09.730748+0000","last_undegraded":"2024-03-26T03:50:09.730748+0000","last_fullsized":"2024-03-26T03:50:09.730748+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:48:41.993340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369768+0000","last_change":"2024-03-26T03:48:44.012922+0000","last_active":"2024-03-26T03:50:09.369768+0000","last_peered":"2024-03-26T03:50:09.369768+0000","last_clean":"2024-03-26T03:50:09.369768+0000","last_became_active":"2024-03-26T03:48:44.012147+0000","last_became_peered":"2024-03-26T03:48:44.012147+0000","last_unstale":"2024-03-26T03:50:09.369768+0000","last_undegraded":"2024-03-26T03:50:09.369768+0000","last_fullsized":"2024-03-26T03:50:09.369768+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":137,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.007007+0000","last_change":"2024-03-26T03:50:13.007007+0000","last_active":"2024-03-26T03:50:13.007007+0000","last_peered":"2024-03-26T03:50:13.007007+0000","last_clean":"2024-03-26T03:50:13.007007+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:50:13.007007+0000","last_undegraded":"2024-03-26T03:50:13.007007+0000","last_fullsized":"2024-03-26T03:50:13.007007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:50:13.006886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.006886+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.897597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730148+0000","last_change":"2024-03-26T03:48:48.024108+0000","last_active":"2024-03-26T03:50:09.730148+0000","last_peered":"2024-03-26T03:50:09.730148+0000","last_clean":"2024-03-26T03:50:09.730148+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:50:09.730148+0000","last_undegraded":"2024-03-26T03:50:09.730148+0000","last_fullsized":"2024-03-26T03:50:09.730148+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:11.829189+0000","last_change":"2024-03-26T03:50:11.829217+0000","last_active":"2024-03-26T03:50:11.829189+0000","last_peered":"2024-03-26T03:50:11.829189+0000","last_clean":"2024-03-26T03:50:11.829189+0000","last_became_active":"2024-03-26T03:48:50.262294+0000","last_became_peered":"2024-03-26T03:48:50.262294+0000","last_unstale":"2024-03-26T03:50:11.829189+0000","last_undegraded":"2024-03-26T03:50:11.829189+0000","last_fullsized":"2024-03-26T03:50:11.829189+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:11.829105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.829105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.079422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369310+0000","last_change":"2024-03-26T03:50:05.944125+0000","last_active":"2024-03-26T03:50:09.369310+0000","last_peered":"2024-03-26T03:50:09.369310+0000","last_clean":"2024-03-26T03:50:09.369310+0000","last_became_active":"2024-03-26T03:48:44.014239+0000","last_became_peered":"2024-03-26T03:48:44.014239+0000","last_unstale":"2024-03-26T03:50:09.369310+0000","last_undegraded":"2024-03-26T03:50:09.369310+0000","last_fullsized":"2024-03-26T03:50:09.369310+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:05.944040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.944040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.216854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.368308+0000","last_change":"2024-03-26T03:49:48.295417+0000","last_active":"2024-03-26T03:50:09.368308+0000","last_peered":"2024-03-26T03:50:09.368308+0000","last_clean":"2024-03-26T03:50:09.368308+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:50:09.368308+0000","last_undegraded":"2024-03-26T03:50:09.368308+0000","last_fullsized":"2024-03-26T03:50:09.368308+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.295312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.295312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.045160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":97,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.971861+0000","last_change":"2024-03-26T03:50:13.971916+0000","last_active":"2024-03-26T03:50:13.971861+0000","last_peered":"2024-03-26T03:50:13.971861+0000","last_clean":"2024-03-26T03:50:13.971861+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:50:13.971861+0000","last_undegraded":"2024-03-26T03:50:13.971861+0000","last_fullsized":"2024-03-26T03:50:13.971861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:50:13.971792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.971792+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:34.626065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369486+0000","last_change":"2024-03-26T03:48:48.020655+0000","last_active":"2024-03-26T03:50:09.369486+0000","last_peered":"2024-03-26T03:50:09.369486+0000","last_clean":"2024-03-26T03:50:09.369486+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:50:09.369486+0000","last_undegraded":"2024-03-26T03:50:09.369486+0000","last_fullsized":"2024-03-26T03:50:09.369486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:15.127534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731511+0000","last_change":"2024-03-26T03:48:50.039906+0000","last_active":"2024-03-26T03:50:09.731511+0000","last_peered":"2024-03-26T03:50:09.731511+0000","last_clean":"2024-03-26T03:50:09.731511+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:50:09.731511+0000","last_undegraded":"2024-03-26T03:50:09.731511+0000","last_fullsized":"2024-03-26T03:50:09.731511+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":303,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.732097+0000","last_change":"2024-03-26T03:48:51.651891+0000","last_active":"2024-03-26T03:50:09.732097+0000","last_peered":"2024-03-26T03:50:09.732097+0000","last_clean":"2024-03-26T03:50:09.732097+0000","last_became_active":"2024-03-26T03:48:34.401623+0000","last_became_peered":"2024-03-26T03:48:34.401623+0000","last_unstale":"2024-03-26T03:50:09.732097+0000","last_undegraded":"2024-03-26T03:50:09.732097+0000","last_fullsized":"2024-03-26T03:50:09.732097+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T03:48:51.651774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:48:51.651774+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:20.226983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660501+0000","last_change":"2024-03-26T03:48:44.017929+0000","last_active":"2024-03-26T03:50:09.660501+0000","last_peered":"2024-03-26T03:50:09.660501+0000","last_clean":"2024-03-26T03:50:09.660501+0000","last_became_active":"2024-03-26T03:48:44.017637+0000","last_became_peered":"2024-03-26T03:48:44.017637+0000","last_unstale":"2024-03-26T03:50:09.660501+0000","last_undegraded":"2024-03-26T03:50:09.660501+0000","last_fullsized":"2024-03-26T03:50:09.660501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.329800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731592+0000","last_change":"2024-03-26T03:50:06.585221+0000","last_active":"2024-03-26T03:50:09.731592+0000","last_peered":"2024-03-26T03:50:09.731592+0000","last_clean":"2024-03-26T03:50:09.731592+0000","last_became_active":"2024-03-26T03:48:46.210282+0000","last_became_peered":"2024-03-26T03:48:46.210282+0000","last_unstale":"2024-03-26T03:50:09.731592+0000","last_undegraded":"2024-03-26T03:50:09.731592+0000","last_fullsized":"2024-03-26T03:50:09.731592+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:50:06.585054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.585054+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.951132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731814+0000","last_change":"2024-03-26T03:50:00.772766+0000","last_active":"2024-03-26T03:50:09.731814+0000","last_peered":"2024-03-26T03:50:09.731814+0000","last_clean":"2024-03-26T03:50:09.731814+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:50:09.731814+0000","last_undegraded":"2024-03-26T03:50:09.731814+0000","last_fullsized":"2024-03-26T03:50:09.731814+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.772625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.974412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370022+0000","last_change":"2024-03-26T03:48:50.043668+0000","last_active":"2024-03-26T03:50:09.370022+0000","last_peered":"2024-03-26T03:50:09.370022+0000","last_clean":"2024-03-26T03:50:09.370022+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:50:09.370022+0000","last_undegraded":"2024-03-26T03:50:09.370022+0000","last_fullsized":"2024-03-26T03:50:09.370022+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.729969+0000","last_change":"2024-03-26T03:48:44.011514+0000","last_active":"2024-03-26T03:50:09.729969+0000","last_peered":"2024-03-26T03:50:09.729969+0000","last_clean":"2024-03-26T03:50:09.729969+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:50:09.729969+0000","last_undegraded":"2024-03-26T03:50:09.729969+0000","last_fullsized":"2024-03-26T03:50:09.729969+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":158,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:12.930901+0000","last_change":"2024-03-26T03:49:45.618585+0000","last_active":"2024-03-26T03:50:12.930901+0000","last_peered":"2024-03-26T03:50:12.930901+0000","last_clean":"2024-03-26T03:50:12.930901+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:50:12.930901+0000","last_undegraded":"2024-03-26T03:50:12.930901+0000","last_fullsized":"2024-03-26T03:50:12.930901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:49:45.618470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:45.618470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.990274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":69,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730213+0000","last_change":"2024-03-26T03:48:46.026122+0000","last_active":"2024-03-26T03:50:09.730213+0000","last_peered":"2024-03-26T03:50:09.730213+0000","last_clean":"2024-03-26T03:50:09.730213+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:50:09.730213+0000","last_undegraded":"2024-03-26T03:50:09.730213+0000","last_fullsized":"2024-03-26T03:50:09.730213+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731792+0000","last_change":"2024-03-26T03:48:48.022621+0000","last_active":"2024-03-26T03:50:09.731792+0000","last_peered":"2024-03-26T03:50:09.731792+0000","last_clean":"2024-03-26T03:50:09.731792+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:50:09.731792+0000","last_undegraded":"2024-03-26T03:50:09.731792+0000","last_fullsized":"2024-03-26T03:50:09.731792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369619+0000","last_change":"2024-03-26T03:48:50.260878+0000","last_active":"2024-03-26T03:50:09.369619+0000","last_peered":"2024-03-26T03:50:09.369619+0000","last_clean":"2024-03-26T03:50:09.369619+0000","last_became_active":"2024-03-26T03:48:50.260476+0000","last_became_peered":"2024-03-26T03:48:50.260476+0000","last_unstale":"2024-03-26T03:50:09.369619+0000","last_undegraded":"2024-03-26T03:50:09.369619+0000","last_fullsized":"2024-03-26T03:50:09.369619+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370174+0000","last_change":"2024-03-26T03:50:00.032886+0000","last_active":"2024-03-26T03:50:09.370174+0000","last_peered":"2024-03-26T03:50:09.370174+0000","last_clean":"2024-03-26T03:50:09.370174+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:50:09.370174+0000","last_undegraded":"2024-03-26T03:50:09.370174+0000","last_fullsized":"2024-03-26T03:50:09.370174+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.032763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.032763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:06.902858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":107,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730303+0000","last_change":"2024-03-26T03:50:03.113111+0000","last_active":"2024-03-26T03:50:09.730303+0000","last_peered":"2024-03-26T03:50:09.730303+0000","last_clean":"2024-03-26T03:50:09.730303+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:50:09.730303+0000","last_undegraded":"2024-03-26T03:50:09.730303+0000","last_fullsized":"2024-03-26T03:50:09.730303+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:03.112983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.112983+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.126914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730959+0000","last_change":"2024-03-26T03:49:54.591020+0000","last_active":"2024-03-26T03:50:09.730959+0000","last_peered":"2024-03-26T03:50:09.730959+0000","last_clean":"2024-03-26T03:50:09.730959+0000","last_became_active":"2024-03-26T03:48:48.021249+0000","last_became_peered":"2024-03-26T03:48:48.021249+0000","last_unstale":"2024-03-26T03:50:09.730959+0000","last_undegraded":"2024-03-26T03:50:09.730959+0000","last_fullsized":"2024-03-26T03:50:09.730959+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:54.590915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.590915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.886951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.729733+0000","last_change":"2024-03-26T03:48:50.045296+0000","last_active":"2024-03-26T03:50:09.729733+0000","last_peered":"2024-03-26T03:50:09.729733+0000","last_clean":"2024-03-26T03:50:09.729733+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:50:09.729733+0000","last_undegraded":"2024-03-26T03:50:09.729733+0000","last_fullsized":"2024-03-26T03:50:09.729733+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:16.928066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731230+0000","last_change":"2024-03-26T03:48:44.014959+0000","last_active":"2024-03-26T03:50:09.731230+0000","last_peered":"2024-03-26T03:50:09.731230+0000","last_clean":"2024-03-26T03:50:09.731230+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:50:09.731230+0000","last_undegraded":"2024-03-26T03:50:09.731230+0000","last_fullsized":"2024-03-26T03:50:09.731230+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":85,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369932+0000","last_change":"2024-03-26T03:49:46.927333+0000","last_active":"2024-03-26T03:50:09.369932+0000","last_peered":"2024-03-26T03:50:09.369932+0000","last_clean":"2024-03-26T03:50:09.369932+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:50:09.369932+0000","last_undegraded":"2024-03-26T03:50:09.369932+0000","last_fullsized":"2024-03-26T03:50:09.369932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:49:46.927230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:46.927230+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:54.304888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730176+0000","last_change":"2024-03-26T03:48:48.019177+0000","last_active":"2024-03-26T03:50:09.730176+0000","last_peered":"2024-03-26T03:50:09.730176+0000","last_clean":"2024-03-26T03:50:09.730176+0000","last_became_active":"2024-03-26T03:48:48.018852+0000","last_became_peered":"2024-03-26T03:48:48.018852+0000","last_unstale":"2024-03-26T03:50:09.730176+0000","last_undegraded":"2024-03-26T03:50:09.730176+0000","last_fullsized":"2024-03-26T03:50:09.730176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731504+0000","last_change":"2024-03-26T03:49:57.800214+0000","last_active":"2024-03-26T03:50:09.731504+0000","last_peered":"2024-03-26T03:50:09.731504+0000","last_clean":"2024-03-26T03:50:09.731504+0000","last_became_active":"2024-03-26T03:48:50.262788+0000","last_became_peered":"2024-03-26T03:48:50.262788+0000","last_unstale":"2024-03-26T03:50:09.731504+0000","last_undegraded":"2024-03-26T03:50:09.731504+0000","last_fullsized":"2024-03-26T03:50:09.731504+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.800113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.800113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:00.814334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":64,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730310+0000","last_change":"2024-03-26T03:50:02.092039+0000","last_active":"2024-03-26T03:50:09.730310+0000","last_peered":"2024-03-26T03:50:09.730310+0000","last_clean":"2024-03-26T03:50:09.730310+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:50:09.730310+0000","last_undegraded":"2024-03-26T03:50:09.730310+0000","last_fullsized":"2024-03-26T03:50:09.730310+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:02.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.091909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.088925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":92,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369473+0000","last_change":"2024-03-26T03:50:06.823874+0000","last_active":"2024-03-26T03:50:09.369473+0000","last_peered":"2024-03-26T03:50:09.369473+0000","last_clean":"2024-03-26T03:50:09.369473+0000","last_became_active":"2024-03-26T03:48:46.016634+0000","last_became_peered":"2024-03-26T03:48:46.016634+0000","last_unstale":"2024-03-26T03:50:09.369473+0000","last_undegraded":"2024-03-26T03:50:09.369473+0000","last_fullsized":"2024-03-26T03:50:09.369473+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:50:06.823723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.823723+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.856329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:10.088926+0000","last_change":"2024-03-26T03:50:10.088926+0000","last_active":"2024-03-26T03:50:10.088926+0000","last_peered":"2024-03-26T03:50:10.088926+0000","last_clean":"2024-03-26T03:50:10.088926+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:50:10.088926+0000","last_undegraded":"2024-03-26T03:50:10.088926+0000","last_fullsized":"2024-03-26T03:50:10.088926+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:10.088836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:10.088836+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.494186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369648+0000","last_change":"2024-03-26T03:48:50.043708+0000","last_active":"2024-03-26T03:50:09.369648+0000","last_peered":"2024-03-26T03:50:09.369648+0000","last_clean":"2024-03-26T03:50:09.369648+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:50:09.369648+0000","last_undegraded":"2024-03-26T03:50:09.369648+0000","last_fullsized":"2024-03-26T03:50:09.369648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660911+0000","last_change":"2024-03-26T03:48:44.013652+0000","last_active":"2024-03-26T03:50:09.660911+0000","last_peered":"2024-03-26T03:50:09.660911+0000","last_clean":"2024-03-26T03:50:09.660911+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:50:09.660911+0000","last_undegraded":"2024-03-26T03:50:09.660911+0000","last_fullsized":"2024-03-26T03:50:09.660911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":125,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660355+0000","last_change":"2024-03-26T03:49:51.111438+0000","last_active":"2024-03-26T03:50:09.660355+0000","last_peered":"2024-03-26T03:50:09.660355+0000","last_clean":"2024-03-26T03:50:09.660355+0000","last_became_active":"2024-03-26T03:48:46.209319+0000","last_became_peered":"2024-03-26T03:48:46.209319+0000","last_unstale":"2024-03-26T03:50:09.660355+0000","last_undegraded":"2024-03-26T03:50:09.660355+0000","last_fullsized":"2024-03-26T03:50:09.660355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.111316+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.974577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369235+0000","last_change":"2024-03-26T03:48:48.029521+0000","last_active":"2024-03-26T03:50:09.369235+0000","last_peered":"2024-03-26T03:50:09.369235+0000","last_clean":"2024-03-26T03:50:09.369235+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:50:09.369235+0000","last_undegraded":"2024-03-26T03:50:09.369235+0000","last_fullsized":"2024-03-26T03:50:09.369235+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.730477+0000","last_change":"2024-03-26T03:48:50.034819+0000","last_active":"2024-03-26T03:50:09.730477+0000","last_peered":"2024-03-26T03:50:09.730477+0000","last_clean":"2024-03-26T03:50:09.730477+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:50:09.730477+0000","last_undegraded":"2024-03-26T03:50:09.730477+0000","last_fullsized":"2024-03-26T03:50:09.730477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.371094+0000","last_change":"2024-03-26T03:50:05.869446+0000","last_active":"2024-03-26T03:50:09.371094+0000","last_peered":"2024-03-26T03:50:09.371094+0000","last_clean":"2024-03-26T03:50:09.371094+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:50:09.371094+0000","last_undegraded":"2024-03-26T03:50:09.371094+0000","last_fullsized":"2024-03-26T03:50:09.371094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:05.869359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.869359+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.254418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660271+0000","last_change":"2024-03-26T03:48:44.017015+0000","last_active":"2024-03-26T03:50:09.660271+0000","last_peered":"2024-03-26T03:50:09.660271+0000","last_clean":"2024-03-26T03:50:09.660271+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:50:09.660271+0000","last_undegraded":"2024-03-26T03:50:09.660271+0000","last_fullsized":"2024-03-26T03:50:09.660271+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":83,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.997520+0000","last_change":"2024-03-26T03:50:09.997564+0000","last_active":"2024-03-26T03:50:09.997520+0000","last_peered":"2024-03-26T03:50:09.997520+0000","last_clean":"2024-03-26T03:50:09.997520+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:50:09.997520+0000","last_undegraded":"2024-03-26T03:50:09.997520+0000","last_fullsized":"2024-03-26T03:50:09.997520+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:09.997449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.997449+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.384605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660561+0000","last_change":"2024-03-26T03:48:50.261659+0000","last_active":"2024-03-26T03:50:09.660561+0000","last_peered":"2024-03-26T03:50:09.660561+0000","last_clean":"2024-03-26T03:50:09.660561+0000","last_became_active":"2024-03-26T03:48:50.260697+0000","last_became_peered":"2024-03-26T03:48:50.260697+0000","last_unstale":"2024-03-26T03:50:09.660561+0000","last_undegraded":"2024-03-26T03:50:09.660561+0000","last_fullsized":"2024-03-26T03:50:09.660561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731788+0000","last_change":"2024-03-26T03:50:02.189077+0000","last_active":"2024-03-26T03:50:09.731788+0000","last_peered":"2024-03-26T03:50:09.731788+0000","last_clean":"2024-03-26T03:50:09.731788+0000","last_became_active":"2024-03-26T03:48:48.019448+0000","last_became_peered":"2024-03-26T03:48:48.019448+0000","last_unstale":"2024-03-26T03:50:09.731788+0000","last_undegraded":"2024-03-26T03:50:09.731788+0000","last_fullsized":"2024-03-26T03:50:09.731788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.188975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.969923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369332+0000","last_change":"2024-03-26T03:49:48.865134+0000","last_active":"2024-03-26T03:50:09.369332+0000","last_peered":"2024-03-26T03:50:09.369332+0000","last_clean":"2024-03-26T03:50:09.369332+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:50:09.369332+0000","last_undegraded":"2024-03-26T03:50:09.369332+0000","last_fullsized":"2024-03-26T03:50:09.369332+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.864997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.477744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:11.925854+0000","last_change":"2024-03-26T03:50:11.925898+0000","last_active":"2024-03-26T03:50:11.925854+0000","last_peered":"2024-03-26T03:50:11.925854+0000","last_clean":"2024-03-26T03:50:11.925854+0000","last_became_active":"2024-03-26T03:48:46.018601+0000","last_became_peered":"2024-03-26T03:48:46.018601+0000","last_unstale":"2024-03-26T03:50:11.925854+0000","last_undegraded":"2024-03-26T03:50:11.925854+0000","last_fullsized":"2024-03-26T03:50:11.925854+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:11.925762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.925762+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.777765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.369396+0000","last_change":"2024-03-26T03:50:07.213828+0000","last_active":"2024-03-26T03:50:09.369396+0000","last_peered":"2024-03-26T03:50:09.369396+0000","last_clean":"2024-03-26T03:50:09.369396+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:50:09.369396+0000","last_undegraded":"2024-03-26T03:50:09.369396+0000","last_fullsized":"2024-03-26T03:50:09.369396+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.213705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.213705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:19.297785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":54,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:12.050833+0000","last_change":"2024-03-26T03:50:12.050853+0000","last_active":"2024-03-26T03:50:12.050833+0000","last_peered":"2024-03-26T03:50:12.050833+0000","last_clean":"2024-03-26T03:50:12.050833+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:50:12.050833+0000","last_undegraded":"2024-03-26T03:50:12.050833+0000","last_fullsized":"2024-03-26T03:50:12.050833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.050772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.050772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.794737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":63,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.731038+0000","last_change":"2024-03-26T03:49:51.516987+0000","last_active":"2024-03-26T03:50:09.731038+0000","last_peered":"2024-03-26T03:50:09.731038+0000","last_clean":"2024-03-26T03:50:09.731038+0000","last_became_active":"2024-03-26T03:48:44.012660+0000","last_became_peered":"2024-03-26T03:48:44.012660+0000","last_unstale":"2024-03-26T03:50:09.731038+0000","last_undegraded":"2024-03-26T03:50:09.731038+0000","last_fullsized":"2024-03-26T03:50:09.731038+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:51.516891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.516891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:57.433207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":87,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.371039+0000","last_change":"2024-03-26T03:50:03.880855+0000","last_active":"2024-03-26T03:50:09.371039+0000","last_peered":"2024-03-26T03:50:09.371039+0000","last_clean":"2024-03-26T03:50:09.371039+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:50:09.371039+0000","last_undegraded":"2024-03-26T03:50:09.371039+0000","last_fullsized":"2024-03-26T03:50:09.371039+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:03.880711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.880711+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.791446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.371098+0000","last_change":"2024-03-26T03:50:02.877487+0000","last_active":"2024-03-26T03:50:09.371098+0000","last_peered":"2024-03-26T03:50:09.371098+0000","last_clean":"2024-03-26T03:50:09.371098+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:50:09.371098+0000","last_undegraded":"2024-03-26T03:50:09.371098+0000","last_fullsized":"2024-03-26T03:50:09.371098+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.877373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.877373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.008619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.368175+0000","last_change":"2024-03-26T03:50:02.300700+0000","last_active":"2024-03-26T03:50:09.368175+0000","last_peered":"2024-03-26T03:50:09.368175+0000","last_clean":"2024-03-26T03:50:09.368175+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:50:09.368175+0000","last_undegraded":"2024-03-26T03:50:09.368175+0000","last_fullsized":"2024-03-26T03:50:09.368175+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:02.300594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.300594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.021817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660628+0000","last_change":"2024-03-26T03:48:44.015913+0000","last_active":"2024-03-26T03:50:09.660628+0000","last_peered":"2024-03-26T03:50:09.660628+0000","last_clean":"2024-03-26T03:50:09.660628+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:50:09.660628+0000","last_undegraded":"2024-03-26T03:50:09.660628+0000","last_fullsized":"2024-03-26T03:50:09.660628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.708831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":66,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.370564+0000","last_change":"2024-03-26T03:48:46.023356+0000","last_active":"2024-03-26T03:50:09.370564+0000","last_peered":"2024-03-26T03:50:09.370564+0000","last_clean":"2024-03-26T03:50:09.370564+0000","last_became_active":"2024-03-26T03:48:46.022937+0000","last_became_peered":"2024-03-26T03:48:46.022937+0000","last_unstale":"2024-03-26T03:50:09.370564+0000","last_undegraded":"2024-03-26T03:50:09.370564+0000","last_fullsized":"2024-03-26T03:50:09.370564+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:10.912372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":599,"ondisk_log_size":599,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":83,"num_read_kb":78,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070486,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28392,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716760,"statfs":{"total":95995035648,"available":95965962240,"internally_reserved":0,"allocated":1409024,"data_stored":942035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364828,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32520,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712632,"statfs":{"total":95995035648,"available":95961735168,"internally_reserved":0,"allocated":1376256,"data_stored":939623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32004,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713148,"statfs":{"total":95995035648,"available":95962263552,"internally_reserved":0,"allocated":847872,"data_stored":351040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1409024,"data_stored":941764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31940,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713212,"statfs":{"total":95995035648,"available":95962329088,"internally_reserved":0,"allocated":782336,"data_stored":349383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":351493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717340,"statfs":{"total":95995035648,"available":95966556160,"internally_reserved":0,"allocated":815104,"data_stored":349873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659167,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27784,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717368,"statfs":{"total":95995035648,"available":95966584832,"internally_reserved":0,"allocated":786432,"data_stored":349581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:50:18.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:50:18.694 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:18.712 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[37501]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:18.712 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[37501]: 5.1 deep-scrub starts 2024-03-26T03:50:18.712 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[37501]: 5.1 deep-scrub ok 2024-03-26T03:50:18.712 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[37501]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:18.712 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:18.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[29446]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:18.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[29446]: 5.1 deep-scrub starts 2024-03-26T03:50:18.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[29446]: 5.1 deep-scrub ok 2024-03-26T03:50:18.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[29446]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:18.713 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:18 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:18 smithi086 ceph-mon[32965]: from='client.24629 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:18 smithi086 ceph-mon[32965]: 5.1 deep-scrub starts 2024-03-26T03:50:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:18 smithi086 ceph-mon[32965]: 5.1 deep-scrub ok 2024-03-26T03:50:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:18 smithi086 ceph-mon[32965]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:18 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:18.981 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:18 smithi006 bash[72485]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T03:50:18.981 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:18 smithi006 bash[72485]: Writing manifest to image destination 2024-03-26T03:50:19.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T03:50:19.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T03:50:19.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 4.18 scrub starts 2024-03-26T03:50:19.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 4.18 scrub ok 2024-03-26T03:50:19.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T03:50:19.639 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:19 smithi006 podman[72485]: 2024-03-26 03:50:19.422431607 +0000 UTC m=+2.688413089 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T03:50:19.639 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:19 smithi006 podman[72485]: 2024-03-26 03:50:19.638140535 +0000 UTC m=+2.904122004 container create 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 4.18 scrub starts 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 4.18 scrub ok 2024-03-26T03:50:19.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T03:50:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:19 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 4.18 scrub starts 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 4.18 scrub ok 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T03:50:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:19 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T03:50:20.536 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T03:50:20.536 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T03:50:20.536 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[37501]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:50:20.537 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T03:50:20.537 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T03:50:20.537 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-mon[29446]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:50:20.537 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 podman[72485]: 2024-03-26 03:50:20.352210834 +0000 UTC m=+3.618192306 container init 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 podman[72485]: 2024-03-26 03:50:20.355328042 +0000 UTC m=+3.621309520 container start 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.359Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.359Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.360Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.360Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.360Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.360Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T03:50:20.538 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T03:50:20.539 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T03:50:20.540 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a[72645]: ts=2024-03-26T03:50:20.361Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 bash[72485]: 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 2024-03-26T03:50:20.541 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 03:50:20 smithi006 systemd[1]: Started Ceph node-exporter.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:20 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T03:50:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:20 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T03:50:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:20 smithi086 ceph-mon[32965]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:50:21.137 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:50:21.139 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:50:21.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T03:50:21.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T03:50:21.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.15 deep-scrub starts 2024-03-26T03:50:21.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.15 deep-scrub ok 2024-03-26T03:50:21.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.11 deep-scrub starts 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.11 deep-scrub ok 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: Deploying daemon node-exporter.b on smithi086 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.15 deep-scrub starts 2024-03-26T03:50:21.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.15 deep-scrub ok 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.11 deep-scrub starts 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.11 deep-scrub ok 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: Deploying daemon node-exporter.b on smithi086 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T03:50:21.762 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:21 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T03:50:21.795 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-26T03:50:19.166575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1310,"num_read_kb":1124,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":808,"ondisk_log_size":808,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":245080,"kb_used_data":8144,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749716136,"statfs":{"total":767960285184,"available":767709323264,"internally_reserved":0,"allocated":8339456,"data_stored":4619396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.798109"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163597+0000","last_change":"2024-03-26T03:48:50.035293+0000","last_active":"2024-03-26T03:50:14.163597+0000","last_peered":"2024-03-26T03:50:14.163597+0000","last_clean":"2024-03-26T03:50:14.163597+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:50:14.163597+0000","last_undegraded":"2024-03-26T03:50:14.163597+0000","last_fullsized":"2024-03-26T03:50:14.163597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193527+0000","last_change":"2024-03-26T03:48:48.025490+0000","last_active":"2024-03-26T03:50:14.193527+0000","last_peered":"2024-03-26T03:50:14.193527+0000","last_clean":"2024-03-26T03:50:14.193527+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:50:14.193527+0000","last_undegraded":"2024-03-26T03:50:14.193527+0000","last_fullsized":"2024-03-26T03:50:14.193527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164048+0000","last_change":"2024-03-26T03:49:49.005275+0000","last_active":"2024-03-26T03:50:14.164048+0000","last_peered":"2024-03-26T03:50:14.164048+0000","last_clean":"2024-03-26T03:50:14.164048+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:50:14.164048+0000","last_undegraded":"2024-03-26T03:50:14.164048+0000","last_fullsized":"2024-03-26T03:50:14.164048+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:49.005129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:49.005129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.159629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:18.012807+0000","last_change":"2024-03-26T03:50:18.012850+0000","last_active":"2024-03-26T03:50:18.012807+0000","last_peered":"2024-03-26T03:50:18.012807+0000","last_clean":"2024-03-26T03:50:18.012807+0000","last_became_active":"2024-03-26T03:48:46.018079+0000","last_became_peered":"2024-03-26T03:48:46.018079+0000","last_unstale":"2024-03-26T03:50:18.012807+0000","last_undegraded":"2024-03-26T03:50:18.012807+0000","last_fullsized":"2024-03-26T03:50:18.012807+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:18.012741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.012741+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:40.758695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:13.102479+0000","last_change":"2024-03-26T03:50:13.102521+0000","last_active":"2024-03-26T03:50:13.102479+0000","last_peered":"2024-03-26T03:50:13.102479+0000","last_clean":"2024-03-26T03:50:13.102479+0000","last_became_active":"2024-03-26T03:48:50.262548+0000","last_became_peered":"2024-03-26T03:48:50.262548+0000","last_unstale":"2024-03-26T03:50:13.102479+0000","last_undegraded":"2024-03-26T03:50:13.102479+0000","last_fullsized":"2024-03-26T03:50:13.102479+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.102383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.102383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.725379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163551+0000","last_change":"2024-03-26T03:48:48.025511+0000","last_active":"2024-03-26T03:50:14.163551+0000","last_peered":"2024-03-26T03:50:14.163551+0000","last_clean":"2024-03-26T03:50:14.163551+0000","last_became_active":"2024-03-26T03:48:48.024923+0000","last_became_peered":"2024-03-26T03:48:48.024923+0000","last_unstale":"2024-03-26T03:50:14.163551+0000","last_undegraded":"2024-03-26T03:50:14.163551+0000","last_fullsized":"2024-03-26T03:50:14.163551+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660931+0000","last_change":"2024-03-26T03:48:44.017642+0000","last_active":"2024-03-26T03:50:09.660931+0000","last_peered":"2024-03-26T03:50:09.660931+0000","last_clean":"2024-03-26T03:50:09.660931+0000","last_became_active":"2024-03-26T03:48:44.017341+0000","last_became_peered":"2024-03-26T03:48:44.017341+0000","last_unstale":"2024-03-26T03:50:09.660931+0000","last_undegraded":"2024-03-26T03:50:09.660931+0000","last_fullsized":"2024-03-26T03:50:09.660931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":124,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660597+0000","last_change":"2024-03-26T03:48:46.020802+0000","last_active":"2024-03-26T03:50:09.660597+0000","last_peered":"2024-03-26T03:50:09.660597+0000","last_clean":"2024-03-26T03:50:09.660597+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:50:09.660597+0000","last_undegraded":"2024-03-26T03:50:09.660597+0000","last_fullsized":"2024-03-26T03:50:09.660597+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163294+0000","last_change":"2024-03-26T03:48:50.037848+0000","last_active":"2024-03-26T03:50:14.163294+0000","last_peered":"2024-03-26T03:50:14.163294+0000","last_clean":"2024-03-26T03:50:14.163294+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:50:14.163294+0000","last_undegraded":"2024-03-26T03:50:14.163294+0000","last_fullsized":"2024-03-26T03:50:14.163294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460759+0000","last_change":"2024-03-26T03:49:59.135484+0000","last_active":"2024-03-26T03:50:14.460759+0000","last_peered":"2024-03-26T03:50:14.460759+0000","last_clean":"2024-03-26T03:50:14.460759+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:50:14.460759+0000","last_undegraded":"2024-03-26T03:50:14.460759+0000","last_fullsized":"2024-03-26T03:50:14.460759+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.135403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.135403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.439280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461602+0000","last_change":"2024-03-26T03:50:09.087479+0000","last_active":"2024-03-26T03:50:14.461602+0000","last_peered":"2024-03-26T03:50:14.461602+0000","last_clean":"2024-03-26T03:50:14.461602+0000","last_became_active":"2024-03-26T03:48:44.015277+0000","last_became_peered":"2024-03-26T03:48:44.015277+0000","last_unstale":"2024-03-26T03:50:14.461602+0000","last_undegraded":"2024-03-26T03:50:14.461602+0000","last_fullsized":"2024-03-26T03:50:14.461602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:09.087395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.087395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.974902+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":102,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163790+0000","last_change":"2024-03-26T03:49:54.881008+0000","last_active":"2024-03-26T03:50:14.163790+0000","last_peered":"2024-03-26T03:50:14.163790+0000","last_clean":"2024-03-26T03:50:14.163790+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:50:14.163790+0000","last_undegraded":"2024-03-26T03:50:14.163790+0000","last_fullsized":"2024-03-26T03:50:14.163790+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:49:54.880892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.880892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.130490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460748+0000","last_change":"2024-03-26T03:48:50.039963+0000","last_active":"2024-03-26T03:50:14.460748+0000","last_peered":"2024-03-26T03:50:14.460748+0000","last_clean":"2024-03-26T03:50:14.460748+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:50:14.460748+0000","last_undegraded":"2024-03-26T03:50:14.460748+0000","last_fullsized":"2024-03-26T03:50:14.460748+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462206+0000","last_change":"2024-03-26T03:48:48.032017+0000","last_active":"2024-03-26T03:50:14.462206+0000","last_peered":"2024-03-26T03:50:14.462206+0000","last_clean":"2024-03-26T03:50:14.462206+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:50:14.462206+0000","last_undegraded":"2024-03-26T03:50:14.462206+0000","last_fullsized":"2024-03-26T03:50:14.462206+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461331+0000","last_change":"2024-03-26T03:48:44.013770+0000","last_active":"2024-03-26T03:50:14.461331+0000","last_peered":"2024-03-26T03:50:14.461331+0000","last_clean":"2024-03-26T03:50:14.461331+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:50:14.461331+0000","last_undegraded":"2024-03-26T03:50:14.461331+0000","last_fullsized":"2024-03-26T03:50:14.461331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'11","reported_seq":75,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164008+0000","last_change":"2024-03-26T03:48:46.023941+0000","last_active":"2024-03-26T03:50:14.164008+0000","last_peered":"2024-03-26T03:50:14.164008+0000","last_clean":"2024-03-26T03:50:14.164008+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:50:14.164008+0000","last_undegraded":"2024-03-26T03:50:14.164008+0000","last_fullsized":"2024-03-26T03:50:14.164008+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164381+0000","last_change":"2024-03-26T03:48:50.261495+0000","last_active":"2024-03-26T03:50:14.164381+0000","last_peered":"2024-03-26T03:50:14.164381+0000","last_clean":"2024-03-26T03:50:14.164381+0000","last_became_active":"2024-03-26T03:48:50.260266+0000","last_became_peered":"2024-03-26T03:48:50.260266+0000","last_unstale":"2024-03-26T03:50:14.164381+0000","last_undegraded":"2024-03-26T03:50:14.164381+0000","last_fullsized":"2024-03-26T03:50:14.164381+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":48,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.194064+0000","last_change":"2024-03-26T03:48:48.025562+0000","last_active":"2024-03-26T03:50:14.194064+0000","last_peered":"2024-03-26T03:50:14.194064+0000","last_clean":"2024-03-26T03:50:14.194064+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:50:14.194064+0000","last_undegraded":"2024-03-26T03:50:14.194064+0000","last_fullsized":"2024-03-26T03:50:14.194064+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163714+0000","last_change":"2024-03-26T03:49:58.053896+0000","last_active":"2024-03-26T03:50:14.163714+0000","last_peered":"2024-03-26T03:50:14.163714+0000","last_clean":"2024-03-26T03:50:14.163714+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:50:14.163714+0000","last_undegraded":"2024-03-26T03:50:14.163714+0000","last_fullsized":"2024-03-26T03:50:14.163714+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.053811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.053811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.133041+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":86,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164353+0000","last_change":"2024-03-26T03:49:53.266632+0000","last_active":"2024-03-26T03:50:14.164353+0000","last_peered":"2024-03-26T03:50:14.164353+0000","last_clean":"2024-03-26T03:50:14.164353+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:50:14.164353+0000","last_undegraded":"2024-03-26T03:50:14.164353+0000","last_fullsized":"2024-03-26T03:50:14.164353+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:49:53.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:53.266536+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.703552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661062+0000","last_change":"2024-03-26T03:48:50.261258+0000","last_active":"2024-03-26T03:50:09.661062+0000","last_peered":"2024-03-26T03:50:09.661062+0000","last_clean":"2024-03-26T03:50:09.661062+0000","last_became_active":"2024-03-26T03:48:50.260688+0000","last_became_peered":"2024-03-26T03:48:50.260688+0000","last_unstale":"2024-03-26T03:50:09.661062+0000","last_undegraded":"2024-03-26T03:50:09.661062+0000","last_fullsized":"2024-03-26T03:50:09.661062+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:13.300744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164056+0000","last_change":"2024-03-26T03:48:48.030756+0000","last_active":"2024-03-26T03:50:14.164056+0000","last_peered":"2024-03-26T03:50:14.164056+0000","last_clean":"2024-03-26T03:50:14.164056+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:50:14.164056+0000","last_undegraded":"2024-03-26T03:50:14.164056+0000","last_fullsized":"2024-03-26T03:50:14.164056+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163806+0000","last_change":"2024-03-26T03:49:58.872361+0000","last_active":"2024-03-26T03:50:14.163806+0000","last_peered":"2024-03-26T03:50:14.163806+0000","last_clean":"2024-03-26T03:50:14.163806+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:50:14.163806+0000","last_undegraded":"2024-03-26T03:50:14.163806+0000","last_fullsized":"2024-03-26T03:50:14.163806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.872258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.872258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.889376+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":94,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660700+0000","last_change":"2024-03-26T03:50:07.047577+0000","last_active":"2024-03-26T03:50:09.660700+0000","last_peered":"2024-03-26T03:50:09.660700+0000","last_clean":"2024-03-26T03:50:09.660700+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:50:09.660700+0000","last_undegraded":"2024-03-26T03:50:09.660700+0000","last_fullsized":"2024-03-26T03:50:09.660700+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:50:07.047453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.047453+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.870720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462429+0000","last_change":"2024-03-26T03:50:02.740650+0000","last_active":"2024-03-26T03:50:14.462429+0000","last_peered":"2024-03-26T03:50:14.462429+0000","last_clean":"2024-03-26T03:50:14.462429+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:50:14.462429+0000","last_undegraded":"2024-03-26T03:50:14.462429+0000","last_fullsized":"2024-03-26T03:50:14.462429+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.740545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.965068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461718+0000","last_change":"2024-03-26T03:48:48.022793+0000","last_active":"2024-03-26T03:50:14.461718+0000","last_peered":"2024-03-26T03:50:14.461718+0000","last_clean":"2024-03-26T03:50:14.461718+0000","last_became_active":"2024-03-26T03:48:48.022480+0000","last_became_peered":"2024-03-26T03:48:48.022480+0000","last_unstale":"2024-03-26T03:50:14.461718+0000","last_undegraded":"2024-03-26T03:50:14.461718+0000","last_fullsized":"2024-03-26T03:50:14.461718+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193618+0000","last_change":"2024-03-26T03:48:44.018169+0000","last_active":"2024-03-26T03:50:14.193618+0000","last_peered":"2024-03-26T03:50:14.193618+0000","last_clean":"2024-03-26T03:50:14.193618+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:50:14.193618+0000","last_undegraded":"2024-03-26T03:50:14.193618+0000","last_fullsized":"2024-03-26T03:50:14.193618+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163839+0000","last_change":"2024-03-26T03:48:46.210691+0000","last_active":"2024-03-26T03:50:14.163839+0000","last_peered":"2024-03-26T03:50:14.163839+0000","last_clean":"2024-03-26T03:50:14.163839+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:50:14.163839+0000","last_undegraded":"2024-03-26T03:50:14.163839+0000","last_fullsized":"2024-03-26T03:50:14.163839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193784+0000","last_change":"2024-03-26T03:48:50.035147+0000","last_active":"2024-03-26T03:50:14.193784+0000","last_peered":"2024-03-26T03:50:14.193784+0000","last_clean":"2024-03-26T03:50:14.193784+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:50:14.193784+0000","last_undegraded":"2024-03-26T03:50:14.193784+0000","last_fullsized":"2024-03-26T03:50:14.193784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163963+0000","last_change":"2024-03-26T03:50:07.149609+0000","last_active":"2024-03-26T03:50:14.163963+0000","last_peered":"2024-03-26T03:50:14.163963+0000","last_clean":"2024-03-26T03:50:14.163963+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:50:14.163963+0000","last_undegraded":"2024-03-26T03:50:14.163963+0000","last_fullsized":"2024-03-26T03:50:14.163963+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.149533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.149533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.646920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660155+0000","last_change":"2024-03-26T03:48:44.016177+0000","last_active":"2024-03-26T03:50:09.660155+0000","last_peered":"2024-03-26T03:50:09.660155+0000","last_clean":"2024-03-26T03:50:09.660155+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:50:09.660155+0000","last_undegraded":"2024-03-26T03:50:09.660155+0000","last_fullsized":"2024-03-26T03:50:09.660155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:11.493384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461663+0000","last_change":"2024-03-26T03:49:58.491993+0000","last_active":"2024-03-26T03:50:14.461663+0000","last_peered":"2024-03-26T03:50:14.461663+0000","last_clean":"2024-03-26T03:50:14.461663+0000","last_became_active":"2024-03-26T03:48:46.023417+0000","last_became_peered":"2024-03-26T03:48:46.023417+0000","last_unstale":"2024-03-26T03:50:14.461663+0000","last_undegraded":"2024-03-26T03:50:14.461663+0000","last_fullsized":"2024-03-26T03:50:14.461663+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:49:58.491875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.491875+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.158055+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462103+0000","last_change":"2024-03-26T03:48:50.029982+0000","last_active":"2024-03-26T03:50:14.462103+0000","last_peered":"2024-03-26T03:50:14.462103+0000","last_clean":"2024-03-26T03:50:14.462103+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:50:14.462103+0000","last_undegraded":"2024-03-26T03:50:14.462103+0000","last_fullsized":"2024-03-26T03:50:14.462103+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460891+0000","last_change":"2024-03-26T03:50:13.565074+0000","last_active":"2024-03-26T03:50:14.460891+0000","last_peered":"2024-03-26T03:50:14.460891+0000","last_clean":"2024-03-26T03:50:14.460891+0000","last_became_active":"2024-03-26T03:48:48.022532+0000","last_became_peered":"2024-03-26T03:48:48.022532+0000","last_unstale":"2024-03-26T03:50:14.460891+0000","last_undegraded":"2024-03-26T03:50:14.460891+0000","last_fullsized":"2024-03-26T03:50:14.460891+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.564988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.564988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.338769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164020+0000","last_change":"2024-03-26T03:50:13.220621+0000","last_active":"2024-03-26T03:50:14.164020+0000","last_peered":"2024-03-26T03:50:14.164020+0000","last_clean":"2024-03-26T03:50:14.164020+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:50:14.164020+0000","last_undegraded":"2024-03-26T03:50:14.164020+0000","last_fullsized":"2024-03-26T03:50:14.164020+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.220474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.220474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.320102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":57,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660239+0000","last_change":"2024-03-26T03:48:46.027982+0000","last_active":"2024-03-26T03:50:09.660239+0000","last_peered":"2024-03-26T03:50:09.660239+0000","last_clean":"2024-03-26T03:50:09.660239+0000","last_became_active":"2024-03-26T03:48:46.027671+0000","last_became_peered":"2024-03-26T03:48:46.027671+0000","last_unstale":"2024-03-26T03:50:09.660239+0000","last_undegraded":"2024-03-26T03:50:09.660239+0000","last_fullsized":"2024-03-26T03:50:09.660239+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660316+0000","last_change":"2024-03-26T03:48:50.261731+0000","last_active":"2024-03-26T03:50:09.660316+0000","last_peered":"2024-03-26T03:50:09.660316+0000","last_clean":"2024-03-26T03:50:09.660316+0000","last_became_active":"2024-03-26T03:48:50.261396+0000","last_became_peered":"2024-03-26T03:48:50.261396+0000","last_unstale":"2024-03-26T03:50:09.660316+0000","last_undegraded":"2024-03-26T03:50:09.660316+0000","last_fullsized":"2024-03-26T03:50:09.660316+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462193+0000","last_change":"2024-03-26T03:48:48.029980+0000","last_active":"2024-03-26T03:50:14.462193+0000","last_peered":"2024-03-26T03:50:14.462193+0000","last_clean":"2024-03-26T03:50:14.462193+0000","last_became_active":"2024-03-26T03:48:48.029501+0000","last_became_peered":"2024-03-26T03:48:48.029501+0000","last_unstale":"2024-03-26T03:50:14.462193+0000","last_undegraded":"2024-03-26T03:50:14.462193+0000","last_fullsized":"2024-03-26T03:50:14.462193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461863+0000","last_change":"2024-03-26T03:50:08.135619+0000","last_active":"2024-03-26T03:50:14.461863+0000","last_peered":"2024-03-26T03:50:14.461863+0000","last_clean":"2024-03-26T03:50:14.461863+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:50:14.461863+0000","last_undegraded":"2024-03-26T03:50:14.461863+0000","last_fullsized":"2024-03-26T03:50:14.461863+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.135537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.135537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.675656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"73'17","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660437+0000","last_change":"2024-03-26T03:48:46.209729+0000","last_active":"2024-03-26T03:50:09.660437+0000","last_peered":"2024-03-26T03:50:09.660437+0000","last_clean":"2024-03-26T03:50:09.660437+0000","last_became_active":"2024-03-26T03:48:46.209381+0000","last_became_peered":"2024-03-26T03:48:46.209381+0000","last_unstale":"2024-03-26T03:50:09.660437+0000","last_undegraded":"2024-03-26T03:50:09.660437+0000","last_fullsized":"2024-03-26T03:50:09.660437+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163016+0000","last_change":"2024-03-26T03:48:50.037564+0000","last_active":"2024-03-26T03:50:14.163016+0000","last_peered":"2024-03-26T03:50:14.163016+0000","last_clean":"2024-03-26T03:50:14.163016+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:50:14.163016+0000","last_undegraded":"2024-03-26T03:50:14.163016+0000","last_fullsized":"2024-03-26T03:50:14.163016+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660573+0000","last_change":"2024-03-26T03:49:56.980611+0000","last_active":"2024-03-26T03:50:09.660573+0000","last_peered":"2024-03-26T03:50:09.660573+0000","last_clean":"2024-03-26T03:50:09.660573+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:50:09.660573+0000","last_undegraded":"2024-03-26T03:50:09.660573+0000","last_fullsized":"2024-03-26T03:50:09.660573+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.980524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.980524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.615228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462481+0000","last_change":"2024-03-26T03:49:50.773333+0000","last_active":"2024-03-26T03:50:14.462481+0000","last_peered":"2024-03-26T03:50:14.462481+0000","last_clean":"2024-03-26T03:50:14.462481+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:50:14.462481+0000","last_undegraded":"2024-03-26T03:50:14.462481+0000","last_fullsized":"2024-03-26T03:50:14.462481+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:50.773202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:50.773202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.850607+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":76,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193861+0000","last_change":"2024-03-26T03:49:51.011344+0000","last_active":"2024-03-26T03:50:14.193861+0000","last_peered":"2024-03-26T03:50:14.193861+0000","last_clean":"2024-03-26T03:50:14.193861+0000","last_became_active":"2024-03-26T03:48:46.017045+0000","last_became_peered":"2024-03-26T03:48:46.017045+0000","last_unstale":"2024-03-26T03:50:14.193861+0000","last_undegraded":"2024-03-26T03:50:14.193861+0000","last_fullsized":"2024-03-26T03:50:14.193861+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:49:51.011226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.011226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:04.692152+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":51,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660216+0000","last_change":"2024-03-26T03:50:02.986818+0000","last_active":"2024-03-26T03:50:09.660216+0000","last_peered":"2024-03-26T03:50:09.660216+0000","last_clean":"2024-03-26T03:50:09.660216+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:50:09.660216+0000","last_undegraded":"2024-03-26T03:50:09.660216+0000","last_fullsized":"2024-03-26T03:50:09.660216+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.986736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.986736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.565698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:16.056387+0000","last_change":"2024-03-26T03:50:16.056409+0000","last_active":"2024-03-26T03:50:16.056387+0000","last_peered":"2024-03-26T03:50:16.056387+0000","last_clean":"2024-03-26T03:50:16.056387+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:50:16.056387+0000","last_undegraded":"2024-03-26T03:50:16.056387+0000","last_fullsized":"2024-03-26T03:50:16.056387+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:16.056326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.056326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:42.021645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164214+0000","last_change":"2024-03-26T03:49:56.925827+0000","last_active":"2024-03-26T03:50:14.164214+0000","last_peered":"2024-03-26T03:50:14.164214+0000","last_clean":"2024-03-26T03:50:14.164214+0000","last_became_active":"2024-03-26T03:48:44.013906+0000","last_became_peered":"2024-03-26T03:48:44.013906+0000","last_unstale":"2024-03-26T03:50:14.164214+0000","last_undegraded":"2024-03-26T03:50:14.164214+0000","last_fullsized":"2024-03-26T03:50:14.164214+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.925725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.925725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.026925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":72,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164096+0000","last_change":"2024-03-26T03:48:46.019374+0000","last_active":"2024-03-26T03:50:14.164096+0000","last_peered":"2024-03-26T03:50:14.164096+0000","last_clean":"2024-03-26T03:50:14.164096+0000","last_became_active":"2024-03-26T03:48:46.019043+0000","last_became_peered":"2024-03-26T03:48:46.019043+0000","last_unstale":"2024-03-26T03:50:14.164096+0000","last_undegraded":"2024-03-26T03:50:14.164096+0000","last_fullsized":"2024-03-26T03:50:14.164096+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163649+0000","last_change":"2024-03-26T03:48:50.046189+0000","last_active":"2024-03-26T03:50:14.163649+0000","last_peered":"2024-03-26T03:50:14.163649+0000","last_clean":"2024-03-26T03:50:14.163649+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:50:14.163649+0000","last_undegraded":"2024-03-26T03:50:14.163649+0000","last_fullsized":"2024-03-26T03:50:14.163649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.027196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461735+0000","last_change":"2024-03-26T03:49:55.131855+0000","last_active":"2024-03-26T03:50:14.461735+0000","last_peered":"2024-03-26T03:50:14.461735+0000","last_clean":"2024-03-26T03:50:14.461735+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:50:14.461735+0000","last_undegraded":"2024-03-26T03:50:14.461735+0000","last_fullsized":"2024-03-26T03:50:14.461735+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:55.131726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:55.131726+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.670201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462363+0000","last_change":"2024-03-26T03:49:48.735316+0000","last_active":"2024-03-26T03:50:14.462363+0000","last_peered":"2024-03-26T03:50:14.462363+0000","last_clean":"2024-03-26T03:50:14.462363+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:50:14.462363+0000","last_undegraded":"2024-03-26T03:50:14.462363+0000","last_fullsized":"2024-03-26T03:50:14.462363+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.735213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.735213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:52.231017+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":90,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660648+0000","last_change":"2024-03-26T03:48:46.209727+0000","last_active":"2024-03-26T03:50:09.660648+0000","last_peered":"2024-03-26T03:50:09.660648+0000","last_clean":"2024-03-26T03:50:09.660648+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:50:09.660648+0000","last_undegraded":"2024-03-26T03:50:09.660648+0000","last_fullsized":"2024-03-26T03:50:09.660648+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.096233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164569+0000","last_change":"2024-03-26T03:50:12.892351+0000","last_active":"2024-03-26T03:50:14.164569+0000","last_peered":"2024-03-26T03:50:14.164569+0000","last_clean":"2024-03-26T03:50:14.164569+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:50:14.164569+0000","last_undegraded":"2024-03-26T03:50:14.164569+0000","last_fullsized":"2024-03-26T03:50:14.164569+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.892200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.892200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.447366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":74,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660989+0000","last_change":"2024-03-26T03:48:48.029400+0000","last_active":"2024-03-26T03:50:09.660989+0000","last_peered":"2024-03-26T03:50:09.660989+0000","last_clean":"2024-03-26T03:50:09.660989+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:50:09.660989+0000","last_undegraded":"2024-03-26T03:50:09.660989+0000","last_fullsized":"2024-03-26T03:50:09.660989+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"66'2","reported_seq":69,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163425+0000","last_change":"2024-03-26T03:48:44.015539+0000","last_active":"2024-03-26T03:50:14.163425+0000","last_peered":"2024-03-26T03:50:14.163425+0000","last_clean":"2024-03-26T03:50:14.163425+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:50:14.163425+0000","last_undegraded":"2024-03-26T03:50:14.163425+0000","last_fullsized":"2024-03-26T03:50:14.163425+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"73'15","reported_seq":81,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461969+0000","last_change":"2024-03-26T03:48:46.210667+0000","last_active":"2024-03-26T03:50:14.461969+0000","last_peered":"2024-03-26T03:50:14.461969+0000","last_clean":"2024-03-26T03:50:14.461969+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:50:14.461969+0000","last_undegraded":"2024-03-26T03:50:14.461969+0000","last_fullsized":"2024-03-26T03:50:14.461969+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":55,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:16.038046+0000","last_change":"2024-03-26T03:50:16.038078+0000","last_active":"2024-03-26T03:50:16.038046+0000","last_peered":"2024-03-26T03:50:16.038046+0000","last_clean":"2024-03-26T03:50:16.038046+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:50:16.038046+0000","last_undegraded":"2024-03-26T03:50:16.038046+0000","last_fullsized":"2024-03-26T03:50:16.038046+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.037958+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.602912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661037+0000","last_change":"2024-03-26T03:48:48.032124+0000","last_active":"2024-03-26T03:50:09.661037+0000","last_peered":"2024-03-26T03:50:09.661037+0000","last_clean":"2024-03-26T03:50:09.661037+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:50:09.661037+0000","last_undegraded":"2024-03-26T03:50:09.661037+0000","last_fullsized":"2024-03-26T03:50:09.661037+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462160+0000","last_change":"2024-03-26T03:49:56.749968+0000","last_active":"2024-03-26T03:50:14.462160+0000","last_peered":"2024-03-26T03:50:14.462160+0000","last_clean":"2024-03-26T03:50:14.462160+0000","last_became_active":"2024-03-26T03:48:44.018391+0000","last_became_peered":"2024-03-26T03:48:44.018391+0000","last_unstale":"2024-03-26T03:50:14.462160+0000","last_undegraded":"2024-03-26T03:50:14.462160+0000","last_fullsized":"2024-03-26T03:50:14.462160+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.749811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.749811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.793876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":93,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163862+0000","last_change":"2024-03-26T03:48:46.209877+0000","last_active":"2024-03-26T03:50:14.163862+0000","last_peered":"2024-03-26T03:50:14.163862+0000","last_clean":"2024-03-26T03:50:14.163862+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:50:14.163862+0000","last_undegraded":"2024-03-26T03:50:14.163862+0000","last_fullsized":"2024-03-26T03:50:14.163862+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462372+0000","last_change":"2024-03-26T03:48:50.263320+0000","last_active":"2024-03-26T03:50:14.462372+0000","last_peered":"2024-03-26T03:50:14.462372+0000","last_clean":"2024-03-26T03:50:14.462372+0000","last_became_active":"2024-03-26T03:48:50.262776+0000","last_became_peered":"2024-03-26T03:48:50.262776+0000","last_unstale":"2024-03-26T03:50:14.462372+0000","last_undegraded":"2024-03-26T03:50:14.462372+0000","last_fullsized":"2024-03-26T03:50:14.462372+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461688+0000","last_change":"2024-03-26T03:50:14.077503+0000","last_active":"2024-03-26T03:50:14.461688+0000","last_peered":"2024-03-26T03:50:14.461688+0000","last_clean":"2024-03-26T03:50:14.461688+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:50:14.461688+0000","last_undegraded":"2024-03-26T03:50:14.461688+0000","last_fullsized":"2024-03-26T03:50:14.461688+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:14.077398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:14.077398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.279309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460638+0000","last_change":"2024-03-26T03:49:52.565054+0000","last_active":"2024-03-26T03:50:14.460638+0000","last_peered":"2024-03-26T03:50:14.460638+0000","last_clean":"2024-03-26T03:50:14.460638+0000","last_became_active":"2024-03-26T03:48:44.014783+0000","last_became_peered":"2024-03-26T03:48:44.014783+0000","last_unstale":"2024-03-26T03:50:14.460638+0000","last_undegraded":"2024-03-26T03:50:14.460638+0000","last_fullsized":"2024-03-26T03:50:14.460638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.564972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.484718+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":81,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.661119+0000","last_change":"2024-03-26T03:50:02.026119+0000","last_active":"2024-03-26T03:50:09.661119+0000","last_peered":"2024-03-26T03:50:09.661119+0000","last_clean":"2024-03-26T03:50:09.661119+0000","last_became_active":"2024-03-26T03:48:46.210541+0000","last_became_peered":"2024-03-26T03:48:46.210541+0000","last_unstale":"2024-03-26T03:50:09.661119+0000","last_undegraded":"2024-03-26T03:50:09.661119+0000","last_fullsized":"2024-03-26T03:50:09.661119+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:50:02.025987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.025987+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.118172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461142+0000","last_change":"2024-03-26T03:49:52.162584+0000","last_active":"2024-03-26T03:50:14.461142+0000","last_peered":"2024-03-26T03:50:14.461142+0000","last_clean":"2024-03-26T03:50:14.461142+0000","last_became_active":"2024-03-26T03:48:50.260705+0000","last_became_peered":"2024-03-26T03:48:50.260705+0000","last_unstale":"2024-03-26T03:50:14.461142+0000","last_undegraded":"2024-03-26T03:50:14.461142+0000","last_fullsized":"2024-03-26T03:50:14.461142+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.162478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.162478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:09.131410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462854+0000","last_change":"2024-03-26T03:50:04.799748+0000","last_active":"2024-03-26T03:50:14.462854+0000","last_peered":"2024-03-26T03:50:14.462854+0000","last_clean":"2024-03-26T03:50:14.462854+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:50:14.462854+0000","last_undegraded":"2024-03-26T03:50:14.462854+0000","last_fullsized":"2024-03-26T03:50:14.462854+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:50:04.799655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:04.799655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.356778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":102,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460794+0000","last_change":"2024-03-26T03:48:46.210551+0000","last_active":"2024-03-26T03:50:14.460794+0000","last_peered":"2024-03-26T03:50:14.460794+0000","last_clean":"2024-03-26T03:50:14.460794+0000","last_became_active":"2024-03-26T03:48:46.210207+0000","last_became_peered":"2024-03-26T03:48:46.210207+0000","last_unstale":"2024-03-26T03:50:14.460794+0000","last_undegraded":"2024-03-26T03:50:14.460794+0000","last_fullsized":"2024-03-26T03:50:14.460794+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462490+0000","last_change":"2024-03-26T03:49:54.753786+0000","last_active":"2024-03-26T03:50:14.462490+0000","last_peered":"2024-03-26T03:50:14.462490+0000","last_clean":"2024-03-26T03:50:14.462490+0000","last_became_active":"2024-03-26T03:48:48.027440+0000","last_became_peered":"2024-03-26T03:48:48.027440+0000","last_unstale":"2024-03-26T03:50:14.462490+0000","last_undegraded":"2024-03-26T03:50:14.462490+0000","last_fullsized":"2024-03-26T03:50:14.462490+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:54.753680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.753680+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:12.334605+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660362+0000","last_change":"2024-03-26T03:48:50.043541+0000","last_active":"2024-03-26T03:50:09.660362+0000","last_peered":"2024-03-26T03:50:09.660362+0000","last_clean":"2024-03-26T03:50:09.660362+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:50:09.660362+0000","last_undegraded":"2024-03-26T03:50:09.660362+0000","last_fullsized":"2024-03-26T03:50:09.660362+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462714+0000","last_change":"2024-03-26T03:48:44.017669+0000","last_active":"2024-03-26T03:50:14.462714+0000","last_peered":"2024-03-26T03:50:14.462714+0000","last_clean":"2024-03-26T03:50:14.462714+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:50:14.462714+0000","last_undegraded":"2024-03-26T03:50:14.462714+0000","last_fullsized":"2024-03-26T03:50:14.462714+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":108,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163805+0000","last_change":"2024-03-26T03:50:13.913444+0000","last_active":"2024-03-26T03:50:14.163805+0000","last_peered":"2024-03-26T03:50:14.163805+0000","last_clean":"2024-03-26T03:50:14.163805+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:50:14.163805+0000","last_undegraded":"2024-03-26T03:50:14.163805+0000","last_fullsized":"2024-03-26T03:50:14.163805+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:50:13.913348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.913348+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:33.729626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164947+0000","last_change":"2024-03-26T03:50:13.104332+0000","last_active":"2024-03-26T03:50:14.164947+0000","last_peered":"2024-03-26T03:50:14.164947+0000","last_clean":"2024-03-26T03:50:14.164947+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:50:14.164947+0000","last_undegraded":"2024-03-26T03:50:14.164947+0000","last_fullsized":"2024-03-26T03:50:14.164947+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.104219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.104219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.804503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460972+0000","last_change":"2024-03-26T03:49:58.807686+0000","last_active":"2024-03-26T03:50:14.460972+0000","last_peered":"2024-03-26T03:50:14.460972+0000","last_clean":"2024-03-26T03:50:14.460972+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:50:14.460972+0000","last_undegraded":"2024-03-26T03:50:14.460972+0000","last_fullsized":"2024-03-26T03:50:14.460972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.807606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.807606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:05.096069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:17.011359+0000","last_change":"2024-03-26T03:50:17.011382+0000","last_active":"2024-03-26T03:50:17.011359+0000","last_peered":"2024-03-26T03:50:17.011359+0000","last_clean":"2024-03-26T03:50:17.011359+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:50:17.011359+0000","last_undegraded":"2024-03-26T03:50:17.011359+0000","last_fullsized":"2024-03-26T03:50:17.011359+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:17.011284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:17.011284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:46.537867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462832+0000","last_change":"2024-03-26T03:50:08.854957+0000","last_active":"2024-03-26T03:50:14.462832+0000","last_peered":"2024-03-26T03:50:14.462832+0000","last_clean":"2024-03-26T03:50:14.462832+0000","last_became_active":"2024-03-26T03:48:44.013295+0000","last_became_peered":"2024-03-26T03:48:44.013295+0000","last_unstale":"2024-03-26T03:50:14.462832+0000","last_undegraded":"2024-03-26T03:50:14.462832+0000","last_fullsized":"2024-03-26T03:50:14.462832+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.854874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.854874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:37.587577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462007+0000","last_change":"2024-03-26T03:48:46.209925+0000","last_active":"2024-03-26T03:50:14.462007+0000","last_peered":"2024-03-26T03:50:14.462007+0000","last_clean":"2024-03-26T03:50:14.462007+0000","last_became_active":"2024-03-26T03:48:46.209516+0000","last_became_peered":"2024-03-26T03:48:46.209516+0000","last_unstale":"2024-03-26T03:50:14.462007+0000","last_undegraded":"2024-03-26T03:50:14.462007+0000","last_fullsized":"2024-03-26T03:50:14.462007+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163101+0000","last_change":"2024-03-26T03:48:50.040712+0000","last_active":"2024-03-26T03:50:14.163101+0000","last_peered":"2024-03-26T03:50:14.163101+0000","last_clean":"2024-03-26T03:50:14.163101+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:50:14.163101+0000","last_undegraded":"2024-03-26T03:50:14.163101+0000","last_fullsized":"2024-03-26T03:50:14.163101+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164079+0000","last_change":"2024-03-26T03:49:59.255850+0000","last_active":"2024-03-26T03:50:14.164079+0000","last_peered":"2024-03-26T03:50:14.164079+0000","last_clean":"2024-03-26T03:50:14.164079+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:50:14.164079+0000","last_undegraded":"2024-03-26T03:50:14.164079+0000","last_fullsized":"2024-03-26T03:50:14.164079+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.255770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:16.644194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461427+0000","last_change":"2024-03-26T03:49:57.098936+0000","last_active":"2024-03-26T03:50:14.461427+0000","last_peered":"2024-03-26T03:50:14.461427+0000","last_clean":"2024-03-26T03:50:14.461427+0000","last_became_active":"2024-03-26T03:48:44.016241+0000","last_became_peered":"2024-03-26T03:48:44.016241+0000","last_unstale":"2024-03-26T03:50:14.461427+0000","last_undegraded":"2024-03-26T03:50:14.461427+0000","last_fullsized":"2024-03-26T03:50:14.461427+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.098828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.273950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164020+0000","last_change":"2024-03-26T03:50:11.191322+0000","last_active":"2024-03-26T03:50:14.164020+0000","last_peered":"2024-03-26T03:50:14.164020+0000","last_clean":"2024-03-26T03:50:14.164020+0000","last_became_active":"2024-03-26T03:48:46.023348+0000","last_became_peered":"2024-03-26T03:48:46.023348+0000","last_unstale":"2024-03-26T03:50:14.164020+0000","last_undegraded":"2024-03-26T03:50:14.164020+0000","last_fullsized":"2024-03-26T03:50:14.164020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:11.191227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.191227+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.173879+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660784+0000","last_change":"2024-03-26T03:48:50.044572+0000","last_active":"2024-03-26T03:50:09.660784+0000","last_peered":"2024-03-26T03:50:09.660784+0000","last_clean":"2024-03-26T03:50:09.660784+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:50:09.660784+0000","last_undegraded":"2024-03-26T03:50:09.660784+0000","last_fullsized":"2024-03-26T03:50:09.660784+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.460981+0000","last_change":"2024-03-26T03:48:47.012857+0000","last_active":"2024-03-26T03:50:14.460981+0000","last_peered":"2024-03-26T03:50:14.460981+0000","last_clean":"2024-03-26T03:50:14.460981+0000","last_became_active":"2024-03-26T03:48:43.008452+0000","last_became_peered":"2024-03-26T03:48:43.008452+0000","last_unstale":"2024-03-26T03:50:14.460981+0000","last_undegraded":"2024-03-26T03:50:14.460981+0000","last_fullsized":"2024-03-26T03:50:14.460981+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:48:41.993340+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163771+0000","last_change":"2024-03-26T03:48:44.012922+0000","last_active":"2024-03-26T03:50:14.163771+0000","last_peered":"2024-03-26T03:50:14.163771+0000","last_clean":"2024-03-26T03:50:14.163771+0000","last_became_active":"2024-03-26T03:48:44.012147+0000","last_became_peered":"2024-03-26T03:48:44.012147+0000","last_unstale":"2024-03-26T03:50:14.163771+0000","last_undegraded":"2024-03-26T03:50:14.163771+0000","last_fullsized":"2024-03-26T03:50:14.163771+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":140,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.194022+0000","last_change":"2024-03-26T03:50:13.007007+0000","last_active":"2024-03-26T03:50:14.194022+0000","last_peered":"2024-03-26T03:50:14.194022+0000","last_clean":"2024-03-26T03:50:14.194022+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:50:14.194022+0000","last_undegraded":"2024-03-26T03:50:14.194022+0000","last_fullsized":"2024-03-26T03:50:14.194022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:50:13.006886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.006886+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.897597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.195791+0000","last_change":"2024-03-26T03:50:14.195816+0000","last_active":"2024-03-26T03:50:14.195791+0000","last_peered":"2024-03-26T03:50:14.195791+0000","last_clean":"2024-03-26T03:50:14.195791+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:50:14.195791+0000","last_undegraded":"2024-03-26T03:50:14.195791+0000","last_fullsized":"2024-03-26T03:50:14.195791+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:14.195713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:14.195713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.803072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462915+0000","last_change":"2024-03-26T03:50:11.829217+0000","last_active":"2024-03-26T03:50:14.462915+0000","last_peered":"2024-03-26T03:50:14.462915+0000","last_clean":"2024-03-26T03:50:14.462915+0000","last_became_active":"2024-03-26T03:48:50.262294+0000","last_became_peered":"2024-03-26T03:48:50.262294+0000","last_unstale":"2024-03-26T03:50:14.462915+0000","last_undegraded":"2024-03-26T03:50:14.462915+0000","last_fullsized":"2024-03-26T03:50:14.462915+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:11.829105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.829105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.079422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193537+0000","last_change":"2024-03-26T03:50:05.944125+0000","last_active":"2024-03-26T03:50:14.193537+0000","last_peered":"2024-03-26T03:50:14.193537+0000","last_clean":"2024-03-26T03:50:14.193537+0000","last_became_active":"2024-03-26T03:48:44.014239+0000","last_became_peered":"2024-03-26T03:48:44.014239+0000","last_unstale":"2024-03-26T03:50:14.193537+0000","last_undegraded":"2024-03-26T03:50:14.193537+0000","last_fullsized":"2024-03-26T03:50:14.193537+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:05.944040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.944040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.216854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":74,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163599+0000","last_change":"2024-03-26T03:49:48.295417+0000","last_active":"2024-03-26T03:50:14.163599+0000","last_peered":"2024-03-26T03:50:14.163599+0000","last_clean":"2024-03-26T03:50:14.163599+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:50:14.163599+0000","last_undegraded":"2024-03-26T03:50:14.163599+0000","last_fullsized":"2024-03-26T03:50:14.163599+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.295312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.295312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:02.045160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":99,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.194182+0000","last_change":"2024-03-26T03:50:13.971916+0000","last_active":"2024-03-26T03:50:14.194182+0000","last_peered":"2024-03-26T03:50:14.194182+0000","last_clean":"2024-03-26T03:50:14.194182+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:50:14.194182+0000","last_undegraded":"2024-03-26T03:50:14.194182+0000","last_fullsized":"2024-03-26T03:50:14.194182+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:50:13.971792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.971792+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:34.626065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.165211+0000","last_change":"2024-03-26T03:48:48.020655+0000","last_active":"2024-03-26T03:50:14.165211+0000","last_peered":"2024-03-26T03:50:14.165211+0000","last_clean":"2024-03-26T03:50:14.165211+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:50:14.165211+0000","last_undegraded":"2024-03-26T03:50:14.165211+0000","last_fullsized":"2024-03-26T03:50:14.165211+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:15.127534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:17.931853+0000","last_change":"2024-03-26T03:50:17.931877+0000","last_active":"2024-03-26T03:50:17.931853+0000","last_peered":"2024-03-26T03:50:17.931853+0000","last_clean":"2024-03-26T03:50:17.931853+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:50:17.931853+0000","last_undegraded":"2024-03-26T03:50:17.931853+0000","last_fullsized":"2024-03-26T03:50:17.931853+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:17.931766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:17.931766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.013525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":305,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462086+0000","last_change":"2024-03-26T03:48:51.651891+0000","last_active":"2024-03-26T03:50:14.462086+0000","last_peered":"2024-03-26T03:50:14.462086+0000","last_clean":"2024-03-26T03:50:14.462086+0000","last_became_active":"2024-03-26T03:48:34.401623+0000","last_became_peered":"2024-03-26T03:48:34.401623+0000","last_unstale":"2024-03-26T03:50:14.462086+0000","last_undegraded":"2024-03-26T03:50:14.462086+0000","last_fullsized":"2024-03-26T03:50:14.462086+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"20'117","last_scrub_stamp":"2024-03-26T03:48:51.651774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:48:51.651774+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:20.226983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660501+0000","last_change":"2024-03-26T03:48:44.017929+0000","last_active":"2024-03-26T03:50:09.660501+0000","last_peered":"2024-03-26T03:50:09.660501+0000","last_clean":"2024-03-26T03:50:09.660501+0000","last_became_active":"2024-03-26T03:48:44.017637+0000","last_became_peered":"2024-03-26T03:48:44.017637+0000","last_unstale":"2024-03-26T03:50:09.660501+0000","last_undegraded":"2024-03-26T03:50:09.660501+0000","last_fullsized":"2024-03-26T03:50:09.660501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:12.329800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461210+0000","last_change":"2024-03-26T03:50:06.585221+0000","last_active":"2024-03-26T03:50:14.461210+0000","last_peered":"2024-03-26T03:50:14.461210+0000","last_clean":"2024-03-26T03:50:14.461210+0000","last_became_active":"2024-03-26T03:48:46.210282+0000","last_became_peered":"2024-03-26T03:48:46.210282+0000","last_unstale":"2024-03-26T03:50:14.461210+0000","last_undegraded":"2024-03-26T03:50:14.461210+0000","last_fullsized":"2024-03-26T03:50:14.461210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:50:06.585054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.585054+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.951132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462596+0000","last_change":"2024-03-26T03:50:00.772766+0000","last_active":"2024-03-26T03:50:14.462596+0000","last_peered":"2024-03-26T03:50:14.462596+0000","last_clean":"2024-03-26T03:50:14.462596+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:50:14.462596+0000","last_undegraded":"2024-03-26T03:50:14.462596+0000","last_fullsized":"2024-03-26T03:50:14.462596+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.772625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.974412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193841+0000","last_change":"2024-03-26T03:48:50.043668+0000","last_active":"2024-03-26T03:50:14.193841+0000","last_peered":"2024-03-26T03:50:14.193841+0000","last_clean":"2024-03-26T03:50:14.193841+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:50:14.193841+0000","last_undegraded":"2024-03-26T03:50:14.193841+0000","last_fullsized":"2024-03-26T03:50:14.193841+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:18.248495+0000","last_change":"2024-03-26T03:50:18.248516+0000","last_active":"2024-03-26T03:50:18.248495+0000","last_peered":"2024-03-26T03:50:18.248495+0000","last_clean":"2024-03-26T03:50:18.248495+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:50:18.248495+0000","last_undegraded":"2024-03-26T03:50:18.248495+0000","last_fullsized":"2024-03-26T03:50:18.248495+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:18.248434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.248434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.519794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":165,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:17.937061+0000","last_change":"2024-03-26T03:49:45.618585+0000","last_active":"2024-03-26T03:50:17.937061+0000","last_peered":"2024-03-26T03:50:17.937061+0000","last_clean":"2024-03-26T03:50:17.937061+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:50:17.937061+0000","last_undegraded":"2024-03-26T03:50:17.937061+0000","last_fullsized":"2024-03-26T03:50:17.937061+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:49:45.618470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:45.618470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:56.990274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":71,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163729+0000","last_change":"2024-03-26T03:48:46.026122+0000","last_active":"2024-03-26T03:50:14.163729+0000","last_peered":"2024-03-26T03:50:14.163729+0000","last_clean":"2024-03-26T03:50:14.163729+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:50:14.163729+0000","last_undegraded":"2024-03-26T03:50:14.163729+0000","last_fullsized":"2024-03-26T03:50:14.163729+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:18.981679+0000","last_change":"2024-03-26T03:50:18.981723+0000","last_active":"2024-03-26T03:50:18.981679+0000","last_peered":"2024-03-26T03:50:18.981679+0000","last_clean":"2024-03-26T03:50:18.981679+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:50:18.981679+0000","last_undegraded":"2024-03-26T03:50:18.981679+0000","last_fullsized":"2024-03-26T03:50:18.981679+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:18.981529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.981529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:41.258512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":44,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193486+0000","last_change":"2024-03-26T03:48:50.260878+0000","last_active":"2024-03-26T03:50:14.193486+0000","last_peered":"2024-03-26T03:50:14.193486+0000","last_clean":"2024-03-26T03:50:14.193486+0000","last_became_active":"2024-03-26T03:48:50.260476+0000","last_became_peered":"2024-03-26T03:48:50.260476+0000","last_unstale":"2024-03-26T03:50:14.193486+0000","last_undegraded":"2024-03-26T03:50:14.193486+0000","last_fullsized":"2024-03-26T03:50:14.193486+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":64,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193927+0000","last_change":"2024-03-26T03:50:00.032886+0000","last_active":"2024-03-26T03:50:14.193927+0000","last_peered":"2024-03-26T03:50:14.193927+0000","last_clean":"2024-03-26T03:50:14.193927+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:50:14.193927+0000","last_undegraded":"2024-03-26T03:50:14.193927+0000","last_fullsized":"2024-03-26T03:50:14.193927+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.032763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.032763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:06.902858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":109,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163865+0000","last_change":"2024-03-26T03:50:03.113111+0000","last_active":"2024-03-26T03:50:14.163865+0000","last_peered":"2024-03-26T03:50:14.163865+0000","last_clean":"2024-03-26T03:50:14.163865+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:50:14.163865+0000","last_undegraded":"2024-03-26T03:50:14.163865+0000","last_fullsized":"2024-03-26T03:50:14.163865+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:03.112983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.112983+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.126914+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462254+0000","last_change":"2024-03-26T03:49:54.591020+0000","last_active":"2024-03-26T03:50:14.462254+0000","last_peered":"2024-03-26T03:50:14.462254+0000","last_clean":"2024-03-26T03:50:14.462254+0000","last_became_active":"2024-03-26T03:48:48.021249+0000","last_became_peered":"2024-03-26T03:48:48.021249+0000","last_unstale":"2024-03-26T03:50:14.462254+0000","last_undegraded":"2024-03-26T03:50:14.462254+0000","last_fullsized":"2024-03-26T03:50:14.462254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:54.590915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.590915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.886951+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163086+0000","last_change":"2024-03-26T03:48:50.045296+0000","last_active":"2024-03-26T03:50:14.163086+0000","last_peered":"2024-03-26T03:50:14.163086+0000","last_clean":"2024-03-26T03:50:14.163086+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:50:14.163086+0000","last_undegraded":"2024-03-26T03:50:14.163086+0000","last_fullsized":"2024-03-26T03:50:14.163086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:16.928066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461790+0000","last_change":"2024-03-26T03:48:44.014959+0000","last_active":"2024-03-26T03:50:14.461790+0000","last_peered":"2024-03-26T03:50:14.461790+0000","last_clean":"2024-03-26T03:50:14.461790+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:50:14.461790+0000","last_undegraded":"2024-03-26T03:50:14.461790+0000","last_fullsized":"2024-03-26T03:50:14.461790+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":87,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.193775+0000","last_change":"2024-03-26T03:49:46.927333+0000","last_active":"2024-03-26T03:50:14.193775+0000","last_peered":"2024-03-26T03:50:14.193775+0000","last_clean":"2024-03-26T03:50:14.193775+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:50:14.193775+0000","last_undegraded":"2024-03-26T03:50:14.193775+0000","last_fullsized":"2024-03-26T03:50:14.193775+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:49:46.927230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:46.927230+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:54.304888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":77,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163599+0000","last_change":"2024-03-26T03:48:48.019177+0000","last_active":"2024-03-26T03:50:14.163599+0000","last_peered":"2024-03-26T03:50:14.163599+0000","last_clean":"2024-03-26T03:50:14.163599+0000","last_became_active":"2024-03-26T03:48:48.018852+0000","last_became_peered":"2024-03-26T03:48:48.018852+0000","last_unstale":"2024-03-26T03:50:14.163599+0000","last_undegraded":"2024-03-26T03:50:14.163599+0000","last_fullsized":"2024-03-26T03:50:14.163599+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462627+0000","last_change":"2024-03-26T03:49:57.800214+0000","last_active":"2024-03-26T03:50:14.462627+0000","last_peered":"2024-03-26T03:50:14.462627+0000","last_clean":"2024-03-26T03:50:14.462627+0000","last_became_active":"2024-03-26T03:48:50.262788+0000","last_became_peered":"2024-03-26T03:48:50.262788+0000","last_unstale":"2024-03-26T03:50:14.462627+0000","last_undegraded":"2024-03-26T03:50:14.462627+0000","last_fullsized":"2024-03-26T03:50:14.462627+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.800113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.800113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:00.814334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":66,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163753+0000","last_change":"2024-03-26T03:50:02.092039+0000","last_active":"2024-03-26T03:50:14.163753+0000","last_peered":"2024-03-26T03:50:14.163753+0000","last_clean":"2024-03-26T03:50:14.163753+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:50:14.163753+0000","last_undegraded":"2024-03-26T03:50:14.163753+0000","last_fullsized":"2024-03-26T03:50:14.163753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:02.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.091909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:03.088925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":94,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163781+0000","last_change":"2024-03-26T03:50:06.823874+0000","last_active":"2024-03-26T03:50:14.163781+0000","last_peered":"2024-03-26T03:50:14.163781+0000","last_clean":"2024-03-26T03:50:14.163781+0000","last_became_active":"2024-03-26T03:48:46.016634+0000","last_became_peered":"2024-03-26T03:48:46.016634+0000","last_unstale":"2024-03-26T03:50:14.163781+0000","last_undegraded":"2024-03-26T03:50:14.163781+0000","last_fullsized":"2024-03-26T03:50:14.163781+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:50:06.823723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.823723+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.856329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":82,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:10.088926+0000","last_change":"2024-03-26T03:50:10.088926+0000","last_active":"2024-03-26T03:50:10.088926+0000","last_peered":"2024-03-26T03:50:10.088926+0000","last_clean":"2024-03-26T03:50:10.088926+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:50:10.088926+0000","last_undegraded":"2024-03-26T03:50:10.088926+0000","last_fullsized":"2024-03-26T03:50:10.088926+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:10.088836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:10.088836+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:10.494186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163666+0000","last_change":"2024-03-26T03:48:50.043708+0000","last_active":"2024-03-26T03:50:14.163666+0000","last_peered":"2024-03-26T03:50:14.163666+0000","last_clean":"2024-03-26T03:50:14.163666+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:50:14.163666+0000","last_undegraded":"2024-03-26T03:50:14.163666+0000","last_fullsized":"2024-03-26T03:50:14.163666+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:18.534183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660911+0000","last_change":"2024-03-26T03:48:44.013652+0000","last_active":"2024-03-26T03:50:09.660911+0000","last_peered":"2024-03-26T03:50:09.660911+0000","last_clean":"2024-03-26T03:50:09.660911+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:50:09.660911+0000","last_undegraded":"2024-03-26T03:50:09.660911+0000","last_fullsized":"2024-03-26T03:50:09.660911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":125,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660355+0000","last_change":"2024-03-26T03:49:51.111438+0000","last_active":"2024-03-26T03:50:09.660355+0000","last_peered":"2024-03-26T03:50:09.660355+0000","last_clean":"2024-03-26T03:50:09.660355+0000","last_became_active":"2024-03-26T03:48:46.209319+0000","last_became_peered":"2024-03-26T03:48:46.209319+0000","last_unstale":"2024-03-26T03:50:09.660355+0000","last_undegraded":"2024-03-26T03:50:09.660355+0000","last_fullsized":"2024-03-26T03:50:09.660355+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.111316+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:53.974577+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:16.977925+0000","last_change":"2024-03-26T03:50:16.977948+0000","last_active":"2024-03-26T03:50:16.977925+0000","last_peered":"2024-03-26T03:50:16.977925+0000","last_clean":"2024-03-26T03:50:16.977925+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:50:16.977925+0000","last_undegraded":"2024-03-26T03:50:16.977925+0000","last_fullsized":"2024-03-26T03:50:16.977925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.977862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.490601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461288+0000","last_change":"2024-03-26T03:48:50.034819+0000","last_active":"2024-03-26T03:50:14.461288+0000","last_peered":"2024-03-26T03:50:14.461288+0000","last_clean":"2024-03-26T03:50:14.461288+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:50:14.461288+0000","last_undegraded":"2024-03-26T03:50:14.461288+0000","last_fullsized":"2024-03-26T03:50:14.461288+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164462+0000","last_change":"2024-03-26T03:50:05.869446+0000","last_active":"2024-03-26T03:50:14.164462+0000","last_peered":"2024-03-26T03:50:14.164462+0000","last_clean":"2024-03-26T03:50:14.164462+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:50:14.164462+0000","last_undegraded":"2024-03-26T03:50:14.164462+0000","last_fullsized":"2024-03-26T03:50:14.164462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:05.869359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.869359+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.254418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660271+0000","last_change":"2024-03-26T03:48:44.017015+0000","last_active":"2024-03-26T03:50:09.660271+0000","last_peered":"2024-03-26T03:50:09.660271+0000","last_clean":"2024-03-26T03:50:09.660271+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:50:09.660271+0000","last_undegraded":"2024-03-26T03:50:09.660271+0000","last_fullsized":"2024-03-26T03:50:09.660271+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.194145+0000","last_change":"2024-03-26T03:50:09.997564+0000","last_active":"2024-03-26T03:50:14.194145+0000","last_peered":"2024-03-26T03:50:14.194145+0000","last_clean":"2024-03-26T03:50:14.194145+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:50:14.194145+0000","last_undegraded":"2024-03-26T03:50:14.194145+0000","last_fullsized":"2024-03-26T03:50:14.194145+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:09.997449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.997449+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.384605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":43,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660561+0000","last_change":"2024-03-26T03:48:50.261659+0000","last_active":"2024-03-26T03:50:09.660561+0000","last_peered":"2024-03-26T03:50:09.660561+0000","last_clean":"2024-03-26T03:50:09.660561+0000","last_became_active":"2024-03-26T03:48:50.260697+0000","last_became_peered":"2024-03-26T03:48:50.260697+0000","last_unstale":"2024-03-26T03:50:09.660561+0000","last_undegraded":"2024-03-26T03:50:09.660561+0000","last_fullsized":"2024-03-26T03:50:09.660561+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.461838+0000","last_change":"2024-03-26T03:50:02.189077+0000","last_active":"2024-03-26T03:50:14.461838+0000","last_peered":"2024-03-26T03:50:14.461838+0000","last_clean":"2024-03-26T03:50:14.461838+0000","last_became_active":"2024-03-26T03:48:48.019448+0000","last_became_peered":"2024-03-26T03:48:48.019448+0000","last_unstale":"2024-03-26T03:50:14.461838+0000","last_undegraded":"2024-03-26T03:50:14.461838+0000","last_fullsized":"2024-03-26T03:50:14.461838+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.188975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.969923+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163594+0000","last_change":"2024-03-26T03:49:48.865134+0000","last_active":"2024-03-26T03:50:14.163594+0000","last_peered":"2024-03-26T03:50:14.163594+0000","last_clean":"2024-03-26T03:50:14.163594+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:50:14.163594+0000","last_undegraded":"2024-03-26T03:50:14.163594+0000","last_fullsized":"2024-03-26T03:50:14.163594+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.864997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:01.477744+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":92,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164319+0000","last_change":"2024-03-26T03:50:11.925898+0000","last_active":"2024-03-26T03:50:14.164319+0000","last_peered":"2024-03-26T03:50:14.164319+0000","last_clean":"2024-03-26T03:50:14.164319+0000","last_became_active":"2024-03-26T03:48:46.018601+0000","last_became_peered":"2024-03-26T03:48:46.018601+0000","last_unstale":"2024-03-26T03:50:14.164319+0000","last_undegraded":"2024-03-26T03:50:14.164319+0000","last_fullsized":"2024-03-26T03:50:14.164319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:11.925762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.925762+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.777765+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.165240+0000","last_change":"2024-03-26T03:50:07.213828+0000","last_active":"2024-03-26T03:50:14.165240+0000","last_peered":"2024-03-26T03:50:14.165240+0000","last_clean":"2024-03-26T03:50:14.165240+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:50:14.165240+0000","last_undegraded":"2024-03-26T03:50:14.165240+0000","last_fullsized":"2024-03-26T03:50:14.165240+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.213705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.213705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:19.297785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":56,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.194135+0000","last_change":"2024-03-26T03:50:12.050853+0000","last_active":"2024-03-26T03:50:14.194135+0000","last_peered":"2024-03-26T03:50:14.194135+0000","last_clean":"2024-03-26T03:50:14.194135+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:50:14.194135+0000","last_undegraded":"2024-03-26T03:50:14.194135+0000","last_fullsized":"2024-03-26T03:50:14.194135+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.050772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.050772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.794737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.462337+0000","last_change":"2024-03-26T03:49:51.516987+0000","last_active":"2024-03-26T03:50:14.462337+0000","last_peered":"2024-03-26T03:50:14.462337+0000","last_clean":"2024-03-26T03:50:14.462337+0000","last_became_active":"2024-03-26T03:48:44.012660+0000","last_became_peered":"2024-03-26T03:48:44.012660+0000","last_unstale":"2024-03-26T03:50:14.462337+0000","last_undegraded":"2024-03-26T03:50:14.462337+0000","last_fullsized":"2024-03-26T03:50:14.462337+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:51.516891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.516891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:57.433207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164346+0000","last_change":"2024-03-26T03:50:03.880855+0000","last_active":"2024-03-26T03:50:14.164346+0000","last_peered":"2024-03-26T03:50:14.164346+0000","last_clean":"2024-03-26T03:50:14.164346+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:50:14.164346+0000","last_undegraded":"2024-03-26T03:50:14.164346+0000","last_fullsized":"2024-03-26T03:50:14.164346+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:03.880711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.880711+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.791446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164474+0000","last_change":"2024-03-26T03:50:02.877487+0000","last_active":"2024-03-26T03:50:14.164474+0000","last_peered":"2024-03-26T03:50:14.164474+0000","last_clean":"2024-03-26T03:50:14.164474+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:50:14.164474+0000","last_undegraded":"2024-03-26T03:50:14.164474+0000","last_fullsized":"2024-03-26T03:50:14.164474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.877373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.877373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.008619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.163482+0000","last_change":"2024-03-26T03:50:02.300700+0000","last_active":"2024-03-26T03:50:14.163482+0000","last_peered":"2024-03-26T03:50:14.163482+0000","last_clean":"2024-03-26T03:50:14.163482+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:50:14.163482+0000","last_undegraded":"2024-03-26T03:50:14.163482+0000","last_fullsized":"2024-03-26T03:50:14.163482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:02.300594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.300594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.021817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":55,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-26T03:50:09.660628+0000","last_change":"2024-03-26T03:48:44.015913+0000","last_active":"2024-03-26T03:50:09.660628+0000","last_peered":"2024-03-26T03:50:09.660628+0000","last_clean":"2024-03-26T03:50:09.660628+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:50:09.660628+0000","last_undegraded":"2024-03-26T03:50:09.660628+0000","last_fullsized":"2024-03-26T03:50:09.660628+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:50:09.708831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":68,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-26T03:50:14.164153+0000","last_change":"2024-03-26T03:48:46.023356+0000","last_active":"2024-03-26T03:50:14.164153+0000","last_peered":"2024-03-26T03:50:14.164153+0000","last_clean":"2024-03-26T03:50:14.164153+0000","last_became_active":"2024-03-26T03:48:46.022937+0000","last_became_peered":"2024-03-26T03:48:46.022937+0000","last_unstale":"2024-03-26T03:50:14.164153+0000","last_undegraded":"2024-03-26T03:50:14.164153+0000","last_fullsized":"2024-03-26T03:50:14.164153+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":599,"ondisk_log_size":599,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070487,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28404,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716748,"statfs":{"total":95995035648,"available":95965949952,"internally_reserved":0,"allocated":1421312,"data_stored":948407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364829,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32536,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712616,"statfs":{"total":95995035648,"available":95961718784,"internally_reserved":0,"allocated":1388544,"data_stored":945995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32016,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713136,"statfs":{"total":95995035648,"available":95962251264,"internally_reserved":0,"allocated":860160,"data_stored":357412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32568,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712584,"statfs":{"total":95995035648,"available":95961686016,"internally_reserved":0,"allocated":1421312,"data_stored":948136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31952,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713200,"statfs":{"total":95995035648,"available":95962316800,"internally_reserved":0,"allocated":794624,"data_stored":355755,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31984,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713168,"statfs":{"total":95995035648,"available":95962284032,"internally_reserved":0,"allocated":827392,"data_stored":351493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":827392,"data_stored":356245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717356,"statfs":{"total":95995035648,"available":95966572544,"internally_reserved":0,"allocated":798720,"data_stored":355953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:50:21.803 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T03:50:21.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.15 deep-scrub starts 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.15 deep-scrub ok 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.11 deep-scrub starts 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.11 deep-scrub ok 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T03:50:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: Deploying daemon node-exporter.b on smithi086 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T03:50:21.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:21 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T03:50:21.839 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:21 smithi086 bash[64390]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T03:50:22.060 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:22.338 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:22 smithi086 bash[64390]: Getting image source signatures 2024-03-26T03:50:22.338 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:22 smithi086 bash[64390]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T03:50:22.338 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:22 smithi086 bash[64390]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T03:50:22.338 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:22 smithi086 bash[64390]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T03:50:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T03:50:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T03:50:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: 6.13 deep-scrub starts 2024-03-26T03:50:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: 6.13 deep-scrub ok 2024-03-26T03:50:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:22 smithi086 ceph-mon[32965]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: 6.13 deep-scrub starts 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: 6.13 deep-scrub ok 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[29446]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T03:50:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T03:50:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: 6.13 deep-scrub starts 2024-03-26T03:50:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: 6.13 deep-scrub ok 2024-03-26T03:50:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:50:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:22 smithi006 ceph-mon[37501]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 4.1b scrub starts 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 4.1b scrub ok 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 4.1b scrub starts 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 4.1b scrub ok 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T03:50:23.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:23.837 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:23 smithi086 bash[64390]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T03:50:23.837 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:23 smithi086 bash[64390]: Writing manifest to image destination 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 4.1b scrub starts 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 4.1b scrub ok 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-03-26T03:50:23.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-03-26T03:50:23.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 90 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:fsid de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:created 2024-03-26T03:43:14.534401+0000 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:modified 2024-03-26T03:50:23.513811+0000 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:crush_version 18 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:full_ratio 0.95 2024-03-26T03:50:24.147 INFO:teuthology.orchestra.run.smithi006.stdout:backfillfull_ratio 0.9 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:nearfull_ratio 0.85 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:require_min_compat_client luminous 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:min_compat_client luminous 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:require_osd_release squid 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:stretch_mode_enabled false 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 69 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:max_osd 8 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6801/1246039834 v2:172.21.15.6:6802/1246039834 exists,up 1d1c84cf-5fa7-437e-8004-3b7aca831d6e 2024-03-26T03:50:24.148 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6805/260311411 v2:172.21.15.6:6806/260311411 exists,up 753f7459-62ec-447b-a0da-64d06e813f16 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6809/3103276794 v2:172.21.15.6:6810/3103276794 exists,up b7d7409f-341a-4711-8570-92fe9082a528 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6813/2795883342 v2:172.21.15.6:6814/2795883342 exists,up a91459a0-dee4-420c-ae2f-940756c985a6 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4 up in weight 1 up_from 32 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6800/931538411 v2:172.21.15.86:6801/931538411 exists,up 087da212-7bf7-4982-b7cc-11f3f66c7ff8 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6804/529279948 v2:172.21.15.86:6805/529279948 exists,up 7a688745-b213-4bbc-a735-42ffd2a5bfdc 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6 up in weight 1 up_from 50 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6808/1580133777 v2:172.21.15.86:6809/1580133777 exists,up 58b58555-59a3-4620-a15f-024f7041713f 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6812/3905426306 v2:172.21.15.86:6813/3905426306 exists,up 49b1833d-0a24-4168-a195-47b8ac1d9bc7 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 4.b [1,5] 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 4.17 [1,2] 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:pg_upmap_items 4.1f [1,2] 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1978459317 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1771670410 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/3964053112 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/14180211 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6800/2547319348 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.149 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/601470125 expires 2024-03-27T03:44:16.095969+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6800/3870917014 expires 2024-03-27T03:44:16.095969+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1892511221 expires 2024-03-27T03:43:49.288943+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/435737617 expires 2024-03-27T03:43:49.288943+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/93567168 expires 2024-03-27T03:49:23.135700+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/2150730999 expires 2024-03-27T03:43:49.288943+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:6800/1166209788 expires 2024-03-27T03:43:49.288943+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1248481299 expires 2024-03-27T03:44:16.095969+0000 2024-03-26T03:50:24.150 INFO:teuthology.orchestra.run.smithi006.stdout:blocklist 172.21.15.6:0/1832156537 expires 2024-03-27T03:44:16.095969+0000 2024-03-26T03:50:24.257 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:24.257 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:23 smithi086 podman[64390]: 2024-03-26 03:50:23.898136454 +0000 UTC m=+2.331783653 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T03:50:24.258 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 podman[64390]: 2024-03-26 03:50:24.02183656 +0000 UTC m=+2.455483763 container create bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:50:24.258 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 podman[64390]: 2024-03-26 03:50:24.257486748 +0000 UTC m=+2.691133946 container init bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:50:24.428 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:50:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 podman[64390]: 2024-03-26 03:50:24.260255111 +0000 UTC m=+2.693902318 container start bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.263Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.263Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.264Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.264Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T03:50:24.525 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T03:50:24.526 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T03:50:24.527 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T03:50:24.528 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.265Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.266Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b[64440]: ts=2024-03-26T03:50:24.266Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 bash[64390]: bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 2024-03-26T03:50:24.529 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 03:50:24 smithi086 systemd[1]: Started Ceph node-exporter.b for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:24.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: 3.3 deep-scrub starts 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: 3.3 deep-scrub ok 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1407578290' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.788 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: 3.3 deep-scrub starts 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: 3.3 deep-scrub ok 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1407578290' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.789 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.790 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.790 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.816 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6801/1246039834 v2:172.21.15.6:6802/1246039834 exists,up 1d1c84cf-5fa7-437e-8004-3b7aca831d6e', 'osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6805/260311411 v2:172.21.15.6:6806/260311411 exists,up 753f7459-62ec-447b-a0da-64d06e813f16', 'osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6809/3103276794 v2:172.21.15.6:6810/3103276794 exists,up b7d7409f-341a-4711-8570-92fe9082a528', 'osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.6:6813/2795883342 v2:172.21.15.6:6814/2795883342 exists,up a91459a0-dee4-420c-ae2f-940756c985a6', 'osd.4 up in weight 1 up_from 32 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6800/931538411 v2:172.21.15.86:6801/931538411 exists,up 087da212-7bf7-4982-b7cc-11f3f66c7ff8', 'osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6804/529279948 v2:172.21.15.86:6805/529279948 exists,up 7a688745-b213-4bbc-a735-42ffd2a5bfdc', 'osd.6 up in weight 1 up_from 50 up_thru 69 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6808/1580133777 v2:172.21.15.86:6809/1580133777 exists,up 58b58555-59a3-4620-a15f-024f7041713f', 'osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) v2:172.21.15.86:6812/3905426306 v2:172.21.15.86:6813/3905426306 exists,up 49b1833d-0a24-4168-a195-47b8ac1d9bc7'] 2024-03-26T03:50:24.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon mon.a config show 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: 3.3 deep-scrub starts 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: 3.3 deep-scrub ok 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 2]}]': finished 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1407578290' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:24.871 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:25.146 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T03:50:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:25 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T03:50:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:25 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T03:50:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T03:50:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T03:50:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:50:26.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T03:50:26.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T03:50:26.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:50:26.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:26 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:50:26.657 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T03:50:26.657 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: pgmap v47: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T03:50:26.658 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:50:26.659 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:26 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:50:27.602 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:50:27.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:50:27.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[37501]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:50:27.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:50:27.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:50:27.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:27 smithi006 ceph-mon[29446]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout: "name": "mon.a", 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster": "ceph", 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout: "admin_socket_mode": "", 2024-03-26T03:50:27.630 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_client_required": "cephx, none", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_cluster_required": "cephx", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_debug": "true", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_required": "cephx", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "auth_supported": "", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio": "true", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_poll_ms": "250", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_aio_reap_max": "16", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_async_discard_threads": "0", 2024-03-26T03:50:27.631 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_block_size": "4096", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_debug_inflight_ios": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_enable_discard": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry": "3", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash": "0", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_hipri": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-26T03:50:27.632 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bdev_type": "", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_alloc_size": "1048576", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_allocator": "hybrid", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_buffered_io": "true", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_for_zeros": "false", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_compact_log_sync": "false", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-26T03:50:27.633 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_log_runway": "4194304", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_max_prefetch": "1048576", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_flush_size": "524288", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_min_log_runway": "1048576", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery": "false", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluefs_sync_write": "false", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocation_from_file": "true", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_allocator": "hybrid", 2024-03-26T03:50:27.634 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_create": "true", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_create": "false", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_path": "", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_db_size": "0", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_path": "", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_preallocate_file": "false", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_size": "107374182400", 2024-03-26T03:50:27.635 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_create": "false", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_path": "", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_block_wal_size": "100663296", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs": "true", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune": "true", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-26T03:50:27.636 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size": "0", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cache_type": "2q", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_clone_cow": "true", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-26T03:50:27.637 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_mode": "none", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_csum_type": "crc32c", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_freelist": "false", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-26T03:50:27.638 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_small_allocations": "0", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_read": "true", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_default_buffered_write": "false", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-26T03:50:27.639 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fail_eio": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-26T03:50:27.640 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount": "false", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_ignore_data_csum": "false", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-26T03:50:27.641 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_log_op_age": "5.000000", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_alloc_size": "0", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size": "0", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_max_deferred_txc": "32", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size": "0", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_nid_prealloc": "1024", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-26T03:50:27.642 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_retry_disk_reads": "3", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cf": "true", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_spdk_mem": "512", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-26T03:50:27.643 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_tracing": "false", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "bluestore_zero_block_detection": "false", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-26T03:50:27.644 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_signatures": "false", 2024-03-26T03:50:27.645 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_cluster_require_version": "2", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_signatures": "false", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_require_version": "2", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_signatures": "false", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_service_require_version": "2", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "cephx_sign_messages": "true", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "chdir": "", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_acl_type": "", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_asio_thread_count": "2", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_mid": "0.750000", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_cache_size": "16384", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_caps_release_delay": "5", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_check_pool_perm": "true", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-26T03:50:27.646 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_force_sync_read": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_getattr_caps": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_debug_inject_tick_delay": "0", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_die_on_failed_remount": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_dirsize_rbytes": "true", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_force_lazyio": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_fs": "", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_inject_release_failure": "false", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_inline_size": "4096", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mds_namespace": "", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_metadata": "", 2024-03-26T03:50:27.647 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_gid": "-1", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_timeout": "300", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mount_uid": "-1", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_mountpoint": "/", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_notify_timeout": "10", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc": "true", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty": "104857600", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_max_objects": "1000", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_size": "209715200", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_oc_target_dirty": "8388608", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_permissions": "true", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota": "true", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_quota_df": "true", 2024-03-26T03:50:27.648 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_bytes": "0", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_max_periods": "4", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_readahead_min": "131072", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_reconnect_stale": "false", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_shutdown_timeout": "30", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_snapdir": ".snap", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_tick_interval": "1", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_trace": "", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_try_dentry_invalidate": "false", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_faked_inos": "false", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "client_use_random_mds": "false", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog": "false", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_graylog_port": "12201", 2024-03-26T03:50:27.649 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_monitors": "default=true", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog": "false", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "clog_to_syslog_level": "info", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "-", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network": "", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_network_interface": "", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_isal": "false", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_level": "5", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zlib_winsize": "-15", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "compressor_zstd_level": "1", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_op_num_threads": "6", 2024-03-26T03:50:27.650 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crimson_seastar_cpu_cores": "", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location": "", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook": "", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_location_hook_timeout": "10", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "daemonize": "false", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "daos_pool": "tank", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_alienstore": "0/5", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_allow_any_pool_priority": "false", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok": "1/5", 2024-03-26T03:50:27.651 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asok_assert_abort": "false", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_asserts_on_shutdown": "true", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_auth": "1/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bdev": "1/3", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluefs": "1/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_bluestore": "1/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_buffer": "0/1", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ceph_exporter": "1/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephfs_mirror": "0/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cephsqlite": "0/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_client": "0/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_compressor": "1/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_context": "0/1", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crimson_interrupt": "0/5", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crush": "1/1", 2024-03-26T03:50:27.652 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_crypto": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_cyanstore": "0/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_deliberately_leak_memory": "false", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_disable_randomized_ping": "false", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_dpdk": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_eventtrace": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filer": "0/1", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_filestore": "1/3", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_finisher": "1/1", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_fuse": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeat_testing_span": "0", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_heartbeatmap": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_javaclient": "1/5", 2024-03-26T03:50:27.653 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journal": "1/3", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_journaler": "0/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_kstore": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_lockdep": "0/1", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mclock": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_balancer": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_locker": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_log_expire": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_migrator": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mds_quiesce": "3/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_memstore": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgr": "2/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mgrc": "1/5", 2024-03-26T03:50:27.654 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_mon": "20/20", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_monc": "0/10", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_ms": "1/1", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_none": "0/5", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objclass": "0/5", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objectcacher": "0/5", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_objecter": "0/1", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_optracker": "0/5", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_osd": "1/5", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_paxos": "20/20", 2024-03-26T03:50:27.655 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_perfcounter": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_prioritycache": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rados": "0/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd": "0/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_mirror": "0/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_pwl": "0/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rbd_replay": "0/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_refs": "0/0", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_reserver": "1/1", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_access": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_datacache": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_dbstore": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_flight": "1/5", 2024-03-26T03:50:27.656 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_notification": "1/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rgw_sync": "1/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_rocksdb": "4/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_backref": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cache": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_cleaner": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_device": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_epm": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_journal": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_lba": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_odata": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_omap": "0/5", 2024-03-26T03:50:27.657 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_onode": "0/5", 2024-03-26T03:50:27.658 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_t": "0/5", 2024-03-26T03:50:27.658 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_seastore_tm": "0/5", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_striper": "0/1", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_test": "0/5", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_throttle": "1/1", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_timer": "0/1", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_tp": "0/5", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "debug_trace": "1/5", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "device_failure_prediction_mode": "none", 2024-03-26T03:50:27.659 INFO:teuthology.orchestra.run.smithi006.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_graylog": "false", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_journald": "false", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_stderr": "true", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "err_to_syslog": "false", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event_tracing": "false", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_addr": "0.0.0.0", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_http_port": "9926", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_prio_limit": "5", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_sort_metrics": "true", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "exporter_stats_period": "5", 2024-03-26T03:50:27.660 INFO:teuthology.orchestra.run.smithi006.stdout: "fake_statfs_for_testing": "0", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "fatal_signal_handlers": "true", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_purge_ops": "10", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filer_max_truncate_ops": "128", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_apply_finisher_threads": "1", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_blackhole": "false", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_clone_range": "true", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_btrfs_snap": "true", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_caller_concurrency": "10", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_collect_device_partition_information": "true", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_commit_timeout": "600.000000", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_inject_read_err": "false", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_omap_check": "false", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_debug_verify_split": "false", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_dump_file": "", 2024-03-26T03:50:27.661 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fadvise": "true", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fail_eio": "true", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_shards": "16", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fd_cache_size": "128", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap": "false", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fiemap_threshold": "4096", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_inject_stall": "0", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_parallel": "false", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_trailing": "false", 2024-03-26T03:50:27.662 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_journal_writeahead": "false", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_kill_at": "0", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs": "0", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size": "0", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-26T03:50:27.663 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_merge_threshold": "-10", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_odsync_write": "false", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend": "rocksdb", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_backend_path": "", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_thread_timeout": "60", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_op_threads": "2", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_punch_hole": "false", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-26T03:50:27.664 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_queue_max_ops": "50", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_seek_data_hole": "false", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc": "false", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_splice": "false", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_multiple": "2", 2024-03-26T03:50:27.665 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_split_rand_factor": "20", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_update_to": "1000", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_enable": "true", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-26T03:50:27.666 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_xfs_extsize": "false", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "filestore_zfs_snap": "false", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fio_dir": "/tmp/fio", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_allow_other": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_atomic_o_trunc": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_big_writes": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_debug": "false", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_default_permissions": "false", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_disable_pagecache": "false", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_max_write": "0", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_multithreaded": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_require_active_mds": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_set_user_groups": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_move": "true", 2024-03-26T03:50:27.667 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_read": "true", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_splice_write": "true", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "fuse_use_invalidate_cb": "true", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "gss_target_name": "ceph", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_file": "", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_inject_failure": "0", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_interval": "5", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "host": "smithi006", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-26T03:50:27.668 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "inject_early_sigterm": "false", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "jaeger_agent_port": "6799", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "jaeger_tracing_enable": "false", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_aio": "true", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_align_min_size": "65536", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_align": "true", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_block_size": "4096", 2024-03-26T03:50:27.669 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_dio": "true", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_discard": "false", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_force_aio": "false", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_ignore_corruption": "false", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_bytes": "10485760", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_max_write_entries": "100", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_replay_from": "0", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_write_header_frequency": "0", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journal_zero_on_create": "false", 2024-03-26T03:50:27.670 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prefetch_periods": "10", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_prezero_periods": "5", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "journaler_write_head_interval": "15", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "key": "", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "keyfile": "", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_backend": "rocksdb", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_default_stripe_size": "65536", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount": "false", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_bytes": "67108864", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_max_ops": "512", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_nid_prealloc": "1024", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_onode_map_size": "1024", 2024-03-26T03:50:27.671 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_submit_transaction": "false", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "kstore_sync_transaction": "false", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "librados_thread_count": "2", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep": "false", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "lockdep_force_backtrace": "false", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_coarse_timestamps": "true", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_flush_on_exit": "false", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_host": "127.0.0.1", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_graylog_port": "12201", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_new": "1000", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_max_recent": "10000", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stderr_prefix": "", 2024-03-26T03:50:27.672 INFO:teuthology.orchestra.run.smithi006.stdout: "log_stop_at_utilization": "0.970000", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_file": "true", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_graylog": "false", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_journald": "false", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_stderr": "false", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "log_to_syslog": "false", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "max_rotating_auth_attempts": "10", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_action_on_write_error": "1", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_alternate_name_max": "8192", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_asio_thread_count": "2", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_export_pin": "true", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_dirs": "true", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_interval": "5", 2024-03-26T03:50:27.673 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_interval": "10", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max": "-1", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_max_until": "-1", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_merge_size": "50", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_midchunk": "0.300000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_min_start": "0.200000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_minchunk": "0.001000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_mode": "0", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_max": "1.200000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_need_min": "0.800000", 2024-03-26T03:50:27.674 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_overload_epochs": "2", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_bits": "3", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_size": "10000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_target_decay": "10.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_grace": "15.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_interval": "4.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_mid": "0.700000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-03-26T03:50:27.675 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_delay": "0", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_sleep": "200", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_splitauth": "true", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_quiesce_threshold": "524288", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_release_free_interval": "10", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_reservation": "0.050000", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_interval": "1", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cache_trim_threshold": "262144", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_prealloc_inos": "1000", 2024-03-26T03:50:27.676 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_connect_bootstrapping": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_damage_table_max_entries": "10000", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_auth_pins": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_frag": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_scatterstat": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_debug_subtrees": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_decay_halflife": "5.000000", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_default_dir_hash": "2", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_defer_session_stale": "true", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_deny_all_reconnect": "false", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_keys_per_op": "16384", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_commit_size": "10", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_max_entries": "0", 2024-03-26T03:50:27.677 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dir_prefetch": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_on_map": "false", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_early_reply": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enable_op_tracker": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_enforce_unique_name": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random": "true", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-26T03:50:27.678 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_health_summarize_threshold": "10", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_health_dummy": "false", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_migrator_session_race": "false", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-26T03:50:27.679 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_join_fs": "", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_journal_format": "1", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_create_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_export_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_import_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_expire_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_journal_replay_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_link_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_mdstable_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_openc_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_rename_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_shutdown_at": "0", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-26T03:50:27.680 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_event_large_threshold": "524288", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_events_per_segment": "1024", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_events": "-1", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_max_segments": "128", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_pause": "false", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_segment_size": "0", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_threshold": "128", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_log_warn_factor": "2.000000", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_caps_per_client": "1048576", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_flushes": "100000", 2024-03-26T03:50:27.681 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_completed_requests": "100000", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_export_size": "20971520", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_file_recover": "32", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_files": "64", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops": "8192", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_max_snaps_per_dir": "100", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_metrics_update_interval": "2", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_per_client": "100", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_min_caps_working_set": "10000", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_numa_node": "-1", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-26T03:50:27.682 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_complaint_time": "30.000000", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_duration": "600", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_size": "20", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_size": "20", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_op_log_threshold": "5", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_grace": "15", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_ping_interval": "5", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_caps": "30000", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-26T03:50:27.683 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_recall_warning_threshold": "262144", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_interval": "1.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_gid": "0", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_root_ino_uid": "0", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-26T03:50:27.684 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_shutdown_check": "0", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_skip_ino": "0", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_max_uid": "4294967294", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_min_uid": "0", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_snap_rstat": "false", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_standby_replay_damaged": "false", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_symlink_recovery": "true", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_exports": "0", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_thrash_fragments": "0", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_tick_interval": "5.000000", 2024-03-26T03:50:27.685 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_valgrind_exit": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_backtrace": "1", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_verify_scatter": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mds_wipe_sessions": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mempool_debug": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_device_bytes": "1073741824", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_set": "false", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "memstore_page_size": "65536", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_bytes": "134217728", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_messages": "512", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-26T03:50:27.686 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_disabled_modules": "", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_max_pg_num_change": "128", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_bytes": "134217728", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mds_messages": "128", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_bytes": "134217728", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_mon_messages": "128", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_bytes": "536870912", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_osd_messages": "8192", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_pool": "true", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_standby_modules": "true", 2024-03-26T03:50:27.687 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_period": "5", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_stats_threshold": "5", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_tick_period": "2", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_delete": "true", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_allow_pool_size_one": "false", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_auth_validate_all_caps": "true", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_bytes": "104857600", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_directed_command_retry": "2", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-26T03:50:27.688 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_hunt_parallel": "3", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_log_interval": "1.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_interval": "10.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_client_target_rank": "-1", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_file_level": "debug", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_file": "true", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-26T03:50:27.689 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_journald": "true", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_bootstrap": "false", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_start": "false", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_compact_on_trim": "true", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_cpu_threads": "4", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_crush_min_required_version": "hammer", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_daemon_bytes": "419430400", 2024-03-26T03:50:27.690 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_crit": "5", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_avail_warn": "5", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_data_size_warn": "16106127360", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_json": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_dump_transactions": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_extra_checks": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_reef": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_no_require_squid": "false", 2024-03-26T03:50:27.691 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_down_mkfs_grace": "60", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_down_uptime_grace": "60", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_default_strategy": "1", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_election_timeout": "5.000000", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_divisor": "2", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_enable_op_tracker": "true", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_fake_pool_delete": "false", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_force_quorum_join": "false", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_globalid_prealloc": "10000", 2024-03-26T03:50:27.692 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_detail_to_clog": "true", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_log_update_period": "5", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_max_detail": "50", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog": "true", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_interval": "600", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host": "[v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0]", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_host_override": "", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_initial_members": "", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-26T03:50:27.693 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease": "5.000000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_full_interval": "50", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max": "10000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_log_max_summary": "50", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_log_epochs": "500", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_osd": "10000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pg_per_osd": "10000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_pool_pg_num": "65536", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-26T03:50:27.694 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_force_trim_to": "0", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mds_skip_sanity": "false", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_autotune": "true", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_memory_target": "2147483648", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_beacon_grace": "30", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_digest_period": "5", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_inactive_grace": "60", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_min_osdmap_epochs": "50", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_complaint_time": "30", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_duration": "600", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_size": "20", 2024-03-26T03:50:27.695 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_size": "20", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_op_log_threshold": "5", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_in": "false", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size": "500", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_interval": "600", 2024-03-26T03:50:27.696 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_force_trim_to": "0", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_down_reporters": "2", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-26T03:50:27.697 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_report_timeout": "900", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_stuck_threshold": "60", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-26T03:50:27.698 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_probe_timeout": "2.000000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_change": "0.050000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_max_osds": "4", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_interval": "86400", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_max_keys": "100", 2024-03-26T03:50:27.699 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_scrub_timeout": "300", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_session_timeout": "300", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_smart_report_timeout": "5", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stat_smooth_intervals": "6", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_min_size": "2", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_pool_size": "4", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_debug": "false", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_provider_kill_at": "0", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_requester_kill_at": "0", 2024-03-26T03:50:27.700 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_sync_timeout": "60.000000", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_target_pg_per_osd": "100", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_tick_interval": "5", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_interval": "300.000000", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_use_min_delay_socket": "false", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_older_version_delay": "604800", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-26T03:50:27.701 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_misplaced": "false", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_older_version": "true", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "monmap": "", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T03:50:27.702 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "motr_tracing_enabled": "false", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_op_threads": "3", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_cm": "false", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_device_name": "", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_dscp": "96", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_local_gid": "", 2024-03-26T03:50:27.703 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_port_num": "1", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_sl": "3", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_support_srq": "true", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_rdma_type": "ib", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_async_reap_threshold": "5", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_before_connect": "false", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv4": "true", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_ipv6": "false", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr1": "false", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_msgr2": "true", 2024-03-26T03:50:27.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_max": "7568", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_port_min": "6800", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_count": "3", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_bind_retry_delay": "5", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_client": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mds": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mgr": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_mon": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_blackhole_osd": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_client_mode": "crc secure", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_mode": "crc secure", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_cluster_type": "", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_compress_secure": "false", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_idle_timeout": "900", 2024-03-26T03:50:27.705 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_connection_ready_timeout": "10", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_data": "true", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_crc_header": "true", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bad_msg": "false", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_bug": "true", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_old_message": "true", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_skipped_message": "false", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_coremask": "0xF", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_enable_tso": "true", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-26T03:50:27.706 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hugepages": "", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_lro": "true", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_memory_channel": "4", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_pmd": "", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_port_id": "0", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_dump_on_send": "false", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_initial_backoff": "0.200000", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_max": "1.000000", 2024-03-26T03:50:27.707 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-26T03:50:27.708 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_delay_type": "", 2024-03-26T03:50:27.708 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-26T03:50:27.708 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_network_congestion": "0", 2024-03-26T03:50:27.708 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_inject_socket_failures": "0", 2024-03-26T03:50:27.709 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_learn_addr_from_peer": "true", 2024-03-26T03:50:27.709 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_accept_failures": "4", 2024-03-26T03:50:27.709 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_max_backoff": "15.000000", 2024-03-26T03:50:27.709 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_client_mode": "secure crc", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_mon_service_mode": "secure crc", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_min_size": "1024", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compress_mode": "none", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_pq_min_cost": "65536", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_public_type": "", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_service_mode": "crc secure", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_listen_backlog": "512", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_nodelay": "true", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_tcp_rcvbuf": "0", 2024-03-26T03:50:27.710 INFO:teuthology.orchestra.run.smithi006.stdout: "ms_type": "async+posix", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "no_config_file": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_completion_locks_per_session": "32", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inflight_ops": "1024", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_tick_interval": "5.000000", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objecter_timeout": "10.000000", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objectstore_blackhole": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "openssl_engine_opts": "", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_delay_time": "5.000000", 2024-03-26T03:50:27.711 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_hist_halflife": "1000", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_low_ops": "2", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_max_ops": "4", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_agent_slop": "0.020000", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_asio_thread_count": "2", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_async_recovery_min_cost": "100", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_max": "512", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backfill_scan_min": "64", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_degraded": "false", 2024-03-26T03:50:27.712 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_peering": "false", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_backoff_on_unfound": "true", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_beacon_report_interval": "300", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_duration": "30", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_max_block_size": "67108864", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_blkin_trace_all": "false", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_for_log_corruption": "false", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-26T03:50:27.713 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_class_update_on_start": "true", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_cap": "256", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_message_size_cap": "524288000", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_op_priority": "63", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_client_watch_timeout": "30", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_command_max_records": "256", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_compact_on_start": "false", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_on_start": "true", 2024-03-26T03:50:27.714 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_crush_update_weight_set": "true", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_feed_pullee": "-1", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_misdirected_ops": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_acting_change": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_no_purge_strays": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_op_order": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-26T03:50:27.715 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_shutdown": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_trim_objects": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_snaps": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_keys": "1024", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-26T03:50:27.716 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_stride": "524288", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_default_notify_timeout": "30", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep": "0.000000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_discard_disconnected_ops": "true", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_enable_op_tracker": "true", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-26T03:50:27.717 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_info": "true", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown": "true", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_function_tracing": "false", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_grace": "20", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_interval": "6", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_peers": "10", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_min_size": "2000", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_stale": "600", 2024-03-26T03:50:27.718 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_max_size": "100000", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_min_size": "1000", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_journal_size": "5120", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_kill_backfill_at": "0", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_cache_size": "50", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_dedup": "true", 2024-03-26T03:50:27.719 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max": "40", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_map_share_max_epochs": "40", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_name_len": "100", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_attr_size": "0", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_backfills": "1", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_count": "5", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_markdown_period": "600", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_name_len": "2048", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_namespace_len": "256", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_object_size": "134217728", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_blocked_by": "16", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_log_entries": "10000", 2024-03-26T03:50:27.720 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_pgls": "1024", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_cost": "8388608", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_push_objects": "10", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_scrubs": "3", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_trimming_pgs": "2", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_op_reply_len": "64", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_max_write_size": "90", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-26T03:50:27.721 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_profile": "balanced", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-26T03:50:27.722 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_base": "805306368", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_min": "134217728", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target": "4294967296", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_autotune": "false", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_pg_log_entries": "250", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_min_recovery_priority": "0", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_interval": "5", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_cache_shards": "32", 2024-03-26T03:50:27.723 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_num_op_tracker_shard": "32", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_auto_affinity": "true", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_node": "-1", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_numa_prefer_iface": "true", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objecter_finishers": "1", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore": "bluestore", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_fuse": "false", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_objectstore_tracing": "false", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_complaint_time": "30.000000", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_duration": "600", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_size": "20", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_size": "20", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_log_threshold": "5", 2024-03-26T03:50:27.724 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards": "0", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_hdd": "5", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_shards_ssd": "8", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_pq_min_cost": "65536", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_queue_cut_off": "high", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_op_thread_timeout": "15", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_open_classes_on_start": "true", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_os_flags": "0", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_peering_op_priority": "255", 2024-03-26T03:50:27.725 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_cost": "1048576", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_delete_priority": "5", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_max": "10000", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_log_trim_min": "100", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-26T03:50:27.726 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_crimson": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_flags": "0", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_min_size": "0", 2024-03-26T03:50:27.727 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-26T03:50:27.728 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pg_num": "32", 2024-03-26T03:50:27.728 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_pgp_num": "0", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_read_ratio": "70", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_size": "3", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_default_type": "replicated", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_push_per_object_cost": "1000", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap": "true", 2024-03-26T03:50:27.729 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_cost": "20971520", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active": "0", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_max_single_start": "1", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_priority": "3", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_priority": "5", 2024-03-26T03:50:27.730 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep": "0.000000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_repair_during_recovery": "false", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_requested_scrub_priority": "120", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair": "false", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_hour": "0", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_begin_week_day": "0", 2024-03-26T03:50:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_max": "25", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_chunk_min": "5", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_cost": "52428800", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_during_recovery": "false", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_hour": "0", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_end_week_day": "0", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_event_cost": "4096", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_invalid_stats": "true", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_max_preemptions": "5", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-26T03:50:27.732 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_priority": "5", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_scrub_sleep": "0.000000", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_skip_data_digest": "false", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_smart_report_timeout": "5", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_cost": "1048576", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_priority": "5", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-26T03:50:27.733 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_target_transaction_size": "30", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-26T03:50:27.734 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_tracing": "false", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_use_stale_snap": "false", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "osdc_blkin_trace_all": "false", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_kill_at": "0", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_max_join_drift": "10", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min": "500", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_min_wait": "0.050000", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_propose_interval": "1.000000", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max": "500", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_service_trim_min": "10", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_stash_full_interval": "25", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_max": "500", 2024-03-26T03:50:27.735 INFO:teuthology.orchestra.run.smithi006.stdout: "paxos_trim_min": "250", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "perf": "true", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "pid_file": "", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "-", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrv": "", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "public_bind_addr": "-", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network": "", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "public_network_interface": "", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_enabled": "false", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "qat_compressor_session_max_number": "256", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_mon_op_timeout": "0", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_osd_op_timeout": "0", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "rados_tracing": "false", 2024-03-26T03:50:27.736 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_atime_update_interval": "60", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_parent_reads": "false", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_balance_snap_reads": "false", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blkin_trace_all": "false", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache": "true", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_policy": "writearound", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_size": "33554432", 2024-03-26T03:50:27.737 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_clone_copy_on_read": "false", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_compression_hint": "none", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_concurrent_management_ops": "10", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_clone_format": "auto", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_data_pool": "", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_features": "61", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_format": "2", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_map_options": "", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_order": "22", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_pool": "rbd", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-26T03:50:27.738 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_count": "0", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_default_stripe_unit": "0", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_enable_alloc_hint": "true", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler": "simple", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-26T03:50:27.739 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_order": "24", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_pool": "", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_journal_splay_width": "4", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_parent_reads": "false", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_localize_snap_reads": "false", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-26T03:50:27.740 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-26T03:50:27.741 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_mtime_update_interval": "60", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_non_blocking_aio": "true", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_thread_timeout": "60", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_op_threads": "1", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_parent_cache_enabled": "false", 2024-03-26T03:50:27.742 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_plugins": "", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_bps_limit": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_exclude_ops": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_iops_limit": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-26T03:50:27.743 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_read_from_replica_policy": "default", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-26T03:50:27.744 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_skip_partial_discard": "true", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_tracing": "false", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_names": "true", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd_validate_pool": "true", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_base_url": "", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "restapi_log_level": "", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_acl_grants_max_num": "100", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_admin_entry": "admin", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_backend_store": "rados", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_barbican_url": "", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_beast_enable_async": "true", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_counters_cache": "false", 2024-03-26T03:50:27.745 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_enabled": "true", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_expiry_interval": "900", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cache_lru_size": "10000", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_config_store": "rados", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_content_length_compat": "false", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress": "true", 2024-03-26T03:50:27.746 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cors_rules_max_num": "100", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_cross_domain_policy": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_addr": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_password": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_kmip_username": "", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_require_ssl": "true", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-26T03:50:27.747 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-26T03:50:27.748 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_addr": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_auth": "token", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_namespace": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_prefix": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_token_file": "", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_buffersize": "524288", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_low_speed_time": "300", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-26T03:50:27.749 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_d4n_port": "6379", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_changes_size": "1000", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_num_shards": "128", 2024-03-26T03:50:27.750 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_log_window": "30", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-26T03:50:27.751 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-26T03:50:27.752 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_name": "", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dns_s3website_name": "", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_dynamic_resharding": "true", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_gc_threads": "true", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_lc_threads": "true", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_ops_log": "false", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_quota_threads": "true", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_static_website": "false", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enable_usage_log": "false", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_enforce_swift_acls": "true", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_exit_timeout_secs": "120", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_expose_bucket": "false", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_extended_http_attrs": "", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_filter": "none", 2024-03-26T03:50:27.753 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_frontends": "beast port=7480", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred": "50", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_objs": "32", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_gc_processor_period": "3600", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-26T03:50:27.754 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_init_timeout": "300", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_connection_idle": "300", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_message_timeout": "5000", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-26T03:50:27.755 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_domain": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_password_path": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_project": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_tenant": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_token_path": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_admin_user": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_api_version": "2", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_domain": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_password": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_project": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-26T03:50:27.756 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_barbican_user": "", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_url": "", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_debug_interval": "-1", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_lock_max_time": "90", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_objs": "32", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_rules": "1000", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_worker": "3", 2024-03-26T03:50:27.757 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lc_thread_delay": "0", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_dnattr": "uid", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_searchfilter": "", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_http_headers": "", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-26T03:50:27.758 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_log_object_name_utc": "false", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_name_len": "0", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attr_size": "0", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_chunk_size": "4194304", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_listing_results": "1000", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_notify_retries": "10", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_param_size": "1048576", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_put_size": "5368709120", 2024-03-26T03:50:27.759 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_max_slo_entries": "1000", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_log_max_shards": "64", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_mp_lock_max_time": "600", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-26T03:50:27.760 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_max_gc": "300", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_async_rados_threads": "32", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_num_control_oids": "8", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_numa_node": "-1", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_chunk_size": "100", 2024-03-26T03:50:27.761 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_gc_interval": "600", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_thread_timeout": "600", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_op_tracing": "false", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_token": "", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_url": "", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_opa_verify_ssl": "true", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_rados": "false", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_ops_log_socket_path": "", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-26T03:50:27.762 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval": "2.000000", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_lanes": "3", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_cache_partitions": "3", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_continue": "true", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-26T03:50:27.763 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rados_tracing": "false", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm": "", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_id": "", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region": "", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_request_uri": "", 2024-03-26T03:50:27.764 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_batch_size": "64", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_max_aio": "128", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_num_logs": "16", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_reshard_thread_interval": "600", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_resolve_cname": "false", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_run_sync_thread": "true", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-26T03:50:27.765 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_scheduler_type": "throttler", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_script_uri": "", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_service_provider_name": "", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_id": "", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_client_secret": "", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_entry": "sts", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_key": "sts", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_min_session_duration": "900", 2024-03-26T03:50:27.766 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sts_token_introspection_url": "", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_account_in_url": "false", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_entry": "auth", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_auth_url": "", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_custom_header": "", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_need_stats": "true", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_tenant_name": "", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_token_expiration": "86400", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url": "", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_url_prefix": "swift", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-26T03:50:27.767 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_lease_period": "120", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_thread_pool_size": "512", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-26T03:50:27.768 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_comment": "", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_createby": "", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_encoding": "", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_flag": "false", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_origin": "", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_torrent_tracker": "", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_trust_forwarded_https": "false", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_shards": "32", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_usage_max_user_shards": "1", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_use_opa_authz": "false", 2024-03-26T03:50:27.769 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_counters_cache": "false", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_max_buckets": "1000", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_policies_max_num": "100", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_user_unique_email": "true", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_verify_ssl": "true", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone": "", 2024-03-26T03:50:27.770 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_id": "", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup": "", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_id": "", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgwlc_auto_session_clear": "true", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_block_size": "4096", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_size": "536870912", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-26T03:50:27.771 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_index_type": "binary_search", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_partition_filters": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_perf": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-26T03:50:27.772 INFO:teuthology.orchestra.run.smithi006.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "run_dir": "/var/run/ceph", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_block_create": "true", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_cache_lru_size": "67108864", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_max_object_size": "16777216", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_device_size": "53687091200", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_capacity": "16", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_main_device_type": "SSD", 2024-03-26T03:50:27.773 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "seastore_segment_size": "67108864", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "set_keepcaps": "false", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "setgroup": "ceph", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser": "ceph", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "setuser_match_path": "", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "thp": "false", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_default_timeout": "60", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-26T03:50:27.774 INFO:teuthology.orchestra.run.smithi006.stdout: "throttler_perf_counter": "true" 2024-03-26T03:50:27.775 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:50:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:27 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:50:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:27 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:50:27.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:27 smithi086 ceph-mon[32965]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T03:50:28.602 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:28 smithi086 ceph-mon[32965]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:50:28.602 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:28 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:28.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-26T03:50:28.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:28 smithi006 ceph-mon[37501]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:50:28.684 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:28 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:28.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:28 smithi006 ceph-mon[29446]: pgmap v49: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 239 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:50:28.684 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:28 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:29.630 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:30.067 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[37501]: pgmap v50: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:29 smithi006 ceph-mon[29446]: pgmap v50: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:50:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:29 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T03:50:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:29 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T03:50:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:29 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:29 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:29 smithi086 ceph-mon[32965]: pgmap v50: 132 pgs: 1 peering, 131 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:50:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:50:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:50:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:31 smithi086 ceph-mon[32965]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:50:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[37501]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T03:50:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:31 smithi006 ceph-mon[29446]: Reconfiguring daemon alertmanager.a on smithi006 2024-03-26T03:50:32.816 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T03:50:32.816 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T03:50:32.816 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[37501]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-26T03:50:32.816 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T03:50:32.816 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T03:50:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T03:50:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T03:50:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[29446]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-26T03:50:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T03:50:32.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:32 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T03:50:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:32 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T03:50:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:32 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T03:50:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:32 smithi086 ceph-mon[32965]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 28 B/s, 1 objects/s recovering 2024-03-26T03:50:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:32 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T03:50:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:32 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T03:50:33.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T03:50:33.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/461238021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T03:50:33.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T03:50:33.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/461238021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:50:33.826 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-03-26T03:50:33.826 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-03-26T03:50:33.826 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-03-26T03:50:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:50:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:50:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T03:50:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T03:50:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/461238021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T03:50:34.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:50:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:50:34.465 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-26T03:50:34.484 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:50:34.516 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:50:34.516 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-26T03:50:34.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-26T03:50:34.517 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-26T03:50:34.517 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:50:34.517 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:50:34.518 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:50:34.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd out 6 2024-03-26T03:50:34.518 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-26T03:50:34.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:34.519 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-26T03:50:34.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:50:34.519 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-26T03:50:34.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:50:34.520 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-26T03:50:34.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:50:34.530 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:34.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:34.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:34.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:34.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:34.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:34.776 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T03:50:34.776 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T03:50:34.776 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[37501]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:50:34.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T03:50:34.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T03:50:34.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:34 smithi006 ceph-mon[29446]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:50:34.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:34.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:34 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T03:50:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:34 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T03:50:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:34 smithi086 ceph-mon[32965]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 591 B/s rd, 0 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:50:34.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:34.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:35.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:35.062 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:35.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:35.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:35.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:35.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:35.307 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:35.310 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:35.311 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:35.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:35.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:35.365 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:35.369 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:35.389 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:35 smithi006 systemd[1]: Stopping Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:35.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.466 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:35.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:35.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:35.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:35.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:50:35.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:50:35.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:50:35.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:35.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:35.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T03:50:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T03:50:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T03:50:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T03:50:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:50:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:35 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:50:35.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:35.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:35.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:35.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:35.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:36.068 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:35 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[71460]: ts=2024-03-26T03:50:35.960Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:50:36.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:36.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.110 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:36.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:36.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:36.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:36.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:36.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:36.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:36.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:36.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:36.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:36.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-26T03:50:36.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 3.12 deep-scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 3.12 deep-scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 4.10 scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 5.1a deep-scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[37501]: 5.1a deep-scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 3.12 deep-scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 3.12 deep-scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 4.10 scrub ok 2024-03-26T03:50:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:50:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:50:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:50:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 5.1a deep-scrub starts 2024-03-26T03:50:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:36 smithi006 ceph-mon[29446]: 5.1a deep-scrub ok 2024-03-26T03:50:36.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:36.753 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:36.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 21 B/s, 1 objects/s recovering 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 3.12 deep-scrub starts 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 3.12 deep-scrub ok 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 4.10 scrub ok 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 5.1a deep-scrub starts 2024-03-26T03:50:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:36 smithi086 ceph-mon[32965]: 5.1a deep-scrub ok 2024-03-26T03:50:36.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:36.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:36.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:36.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:36.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:37.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:37.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:37.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:37.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:37.390 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:37 smithi006 podman[74311]: 2024-03-26 03:50:37.210869847 +0000 UTC m=+1.560467399 container died 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:37.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:37.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:37.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:37.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:37.649 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:37.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:37.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:37 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:37 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:37 smithi086 ceph-mon[32965]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:37 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:37 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T03:50:37.838 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[37501]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:50:37.839 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:37 smithi006 podman[74311]: 2024-03-26 03:50:37.637416507 +0000 UTC m=+1.987014059 container cleanup 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:37.839 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:37 smithi006 bash[74311]: ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[29446]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T03:50:37.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:50:37.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:37 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:50:37.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:37.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:37.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:38.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:38.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:38.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:38.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:38.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:38.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:38.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:38.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:38.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:38.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:38.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:38.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:38.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:38.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T03:50:38.551 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T03:50:38.552 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[29446]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:50:38.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:38.668 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:38.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:38.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:38.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:38 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T03:50:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:38 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T03:50:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:38 smithi086 ceph-mon[32965]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:50:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T03:50:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T03:50:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[37501]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T03:50:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:38.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:38.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:38.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:39.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:39.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:39.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:39.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:39.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:39.199 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:38 smithi006 podman[74346]: 2024-03-26 03:50:38.944986444 +0000 UTC m=+2.963889018 container remove 959ac0074198e4ac9a46e5c3e073f3deead7c0228e164716b6fd2169de8d3445 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:39.199 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:39 smithi006 podman[74346]: 2024-03-26 03:50:39.037479354 +0000 UTC m=+3.056381926 volume remove a744f8f103b7829be823a909a79d7855aa844e91855abc522c2029f02aa6365d 2024-03-26T03:50:39.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:39.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:39.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:39.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:39.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:39.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:39.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:39.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:39.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:39.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:39.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:39.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:39.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:39.943 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:39.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:40.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.069 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:40.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:40.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:40.317 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:40.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:40.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:40.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:40.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:40.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:40.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:40.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:40.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:40.825 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T03:50:40.825 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T03:50:40.825 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[37501]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:40.825 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:40 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T03:50:40.825 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:40 smithi006 systemd[1]: Stopped Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:40.825 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:40 smithi006 systemd[1]: Starting Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:40.826 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T03:50:40.826 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T03:50:40.826 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:40 smithi006 ceph-mon[29446]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:40.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:40.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:40 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T03:50:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:40 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T03:50:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:40 smithi086 ceph-mon[32965]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:40.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:40.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:40.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:40.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:41.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:41.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:41.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:41.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.205 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:41.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:41.329 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:41.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:41.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:41.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:41.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:41.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:41.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:41.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:41 smithi086 ceph-mon[32965]: 5.19 deep-scrub starts 2024-03-26T03:50:41.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:41 smithi086 ceph-mon[32965]: 5.19 deep-scrub ok 2024-03-26T03:50:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:41 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T03:50:41.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:41 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T03:50:41.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[37501]: 5.19 deep-scrub starts 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[37501]: 5.19 deep-scrub ok 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[29446]: 5.19 deep-scrub starts 2024-03-26T03:50:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[29446]: 5.19 deep-scrub ok 2024-03-26T03:50:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T03:50:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:41 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T03:50:41.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:41.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:41.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:42.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:42.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:42.226 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:42.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:42.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:42.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:42.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:42.481 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:42.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:42.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:42.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:42.726 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:42 smithi006 ceph-mon[37501]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:42.726 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:42 smithi006 ceph-mon[29446]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:42.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:42 smithi086 ceph-mon[32965]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:42.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:42.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:42.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:42.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:42.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:43.047 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:50:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:50:43.048 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T03:50:43.048 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:50:43.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:43.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:43.247 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:43.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:43.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.499 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' created 2024-03-26T03:50:43.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:43.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.625 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:43.698 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:50:43.718 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:50:43.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:43.737 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:50:43.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:43.757 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:50:43.778 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:50:43.798 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:50:43.818 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:50:43.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3985812705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.819 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:43 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3985812705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:43 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3985812705' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:43 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-26T03:50:43.838 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:50:43.839 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:50:43.839 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:50:43.840 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:50:43.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:43.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:43.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:43.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:43.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:44.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:44.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:44.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:44.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:44.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:44.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:44.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:44.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:44.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:44.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:44.388 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:50:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:50:44.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T03:50:44.304+0000 7fd465f8b700 -1 mon.a@0(leader).osd e95 definitely_dead 0 2024-03-26T03:50:44.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:44.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:44.582 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.6. 2024-03-26T03:50:44.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:44.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:44.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:44.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:50:44.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:50:44.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/805418739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:44.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3940837213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[37501]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/805418739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3940837213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:44 smithi006 ceph-mon[29446]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-26T03:50:44.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:44.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:44.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/805418739' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3940837213' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-26T03:50:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:44 smithi086 ceph-mon[32965]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-26T03:50:44.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:44.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:44.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:44.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:45.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:45.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:45.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:45.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:45.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.271 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:45.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:45.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:45.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:45.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.522 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:45.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:45.578 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:50:45.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:45.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:45.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.774 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:45.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: osdmap e96: 8 total, 8 up, 7 in 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:50:45.855 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[37501]: osdmap e97: 8 total, 8 up, 7 in 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: osdmap e96: 8 total, 8 up, 7 in 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:50:45.856 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:45 smithi006 ceph-mon[29446]: osdmap e97: 8 total, 8 up, 7 in 2024-03-26T03:50:45.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:45.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-26T03:50:45.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:45.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:46.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:46.031 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: osdmap e96: 8 total, 8 up, 7 in 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1246250528' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:50:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:45 smithi086 ceph-mon[32965]: osdmap e97: 8 total, 8 up, 7 in 2024-03-26T03:50:46.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:46.139 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 podman[75605]: 2024-03-26 03:50:45.906445148 +0000 UTC m=+0.017870892 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T03:50:46.139 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 podman[75605]: 2024-03-26 03:50:46.022891431 +0000 UTC m=+0.134317148 volume create f0408a240d868d02129df5476b426e30c7536f19bd390bc63fb676cf49a35c73 2024-03-26T03:50:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:46.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:46.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:46.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:46.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:46.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:46.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:46.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:46.551 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 podman[75605]: 2024-03-26 03:50:46.140292323 +0000 UTC m=+0.251718048 container create eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:46.552 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 podman[75605]: 2024-03-26 03:50:46.552051067 +0000 UTC m=+0.663476789 container init eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:46.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.668 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:46.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:46.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:46.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:46.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:50:46.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T03:50:46.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T03:50:46.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[37501]: pgmap v62: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:46.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:50:46.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[37501]: osdmap e98: 8 total, 8 up, 7 in 2024-03-26T03:50:46.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T03:50:46.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T03:50:46.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[29446]: pgmap v62: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:46.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:50:46.833 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-mon[29446]: osdmap e98: 8 total, 8 up, 7 in 2024-03-26T03:50:46.833 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 podman[75605]: 2024-03-26 03:50:46.554981709 +0000 UTC m=+0.666407430 container start eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:46.833 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.567Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T03:50:46.833 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.567Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T03:50:46.833 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.568Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.6 port=9094 2024-03-26T03:50:46.833 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.569Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.595Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.596Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.599Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:46.599Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 bash[75605]: eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 2024-03-26T03:50:46.834 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:46 smithi006 systemd[1]: Started Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:46.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:46.922 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:46.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:47.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:47.048 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:46 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T03:50:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:46 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T03:50:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:46 smithi086 ceph-mon[32965]: pgmap v62: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:46 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:50:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:46 smithi086 ceph-mon[32965]: osdmap e98: 8 total, 8 up, 7 in 2024-03-26T03:50:47.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:47.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:47.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:47.299 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:47.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:47.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:47.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:50:47.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:47.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:47.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:47.552 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:47.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:47.678 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:47.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:47.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:47.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:47.825 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:47 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:47.825 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:47 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:47.825 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:47 smithi086 ceph-mon[32965]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:50:47.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:47.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:47.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:47.931 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:47.931 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:48.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:48.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[37501]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:48.059 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:47 smithi006 ceph-mon[29446]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T03:50:48.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:48.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:48.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:48.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:48.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:48.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:48.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:48.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:48.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:48.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:48.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:48.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:48.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:48.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:48.723 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:48.570Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000507958s 2024-03-26T03:50:48.723 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[29446]: Reconfiguring daemon prometheus.a on smithi086 2024-03-26T03:50:48.723 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[29446]: pgmap v65: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:48.723 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[29446]: osdmap e99: 8 total, 8 up, 7 in 2024-03-26T03:50:48.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:48.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:48.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:48.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:48.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:48 smithi086 ceph-mon[32965]: Reconfiguring daemon prometheus.a on smithi086 2024-03-26T03:50:48.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:48 smithi086 ceph-mon[32965]: pgmap v65: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:48.914 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:48 smithi086 ceph-mon[32965]: osdmap e99: 8 total, 8 up, 7 in 2024-03-26T03:50:48.937 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:48.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:48.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:49.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:49.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:50:49.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[37501]: Reconfiguring daemon prometheus.a on smithi086 2024-03-26T03:50:49.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[37501]: pgmap v65: 148 pgs: 16 creating+peering, 132 active+clean; 582 KiB data, 207 MiB used, 626 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:50:49.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:48 smithi006 ceph-mon[37501]: osdmap e99: 8 total, 8 up, 7 in 2024-03-26T03:50:49.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:49.190 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:49.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:49.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:49.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:49.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:49.442 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:49.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:49.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:49.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:49.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:49.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:49.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:49.819 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:49.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:49.944 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:49.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:50.030 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:49 smithi006 ceph-mon[37501]: osdmap e100: 8 total, 8 up, 7 in 2024-03-26T03:50:50.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:49 smithi006 ceph-mon[29446]: osdmap e100: 8 total, 8 up, 7 in 2024-03-26T03:50:50.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:50.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:49 smithi086 ceph-mon[32965]: osdmap e100: 8 total, 8 up, 7 in 2024-03-26T03:50:50.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:50.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:50.288 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:50:50.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:50.307 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:50:50.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:50.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:50.327 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:50:50.346 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:50:50.367 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:50:50.387 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:50:50.408 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:50:50.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:50:50.535 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:50:50.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:50.560 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 systemd[1]: Stopping Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:50.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:50.684 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:50.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:50.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-mon[32965]: pgmap v68: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 144 KiB/s, 16 objects/s recovering 2024-03-26T03:50:50.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1755198031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T03:50:50.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.559Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T03:50:50.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.560Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T03:50:50.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.560Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T03:50:50.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[62125]: ts=2024-03-26T03:50:50.560Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T03:50:50.839 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 podman[66308]: 2024-03-26 03:50:50.569640649 +0000 UTC m=+0.113744693 container died a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:50.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:50.930 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:50.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:51.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:51.042 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:50 smithi006 ceph-mon[37501]: pgmap v68: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 144 KiB/s, 16 objects/s recovering 2024-03-26T03:50:51.042 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1755198031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:51.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:50 smithi006 ceph-mon[29446]: pgmap v68: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 511 B/s rd, 0 op/s; 144 KiB/s, 16 objects/s recovering 2024-03-26T03:50:51.042 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1755198031' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:50:51.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:51.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:51.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.189 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:51.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:51.210 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 podman[66308]: 2024-03-26 03:50:50.942520112 +0000 UTC m=+0.486624150 container cleanup a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:51.210 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:50 smithi086 bash[66308]: ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T03:50:51.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:51.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:51.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:51.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:51.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:51.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:51.584 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 podman[66322]: 2024-03-26 03:50:51.210540317 +0000 UTC m=+0.643389852 container remove a74184cde0c3060e62a010f0d1d81fb67ffcd03fd14d82be412108cee44a9105 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:51.584 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T03:50:51.584 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 systemd[1]: Stopped Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:51.584 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.086s CPU time. 2024-03-26T03:50:51.584 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 systemd[1]: Starting Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:50:51.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:51.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:51.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:51.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:51.837 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 podman[66501]: 2024-03-26 03:50:51.487725522 +0000 UTC m=+0.019338177 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T03:50:51.837 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 podman[66501]: 2024-03-26 03:50:51.605650669 +0000 UTC m=+0.137263362 container create 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:51.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 podman[66501]: 2024-03-26 03:50:51.830673014 +0000 UTC m=+0.362285675 container init 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:51.838 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 podman[66501]: 2024-03-26 03:50:51.833144694 +0000 UTC m=+0.364757346 container start 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T03:50:51.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:51.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:51.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:52.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:52.070 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:52.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:52.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:52.268 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPING 2024-03-26T03:50:52.268 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPED 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTING 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Serving on http://:::9283 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTED 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPING 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPED 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTING 2024-03-26T03:50:52.269 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Serving on http://:::9283 2024-03-26T03:50:52.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTED 2024-03-26T03:50:52.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPING 2024-03-26T03:50:52.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T03:50:52.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STOPPED 2024-03-26T03:50:52.270 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTING 2024-03-26T03:50:52.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:52.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:52.337 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.856Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.856Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.856Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi086 (none))" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.856Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.856Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.857Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.858Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.859Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.859Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.861Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.861Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.749µs 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.861Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.862Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T03:50:52.338 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.862Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.862Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=19.645µs wal_replay_duration=572.503µs wbl_replay_duration=83ns total_replay_duration=610.174µs 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.863Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.863Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.863Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.883Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.672599ms db_storage=731ns remote_storage=1.512µs web_handler=321ns query_engine=495ns scrape=405.352µs scrape_sd=93.945µs notify=9.801µs notify_sd=14.538µs rules=19.874583ms tracing=6.369µs 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.883Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T03:50:51.883Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 bash[66501]: 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 2024-03-26T03:50:52.339 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 03:50:51 smithi086 systemd[1]: Started Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:50:52.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.450 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:52.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:52.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:52.564 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Serving on http://:::9283 2024-03-26T03:50:52.564 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:03:50:52] ENGINE Bus STARTED 2024-03-26T03:50:52.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:52.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:52.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:52.801 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:52.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:52.830 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:52.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 364 B/s rd, 0 op/s; 103 KiB/s, 13 objects/s recovering 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 364 B/s rd, 0 op/s; 103 KiB/s, 13 objects/s recovering 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:52.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:52.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:52.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:52.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:52.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:50:52.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:50:52.917 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-26T03:50:52.917 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:50:52.917 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:50:52.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-26T03:50:52.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:52.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:52.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:52.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: pgmap v69: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 364 B/s rd, 0 op/s; 103 KiB/s, 13 objects/s recovering 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:52.992 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:52.993 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:50:53.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:53.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:53.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:53.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:53.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:53.350 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:50:53.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:53.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:53.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:53.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:53.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:53.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:53.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:53.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1698095182' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:53.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1698095182' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:53.749 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:53.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:53.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:53.755 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-26T03:50:53.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:53.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:53.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:53.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:53.996 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:53.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:54.074 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi006.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi086.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi086.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1698095182' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:54.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:50:54.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:54.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:54.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.247 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:54.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:54.338 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:50:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:50:54.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:54.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:54.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:54.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.499 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:54.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:54.512 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:50:54.512 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:50:54.512 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:50:54.512 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:50:54.513 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:50:54.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:50:54.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:54.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:54.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:54.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:54.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:54.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:54.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:54.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:54.970 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:50:54.970 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:50:54.970 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:50:54.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:54 smithi086 ceph-mon[32965]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 310 B/s rd, 0 op/s; 88 KiB/s, 11 objects/s recovering 2024-03-26T03:50:54.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:54 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:50:54.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:54 smithi086 ceph-mon[32965]: osdmap e101: 8 total, 8 up, 7 in 2024-03-26T03:50:54.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:55.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:55.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.127 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:55.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[37501]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 310 B/s rd, 0 op/s; 88 KiB/s, 11 objects/s recovering 2024-03-26T03:50:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:50:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[37501]: osdmap e101: 8 total, 8 up, 7 in 2024-03-26T03:50:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[29446]: pgmap v70: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 310 B/s rd, 0 op/s; 88 KiB/s, 11 objects/s recovering 2024-03-26T03:50:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:50:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:54 smithi006 ceph-mon[29446]: osdmap e101: 8 total, 8 up, 7 in 2024-03-26T03:50:55.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:55.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.377 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:55.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:55.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:55.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:55.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:50:55.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:55.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:55.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:55.754 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:55.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:55.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:55.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:55.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:55.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:56.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:50:56.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:56.125 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:56.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:56.177 DEBUG:teuthology.orchestra.run.smithi006:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-26T03:50:56.179 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:50:56.179 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:50:56.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:56.258 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:hints = 1 2024-03-26T03:50:56.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:50:56.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object prefix: benchmark_data_smithi006_77260 2024-03-26T03:50:56.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:50:56.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:50:56.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:56.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:56.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:56.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:56.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:56.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:56.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:56.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:56.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:56.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:56.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:56.801 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:50:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[37501]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 959 B/s rd, 0 op/s; 77 KiB/s, 9 objects/s recovering 2024-03-26T03:50:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:50:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:50:56.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T03:50:56.572Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003294415s 2024-03-26T03:50:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[29446]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 959 B/s rd, 0 op/s; 77 KiB/s, 9 objects/s recovering 2024-03-26T03:50:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:50:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:56 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:50:56.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:56.941 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:56.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:57.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:57.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:57.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.072 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:56 smithi086 ceph-mon[32965]: pgmap v72: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 959 B/s rd, 0 op/s; 77 KiB/s, 9 objects/s recovering 2024-03-26T03:50:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:56 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:50:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:56 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:50:57.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:57.214 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:57.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:57.258 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 1 16 2803 2787 174.177 174.188 0.00754106 0.005719 2024-03-26T03:50:57.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:57.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:57.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:57.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:57.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:57.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:57.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:57.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:50:57.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:50:57.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:50:57.881 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:57.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:50:57.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:57.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:50:57.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[37501]: osdmap e102: 8 total, 8 up, 7 in 2024-03-26T03:50:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:50:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:57 smithi006 ceph-mon[29446]: osdmap e102: 8 total, 8 up, 7 in 2024-03-26T03:50:57.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:58.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:58.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:58.032 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:57 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:50:58.032 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1229039175' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:50:58.032 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:57 smithi086 ceph-mon[32965]: osdmap e102: 8 total, 8 up, 7 in 2024-03-26T03:50:58.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:58.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:58.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:58.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:58.258 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 2 15 5652 5637 176.142 178.125 0.00827191 0.00566699 2024-03-26T03:50:58.277 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:50:58.290 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:58.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:58.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:58.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:58.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:58.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:50:58.693 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:50:58.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:58.836 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:58.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:58.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:50:58.976 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:58.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:58.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[37501]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[29446]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:50:58 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:58 smithi086 ceph-mon[32965]: pgmap v74: 148 pgs: 148 active+clean; 582 KiB data, 208 MiB used, 626 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 13 B/s, 0 objects/s recovering 2024-03-26T03:50:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:58 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:50:58 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:50:59.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:59.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:50:59.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:50:59.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 3 16 8069 8053 167.748 151 0.00504598 0.00595144 2024-03-26T03:50:59.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:59.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:59.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:59.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:59.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:50:59.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:59.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:50:59.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:50:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:50:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:50:59.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:50:59.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:50:59.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:50:59.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:50:59.923 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:50:59.969 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:51:00.012 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:51:00.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:00.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:00.064 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:51:00.106 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:51:00.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:00.149 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:51:00.182 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:00.195 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:51:00.242 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:51:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:51:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:51:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:51:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:51:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:51:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:51:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:51:00.252 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:51:00.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 4 16 10319 10303 160.964 140.625 0.00863 0.00620156 2024-03-26T03:51:00.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:00.322 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:00.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:00.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:00.466 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:00.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:00.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:00.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:00.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:00.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:00.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:00.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:00.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:00.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:00.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:01.010 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:00 smithi006 ceph-mon[29446]: pgmap v75: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:01.010 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1272811221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:01.010 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:00 smithi006 ceph-mon[37501]: pgmap v75: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:01.010 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1272811221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:00 smithi086 ceph-mon[32965]: pgmap v75: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1272811221' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:01.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:01.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:51:01.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:01.146 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:51:01.147 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:51:01.147 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-26T03:51:01.147 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:51:01.147 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:51:01.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-26T03:51:01.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:01.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:01.259 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 5 16 12497 12481 155.983 136.125 0.00810743 0.00640279 2024-03-26T03:51:01.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:01.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:01.470 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:51:01.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:01.534 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:01.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:01.677 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:01.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:01.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:01.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:01.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:01.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:01.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:01.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:02.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:02.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:02.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:02.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:02.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:02.260 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 6 16 14837 14821 154.338 146.25 0.00564282 0.00647095 2024-03-26T03:51:02.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:02.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:02.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:02.482 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:02.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:02.515 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:51:02.515 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:51:02.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:02.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:02.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:02.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:02.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:02.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:02 smithi006 ceph-mon[29446]: pgmap v76: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:02 smithi006 ceph-mon[37501]: pgmap v76: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:02.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:02.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:03.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:03.055 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:03.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:03.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:02 smithi086 ceph-mon[32965]: pgmap v76: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 976 op/s 2024-03-26T03:51:03.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:03.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:03.260 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 7 16 16738 16722 149.262 118.812 0.00378123 0.00666633 2024-03-26T03:51:03.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:03.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:03.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:03.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:03.435 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:03.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:03.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:03.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:03.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:03.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:03.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:03.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:03.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:03.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:03.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:51:03.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:03.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:03.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:04.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:04.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:04.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:04.237 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:51:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:51:04.243 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:51:04.261 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 8 16 18474 18458 144.162 108.5 0.00865909 0.00692874 2024-03-26T03:51:04.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:04.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:04.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:04.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:04.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:04.589 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:04.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:04.623 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:04.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:04.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:04.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:04.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:04.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:04.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:04.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:04.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:05.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:04 smithi086 ceph-mon[32965]: pgmap v77: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 827 op/s 2024-03-26T03:51:05.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:05.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:04 smithi006 ceph-mon[29446]: pgmap v77: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 827 op/s 2024-03-26T03:51:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:04 smithi006 ceph-mon[37501]: pgmap v77: 148 pgs: 148 active+clean; 488 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 52 MiB/s wr, 827 op/s 2024-03-26T03:51:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.172 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:05.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:05.261 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 9 15 20346 20331 141.151 117.062 0.012556 0.00707763 2024-03-26T03:51:05.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:05.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:05.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:05.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:05.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:05.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:05.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:05.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:05.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:05.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:05.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:05.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:51:05.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:05.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:05.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:06.004 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:06.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:06.121 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:06.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:06.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:06.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:06.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:06.261 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 10 15 22343 22328 139.516 124.812 0.00737123 0.00715956 2024-03-26T03:51:06.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:06.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:06.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:06.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:06.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:06.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:06.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:06.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:06.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:06.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:06.883 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:06.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:06.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:07.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:07.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:06 smithi086 ceph-mon[32965]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:51:07.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:07.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:07.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:06 smithi006 ceph-mon[37501]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:51:07.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:06 smithi006 ceph-mon[29446]: pgmap v78: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 114 MiB/s wr, 1.83k op/s 2024-03-26T03:51:07.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:07.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 11 15 23810 23795 135.168 91.6875 0.130921 0.00733413 2024-03-26T03:51:07.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:07.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:07.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:07.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:07.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.544 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:07.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:07.686 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:07.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:07.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:07.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:07.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:08.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:08.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:08.129 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:07 smithi086 ceph-mon[32965]: pgmap v79: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 984 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-26T03:51:08.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:07 smithi006 ceph-mon[37501]: pgmap v79: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 984 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-26T03:51:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:07 smithi006 ceph-mon[29446]: pgmap v79: 148 pgs: 148 active+clean; 1.1 GiB data, 3.4 GiB used, 622 GiB / 626 GiB avail; 984 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-26T03:51:08.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:08.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:08.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:08.261 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 12 16 25348 25332 131.907 96.0625 0.0066052 0.00757422 2024-03-26T03:51:08.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:08.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:08.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:08.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:08.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:08.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:08.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:08.626 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.627 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:08.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:08.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:08.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:08.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:08.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:08.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:09.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:09.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:09.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:09.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:09.262 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 13 16 26842 26826 128.942 93.375 0.00952528 0.00774856 2024-03-26T03:51:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:09.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:09.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:09.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:09.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[37501]: osdmap e103: 8 total, 8 up, 7 in 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:09 smithi006 ceph-mon[29446]: osdmap e103: 8 total, 8 up, 7 in 2024-03-26T03:51:09.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:09.709 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:09.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:09.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:09.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:09 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:09 smithi086 ceph-mon[32965]: osdmap e103: 8 total, 8 up, 7 in 2024-03-26T03:51:09.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:51:09.922 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:51:09.922 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-03-26T03:51:09.922 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-03-26T03:51:09.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:51:09.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:10.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:10.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:10.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:10.170 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:10.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 14 16 28525 28509 127.234 105.188 0.00809607 0.00785436 2024-03-26T03:51:10.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:10.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:10.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:10.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:10.496 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:10 smithi006 ceph-mon[29446]: pgmap v81: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:10.496 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:10 smithi006 ceph-mon[37501]: pgmap v81: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:10.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:10.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:10.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:10.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:10.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:10.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:10.829 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:10.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:10 smithi086 ceph-mon[32965]: pgmap v81: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:10.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:10.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:10.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:11.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:11.125 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:11.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:11.253 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:11.262 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 15 16 30281 30265 126.068 109.75 0.0066013 0.00792682 2024-03-26T03:51:11.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:11.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:11.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:11.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.525 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:11.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:11.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.675 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:11.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:11.776 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:11.818 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:11.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:11.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:11.962 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:11.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:12.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:12.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:12.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:12.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:12.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 16 16 31887 31871 124.462 100.375 0.00879764 0.00802755 2024-03-26T03:51:12.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:12.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:12.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:12.376 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:12.437 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:51:12.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:12.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:12.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:12.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:12.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: pgmap v82: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: osdmap e104: 8 total, 8 up, 7 in 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/108136412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:12.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: pgmap v82: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:12.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: osdmap e104: 8 total, 8 up, 7 in 2024-03-26T03:51:12.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/108136412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:12 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:12.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.792 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:12.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: pgmap v82: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-03-26T03:51:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: osdmap e104: 8 total, 8 up, 7 in 2024-03-26T03:51:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/108136412' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:51:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:12 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:12.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:12.955 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:12.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:12.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:13.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.101 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:13.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:13.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:13.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:13.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 17 16 33372 33356 122.6 92.8125 0.0101431 0.00815069 2024-03-26T03:51:13.319 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:13.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:13.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:13.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:13.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:13.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[37501]: osdmap e105: 8 total, 8 up, 7 in 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T03:51:13.576 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:13.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:51:13.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[29446]: osdmap e105: 8 total, 8 up, 7 in 2024-03-26T03:51:13.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T03:51:13.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:13 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T03:51:13.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:13.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:13.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:13.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:13 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:51:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:13 smithi086 ceph-mon[32965]: osdmap e105: 8 total, 8 up, 7 in 2024-03-26T03:51:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:13 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T03:51:13.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:13 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T03:51:13.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:13.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:13.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:13.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:14.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:14.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:14.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:14.216 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:14.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:14.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 18 16 34892 34876 121.067 95 0.0158975 0.00825463 2024-03-26T03:51:14.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:14.361 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:14.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:14.415 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:14.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:14.503 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:14.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1614492355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: osdmap e106: 8 total, 8 up, 7 in 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:51:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:14 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:14.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:14.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.644 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:14.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:14.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:14.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:14.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1614492355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: osdmap e106: 8 total, 8 up, 7 in 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:51:14.786 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: pgmap v85: 148 pgs: 148 active+clean; 1.6 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1614492355' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: osdmap e106: 8 total, 8 up, 7 in 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:51:14.787 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:14 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:14.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:14.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:14.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:14.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:14.929 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:51:14.929 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:51:14.929 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:51:14.930 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:51:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:51:14.931 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:51:15.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:15.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:15.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:15.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:15.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:15.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:15.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 19 16 36517 36501 120.04 101.562 0.0132872 0.00832461 2024-03-26T03:51:15.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:15.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:15.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:51:15.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:15.456 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:15.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:15.500 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:15.500 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:15.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:15.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:15.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:15.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:15.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:15.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:15.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:15.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:15.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:16.001 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:16.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:16.129 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:16.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:16.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:51:16.263885+0000 min lat: 0.00129465 max lat: 0.140595 avg lat: 0.00842511 2024-03-26T03:51:16.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:51:16.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 20 16 37978 37962 118.603 91.3125 0.00669104 0.00842511 2024-03-26T03:51:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:16.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:16.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:16.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:16.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:16.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:16.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:16.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:16.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:16.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:16.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T03:51:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T03:51:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T03:51:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T03:51:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: pgmap v87: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T03:51:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:16 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: pgmap v87: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T03:51:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: pgmap v87: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T03:51:16.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:16 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T03:51:16.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:16.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:16.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:17.024 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:37282 7.d 7:bf4540ab:::benchmark_data_smithi006_77260_object37281:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.810885+0000", 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0089248209999999994, 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0089712760000000003, 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:51:17.025 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 37282 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810885+0000", 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810885+0000", 2024-03-26T03:51:17.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810886+0000", 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0720000000000001e-06 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810906+0000", 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0194e-05 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810984+0000", 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8022999999999995e-05 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.810998+0000", 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3981999999999999e-05 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.811664+0000", 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00066575 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.811778+0000", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114206 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.811903+0000", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000125222 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819505+0000", 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076013699999999997 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819559+0000", 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4305999999999998e-05 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:37285 7.d 7:bbe0a387:::benchmark_data_smithi006_77260_object37284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.812815+0000", 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0069951520000000001, 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0071565650000000001, 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 37285 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812815+0000", 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812815+0000", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812816+0000", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.432e-06 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812840+0000", 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.402e-05 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812852+0000", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1616e-05 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812859+0000", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5000000000000002e-06 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.814511+0000", 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001651921 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.814620+0000", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000108849 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.814744+0000", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012405200000000001 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819511+0000", 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047665880000000004 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.034 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:37286 7.d 7:bb6cf7f1:::benchmark_data_smithi006_77260_object37285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.812878+0000", 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0069313780000000002, 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0071973970000000003, 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:51:17.035 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 37286 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812878+0000", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812878+0000", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812879+0000", 2024-03-26T03:51:17.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2399999999999998e-07 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812898+0000", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8821000000000002e-05 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812905+0000", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9009999999999997e-06 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.812910+0000", 2024-03-26T03:51:17.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7069999999999997e-06 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817642+0000", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047318059999999999 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817792+0000", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014982099999999999 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817940+0000", 2024-03-26T03:51:17.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014845099999999999 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819572+0000", 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016311100000000001 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:37293 7.1 e106/96 7:8e5de40f:::benchmark_data_smithi006_77260_object37292:head v 106'2353, mlcod=106'2351)", 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.816134+0000", 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0036760230000000001, 2024-03-26T03:51:17.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040351720000000001, 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816134+0000", 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816134+0000", 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816134+0000", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0999999999999998e-07 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816229+0000", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4112999999999998e-05 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816243+0000", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4197e-05 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816250+0000", 2024-03-26T03:51:17.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0709999999999997e-06 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817417+0000", 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011671870000000001 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817436+0000", 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8610999999999999e-05 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:37295 7.d 7:b298dc99:::benchmark_data_smithi006_77260_object37294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e106)", 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.816288+0000", 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0035221150000000001, 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039551839999999996, 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 37295 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816288+0000", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816288+0000", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816290+0000", 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8810000000000001e-06 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816310+0000", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0159000000000001e-05 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816319+0000", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7680000000000003e-06 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.816326+0000", 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9099999999999999e-06 2024-03-26T03:51:17.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819657+0000", 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033309920000000001 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819782+0000", 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012462899999999999 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,3", 2024-03-26T03:51:17.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.820130+0000", 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00034810800000000002 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:37292 7.7 e106/96 7:e8b1afc2:::benchmark_data_smithi006_77260_object37291:head v 106'2295, mlcod=106'2293)", 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.817171+0000", 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.002638385, 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003155326, 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817171+0000", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817171+0000", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817172+0000", 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6499999999999999e-07 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817195+0000", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.298e-05 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817204+0000", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8489999999999995e-06 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.817210+0000", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3679999999999998e-06 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:17.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.818592+0000", 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001382 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.818611+0000", 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8970999999999998e-05 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:37285 7.d e106/96)", 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.819823+0000", 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4294967295.9999871, 2024-03-26T03:51:17.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057250599999999997, 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819823+0000", 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819823+0000", 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819824+0000", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.78e-07 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819829+0000", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9409999999999998e-06 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819837+0000", 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8710000000000003e-06 2024-03-26T03:51:17.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819844+0000", 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9380000000000003e-06 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:37286 7.d e106/96)", 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:15.819866+0000", 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4294967295.9999442, 2024-03-26T03:51:17.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00058315999999999999, 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819866+0000", 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:17.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819866+0000", 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819866+0000", 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0699999999999998e-07 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819869+0000", 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9100000000000001e-06 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819876+0000", 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0620000000000003e-06 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:15.819882+0000", 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6879999999999996e-06 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 8 2024-03-26T03:51:17.056 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:51:17.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:17.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:17.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:17.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:17.225 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:17.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 21 16 39589 39573 117.75 100.688 0.0155776 0.0084869 2024-03-26T03:51:17.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:17.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:17.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:17.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:17.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:17.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T03:51:17.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T03:51:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:17 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T03:51:17.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:17.778 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:17.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:51:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T03:51:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T03:51:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T03:51:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T03:51:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T03:51:17.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:17 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T03:51:17.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:17.940 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:17.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:18.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:18.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:18.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:18.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:18.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:18.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:51:18.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 22 16 41062 41046 116.582 92.0625 0.0103338 0.00857228 2024-03-26T03:51:18.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:18.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:18.469 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:18.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:18.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:18.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.507 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:18.533 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:18.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T03:51:18.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T03:51:18.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:18 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T03:51:18.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.667 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:18.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:18.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:18.812 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T03:51:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T03:51:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: pgmap v88: 148 pgs: 148 active+clean; 2.1 GiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.35k op/s 2024-03-26T03:51:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T03:51:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T03:51:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T03:51:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T03:51:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T03:51:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:18 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T03:51:18.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:18.931 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:51:18.945 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:51:18.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:18.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:18.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:19.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:19.084 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:19.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:19.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:19.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:19.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:19.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 23 16 42310 42294 114.904 78 0.0124615 0.00869678 2024-03-26T03:51:19.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:19.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:19.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:19.500 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:19.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:19.641 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T03:51:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T03:51:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T03:51:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[29446]: osdmap e107: 8 total, 8 up, 7 in 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T03:51:19.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T03:51:19.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T03:51:19.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T03:51:19.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T03:51:19.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:19 smithi006 ceph-mon[37501]: osdmap e107: 8 total, 8 up, 7 in 2024-03-26T03:51:19.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:19.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.790 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:19.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T03:51:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T03:51:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:19 smithi086 ceph-mon[32965]: osdmap e107: 8 total, 8 up, 7 in 2024-03-26T03:51:19.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:19.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:19.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:20.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:20.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:20.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:20.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:20.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:20.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 24 15 43840 43825 114.103 95.6875 0.00990208 0.00875783 2024-03-26T03:51:20.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:20.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:20.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:20.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:20.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:20.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:20.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:20.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:20.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: from='client.24673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: pgmap v90: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:51:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:20 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:51:20.844 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:20.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: from='client.24673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: pgmap v90: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: from='client.24673 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:51:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: pgmap v90: 148 pgs: 148 active+clean; 2.5 GiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1.5 KiB/s rd, 126 MiB/s wr, 2.01k op/s 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:51:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:20 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:51:20.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:20.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:20.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:21.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:21.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:21.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:21.237 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:21.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 25 16 44839 44823 112.034 62.375 0.0105291 0.00890008 2024-03-26T03:51:21.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:21.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:21.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:21.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:21.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:21.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:21.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:21.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:21.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T03:51:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:21 smithi086 ceph-mon[32965]: osdmap e108: 8 total, 8 up, 7 in 2024-03-26T03:51:21.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:21.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:21.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:21.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[37501]: osdmap e108: 8 total, 8 up, 7 in 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T03:51:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T03:51:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T03:51:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:21 smithi006 ceph-mon[29446]: osdmap e108: 8 total, 8 up, 7 in 2024-03-26T03:51:21.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:22.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:22.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:22.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:22.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:22.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:22.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:22.263 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 26 16 45960 45944 110.42 70.0625 0.00357868 0.00902597 2024-03-26T03:51:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:22.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:22.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:22.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:22.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:22.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:22.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:22.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:22.693 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:22.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:22.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:22.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 6.2 scrub starts 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 6.2 scrub ok 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 6.2 scrub starts 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 6.2 scrub ok 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T03:51:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: pgmap v92: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 729 B/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 6.2 scrub starts 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 6.2 scrub ok 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:22.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:22.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:22.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:23.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:23.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:23.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:23.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:23.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:23.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 27 16 46965 46949 108.657 62.8125 0.0160797 0.00919764 2024-03-26T03:51:23.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:23.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:23.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:23.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:23.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:23.597 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:23.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:23.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[37501]: osdmap e109: 8 total, 8 up, 7 in 2024-03-26T03:51:23.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[29446]: osdmap e109: 8 total, 8 up, 7 in 2024-03-26T03:51:23.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T03:51:23.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:23.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:23.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:23 smithi086 ceph-mon[32965]: osdmap e109: 8 total, 8 up, 7 in 2024-03-26T03:51:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:23 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T03:51:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:23 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T03:51:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:23 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:23.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:23.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:23 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:23.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:24.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:24.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:24.138 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:24.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:24.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:24.264 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 28 16 48169 48153 107.463 75.25 0.0151073 0.00929906 2024-03-26T03:51:24.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:24.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:24.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:24.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:24.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:24.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:24.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.541 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:24.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:24.559 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:24.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:24.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:24.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:24.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T03:51:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: pgmap v94: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: osdmap e110: 8 total, 8 up, 7 in 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T03:51:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:24 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: pgmap v94: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T03:51:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: osdmap e110: 8 total, 8 up, 7 in 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: pgmap v94: 148 pgs: 148 active+clean; 2.6 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T03:51:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: osdmap e110: 8 total, 8 up, 7 in 2024-03-26T03:51:24.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T03:51:24.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:24 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T03:51:24.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:24.940 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:24.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:25.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:25.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:25.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:25.171 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-03-26T03:51:17.193508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2280987048,"num_objects":35026,"num_object_clones":0,"num_object_copies":105078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35026,"num_whiteouts":0,"num_read":1362,"num_read_kb":1176,"num_write":35603,"num_write_kb":2229405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":95,"num_bytes_recovered":2362635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35621,"ondisk_log_size":35621,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6862364,"kb_used_data":6538580,"kb_used_omap":10,"kb_used_meta":323637,"kb_avail":649353700,"statfs":{"total":671965249536,"available":664938188800,"internally_reserved":0,"allocated":6695505920,"data_stored":6692526563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":331404394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":532807680,"num_objects":8130,"num_object_clones":0,"num_object_copies":24390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8130,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8130,"num_write_kb":520320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002896"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":85,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414663+0000","last_change":"2024-03-26T03:50:34.035051+0000","last_active":"2024-03-26T03:51:12.414663+0000","last_peered":"2024-03-26T03:51:12.414663+0000","last_clean":"2024-03-26T03:51:12.414663+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:51:12.414663+0000","last_undegraded":"2024-03-26T03:51:12.414663+0000","last_fullsized":"2024-03-26T03:51:12.414663+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:34.034948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:34.034948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:03.490512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.437436+0000","last_change":"2024-03-26T03:50:40.094687+0000","last_active":"2024-03-26T03:51:12.437436+0000","last_peered":"2024-03-26T03:51:12.437436+0000","last_clean":"2024-03-26T03:51:12.437436+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:51:12.437436+0000","last_undegraded":"2024-03-26T03:51:12.437436+0000","last_fullsized":"2024-03-26T03:51:12.437436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:50:40.094601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:51.416643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426965+0000","last_change":"2024-03-26T03:49:49.005275+0000","last_active":"2024-03-26T03:51:12.426965+0000","last_peered":"2024-03-26T03:51:12.426965+0000","last_clean":"2024-03-26T03:51:12.426965+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:51:12.426965+0000","last_undegraded":"2024-03-26T03:51:12.426965+0000","last_fullsized":"2024-03-26T03:51:12.426965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:49.005129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:49.005129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.159629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":140,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414592+0000","last_change":"2024-03-26T03:50:45.623345+0000","last_active":"2024-03-26T03:51:12.414592+0000","last_peered":"2024-03-26T03:51:12.414592+0000","last_clean":"2024-03-26T03:51:12.414592+0000","last_became_active":"2024-03-26T03:50:45.598168+0000","last_became_peered":"2024-03-26T03:50:45.598168+0000","last_unstale":"2024-03-26T03:51:12.414592+0000","last_undegraded":"2024-03-26T03:51:12.414592+0000","last_fullsized":"2024-03-26T03:51:12.414592+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:18.012741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.012741+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.264234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424014+0000","last_change":"2024-03-26T03:50:45.598350+0000","last_active":"2024-03-26T03:51:12.424014+0000","last_peered":"2024-03-26T03:51:12.424014+0000","last_clean":"2024-03-26T03:51:12.424014+0000","last_became_active":"2024-03-26T03:50:45.597941+0000","last_became_peered":"2024-03-26T03:50:45.597941+0000","last_unstale":"2024-03-26T03:51:12.424014+0000","last_undegraded":"2024-03-26T03:51:12.424014+0000","last_fullsized":"2024-03-26T03:51:12.424014+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.102383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.102383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:37.891662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414466+0000","last_change":"2024-03-26T03:50:45.601271+0000","last_active":"2024-03-26T03:51:12.414466+0000","last_peered":"2024-03-26T03:51:12.414466+0000","last_clean":"2024-03-26T03:51:12.414466+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:51:12.414466+0000","last_undegraded":"2024-03-26T03:51:12.414466+0000","last_fullsized":"2024-03-26T03:51:12.414466+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:32.005669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:32.005669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:51.893383+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424082+0000","last_change":"2024-03-26T03:50:45.582343+0000","last_active":"2024-03-26T03:51:12.424082+0000","last_peered":"2024-03-26T03:51:12.424082+0000","last_clean":"2024-03-26T03:51:12.424082+0000","last_became_active":"2024-03-26T03:50:45.581989+0000","last_became_peered":"2024-03-26T03:50:45.581989+0000","last_unstale":"2024-03-26T03:51:12.424082+0000","last_undegraded":"2024-03-26T03:51:12.424082+0000","last_fullsized":"2024-03-26T03:51:12.424082+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":166,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427011+0000","last_change":"2024-03-26T03:50:40.221494+0000","last_active":"2024-03-26T03:51:12.427011+0000","last_peered":"2024-03-26T03:51:12.427011+0000","last_clean":"2024-03-26T03:51:12.427011+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:51:12.427011+0000","last_undegraded":"2024-03-26T03:51:12.427011+0000","last_fullsized":"2024-03-26T03:51:12.427011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:50:40.221401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:40.221401+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:50.432973+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427552+0000","last_change":"2024-03-26T03:48:50.037848+0000","last_active":"2024-03-26T03:51:12.427552+0000","last_peered":"2024-03-26T03:51:12.427552+0000","last_clean":"2024-03-26T03:51:12.427552+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:51:12.427552+0000","last_undegraded":"2024-03-26T03:51:12.427552+0000","last_fullsized":"2024-03-26T03:51:12.427552+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423557+0000","last_change":"2024-03-26T03:49:59.135484+0000","last_active":"2024-03-26T03:51:12.423557+0000","last_peered":"2024-03-26T03:51:12.423557+0000","last_clean":"2024-03-26T03:51:12.423557+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:51:12.423557+0000","last_undegraded":"2024-03-26T03:51:12.423557+0000","last_fullsized":"2024-03-26T03:51:12.423557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.135403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.135403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.439280+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":103,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423938+0000","last_change":"2024-03-26T03:50:45.596100+0000","last_active":"2024-03-26T03:51:12.423938+0000","last_peered":"2024-03-26T03:51:12.423938+0000","last_clean":"2024-03-26T03:51:12.423938+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:51:12.423938+0000","last_undegraded":"2024-03-26T03:51:12.423938+0000","last_fullsized":"2024-03-26T03:51:12.423938+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:09.087395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.087395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":135,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414857+0000","last_change":"2024-03-26T03:49:54.881008+0000","last_active":"2024-03-26T03:51:12.414857+0000","last_peered":"2024-03-26T03:51:12.414857+0000","last_clean":"2024-03-26T03:51:12.414857+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:51:12.414857+0000","last_undegraded":"2024-03-26T03:51:12.414857+0000","last_fullsized":"2024-03-26T03:51:12.414857+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:49:54.880892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.880892+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.321632+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":85,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423250+0000","last_change":"2024-03-26T03:50:23.965433+0000","last_active":"2024-03-26T03:51:12.423250+0000","last_peered":"2024-03-26T03:51:12.423250+0000","last_clean":"2024-03-26T03:51:12.423250+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:51:12.423250+0000","last_undegraded":"2024-03-26T03:51:12.423250+0000","last_fullsized":"2024-03-26T03:51:12.423250+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:23.965318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:23.965318+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:42.423797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420212+0000","last_change":"2024-03-26T03:50:35.757657+0000","last_active":"2024-03-26T03:51:12.420212+0000","last_peered":"2024-03-26T03:51:12.420212+0000","last_clean":"2024-03-26T03:51:12.420212+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:51:12.420212+0000","last_undegraded":"2024-03-26T03:51:12.420212+0000","last_fullsized":"2024-03-26T03:51:12.420212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.757523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:53.370054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423788+0000","last_change":"2024-03-26T03:50:24.935787+0000","last_active":"2024-03-26T03:51:12.423788+0000","last_peered":"2024-03-26T03:51:12.423788+0000","last_clean":"2024-03-26T03:51:12.423788+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:51:12.423788+0000","last_undegraded":"2024-03-26T03:51:12.423788+0000","last_fullsized":"2024-03-26T03:51:12.423788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:24.935705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:24.935705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:45.361829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'11","reported_seq":116,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414952+0000","last_change":"2024-03-26T03:50:21.934154+0000","last_active":"2024-03-26T03:51:12.414952+0000","last_peered":"2024-03-26T03:51:12.414952+0000","last_clean":"2024-03-26T03:51:12.414952+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:51:12.414952+0000","last_undegraded":"2024-03-26T03:51:12.414952+0000","last_fullsized":"2024-03-26T03:51:12.414952+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:50:21.934063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.934063+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.627389+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.416071+0000","last_change":"2024-03-26T03:50:45.588768+0000","last_active":"2024-03-26T03:51:12.416071+0000","last_peered":"2024-03-26T03:51:12.416071+0000","last_clean":"2024-03-26T03:51:12.416071+0000","last_became_active":"2024-03-26T03:50:45.588396+0000","last_became_peered":"2024-03-26T03:50:45.588396+0000","last_unstale":"2024-03-26T03:51:12.416071+0000","last_undegraded":"2024-03-26T03:51:12.416071+0000","last_fullsized":"2024-03-26T03:51:12.416071+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:35.965675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.965675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:47.887399+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":80,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432627+0000","last_change":"2024-03-26T03:48:48.025562+0000","last_active":"2024-03-26T03:51:12.432627+0000","last_peered":"2024-03-26T03:51:12.432627+0000","last_clean":"2024-03-26T03:51:12.432627+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:51:12.432627+0000","last_undegraded":"2024-03-26T03:51:12.432627+0000","last_fullsized":"2024-03-26T03:51:12.432627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427074+0000","last_change":"2024-03-26T03:49:58.053896+0000","last_active":"2024-03-26T03:51:12.427074+0000","last_peered":"2024-03-26T03:51:12.427074+0000","last_clean":"2024-03-26T03:51:12.427074+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:51:12.427074+0000","last_undegraded":"2024-03-26T03:51:12.427074+0000","last_fullsized":"2024-03-26T03:51:12.427074+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.053811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.053811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":119,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417248+0000","last_change":"2024-03-26T03:49:53.266632+0000","last_active":"2024-03-26T03:51:12.417248+0000","last_peered":"2024-03-26T03:51:12.417248+0000","last_clean":"2024-03-26T03:51:12.417248+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:51:12.417248+0000","last_undegraded":"2024-03-26T03:51:12.417248+0000","last_fullsized":"2024-03-26T03:51:12.417248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:49:53.266536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:53.266536+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.703552+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.439548+0000","last_change":"2024-03-26T03:50:45.591763+0000","last_active":"2024-03-26T03:51:12.439548+0000","last_peered":"2024-03-26T03:51:12.439548+0000","last_clean":"2024-03-26T03:51:12.439548+0000","last_became_active":"2024-03-26T03:50:45.591442+0000","last_became_peered":"2024-03-26T03:50:45.591442+0000","last_unstale":"2024-03-26T03:51:12.439548+0000","last_undegraded":"2024-03-26T03:51:12.439548+0000","last_fullsized":"2024-03-26T03:51:12.439548+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.311092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415231+0000","last_change":"2024-03-26T03:50:31.053785+0000","last_active":"2024-03-26T03:51:12.415231+0000","last_peered":"2024-03-26T03:51:12.415231+0000","last_clean":"2024-03-26T03:51:12.415231+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:51:12.415231+0000","last_undegraded":"2024-03-26T03:51:12.415231+0000","last_fullsized":"2024-03-26T03:51:12.415231+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:31.053677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:31.053677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:43.167547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:13.376075+0000","last_change":"2024-03-26T03:51:13.376075+0000","last_active":"2024-03-26T03:51:13.376075+0000","last_peered":"2024-03-26T03:51:13.376075+0000","last_clean":"2024-03-26T03:51:13.376075+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:51:13.376075+0000","last_undegraded":"2024-03-26T03:51:13.376075+0000","last_fullsized":"2024-03-26T03:51:13.376075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:13.375964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:13.375964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:30.878580+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":128,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.425345+0000","last_change":"2024-03-26T03:50:07.047577+0000","last_active":"2024-03-26T03:51:12.425345+0000","last_peered":"2024-03-26T03:51:12.425345+0000","last_clean":"2024-03-26T03:51:12.425345+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:51:12.425345+0000","last_undegraded":"2024-03-26T03:51:12.425345+0000","last_fullsized":"2024-03-26T03:51:12.425345+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:50:07.047453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.047453+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.870720+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420649+0000","last_change":"2024-03-26T03:50:02.740650+0000","last_active":"2024-03-26T03:51:12.420649+0000","last_peered":"2024-03-26T03:51:12.420649+0000","last_clean":"2024-03-26T03:51:12.420649+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:51:12.420649+0000","last_undegraded":"2024-03-26T03:51:12.420649+0000","last_fullsized":"2024-03-26T03:51:12.420649+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.740545+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.965068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":69,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415714+0000","last_change":"2024-03-26T03:50:45.614685+0000","last_active":"2024-03-26T03:51:12.415714+0000","last_peered":"2024-03-26T03:51:12.415714+0000","last_clean":"2024-03-26T03:51:12.415714+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:51:12.415714+0000","last_undegraded":"2024-03-26T03:51:12.415714+0000","last_fullsized":"2024-03-26T03:51:12.415714+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:30.502897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:30.502897+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:39.298920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.437380+0000","last_change":"2024-03-26T03:50:45.116437+0000","last_active":"2024-03-26T03:51:12.437380+0000","last_peered":"2024-03-26T03:51:12.437380+0000","last_clean":"2024-03-26T03:51:12.437380+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:51:12.437380+0000","last_undegraded":"2024-03-26T03:51:12.437380+0000","last_fullsized":"2024-03-26T03:51:12.437380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:45.116355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:45.116355+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:54.408122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.429176+0000","last_change":"2024-03-26T03:50:32.222931+0000","last_active":"2024-03-26T03:51:12.429176+0000","last_peered":"2024-03-26T03:51:12.429176+0000","last_clean":"2024-03-26T03:51:12.429176+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:51:12.429176+0000","last_undegraded":"2024-03-26T03:51:12.429176+0000","last_fullsized":"2024-03-26T03:51:12.429176+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:50:32.222835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:32.222835+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:51.154781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432297+0000","last_change":"2024-03-26T03:50:25.006849+0000","last_active":"2024-03-26T03:51:12.432297+0000","last_peered":"2024-03-26T03:51:12.432297+0000","last_clean":"2024-03-26T03:51:12.432297+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:51:12.432297+0000","last_undegraded":"2024-03-26T03:51:12.432297+0000","last_fullsized":"2024-03-26T03:51:12.432297+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:25.006765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:25.006765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:48.097037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.429489+0000","last_change":"2024-03-26T03:50:07.149609+0000","last_active":"2024-03-26T03:51:12.429489+0000","last_peered":"2024-03-26T03:51:12.429489+0000","last_clean":"2024-03-26T03:51:12.429489+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:51:12.429489+0000","last_undegraded":"2024-03-26T03:51:12.429489+0000","last_fullsized":"2024-03-26T03:51:12.429489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.149533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.149533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":92,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.194115+0000","last_change":"2024-03-26T03:48:44.016177+0000","last_active":"2024-03-26T03:51:14.194115+0000","last_peered":"2024-03-26T03:51:14.194115+0000","last_clean":"2024-03-26T03:51:14.194115+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:51:14.194115+0000","last_undegraded":"2024-03-26T03:51:14.194115+0000","last_fullsized":"2024-03-26T03:51:14.194115+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424054+0000","last_change":"2024-03-26T03:50:45.618620+0000","last_active":"2024-03-26T03:51:12.424054+0000","last_peered":"2024-03-26T03:51:12.424054+0000","last_clean":"2024-03-26T03:51:12.424054+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:51:12.424054+0000","last_undegraded":"2024-03-26T03:51:12.424054+0000","last_fullsized":"2024-03-26T03:51:12.424054+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:49:58.491875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.491875+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.592161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420187+0000","last_change":"2024-03-26T03:50:32.739375+0000","last_active":"2024-03-26T03:51:12.420187+0000","last_peered":"2024-03-26T03:51:12.420187+0000","last_clean":"2024-03-26T03:51:12.420187+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:51:12.420187+0000","last_undegraded":"2024-03-26T03:51:12.420187+0000","last_fullsized":"2024-03-26T03:51:12.420187+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:32.739258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:32.739258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:33.856103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":28,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414582+0000","last_change":"2024-03-26T03:50:45.596765+0000","last_active":"2024-03-26T03:51:12.414582+0000","last_peered":"2024-03-26T03:51:12.414582+0000","last_clean":"2024-03-26T03:51:12.414582+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:51:12.414582+0000","last_undegraded":"2024-03-26T03:51:12.414582+0000","last_fullsized":"2024-03-26T03:51:12.414582+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.564988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.564988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.759468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427176+0000","last_change":"2024-03-26T03:50:13.220621+0000","last_active":"2024-03-26T03:51:12.427176+0000","last_peered":"2024-03-26T03:51:12.427176+0000","last_clean":"2024-03-26T03:51:12.427176+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:51:12.427176+0000","last_undegraded":"2024-03-26T03:51:12.427176+0000","last_fullsized":"2024-03-26T03:51:12.427176+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.220474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.220474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.320102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":114,"reported_epoch":106,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.194217+0000","last_change":"2024-03-26T03:50:45.606641+0000","last_active":"2024-03-26T03:51:14.194217+0000","last_peered":"2024-03-26T03:51:14.194217+0000","last_clean":"2024-03-26T03:51:14.194217+0000","last_became_active":"2024-03-26T03:50:45.587808+0000","last_became_peered":"2024-03-26T03:50:45.587808+0000","last_unstale":"2024-03-26T03:51:14.194217+0000","last_undegraded":"2024-03-26T03:51:14.194217+0000","last_fullsized":"2024-03-26T03:51:14.194217+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-03-26T03:50:35.236237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.236237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:54.169603+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424397+0000","last_change":"2024-03-26T03:50:45.590379+0000","last_active":"2024-03-26T03:51:12.424397+0000","last_peered":"2024-03-26T03:51:12.424397+0000","last_clean":"2024-03-26T03:51:12.424397+0000","last_became_active":"2024-03-26T03:50:45.589899+0000","last_became_peered":"2024-03-26T03:50:45.589899+0000","last_unstale":"2024-03-26T03:51:12.424397+0000","last_undegraded":"2024-03-26T03:51:12.424397+0000","last_fullsized":"2024-03-26T03:51:12.424397+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.094208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:42.964469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420390+0000","last_change":"2024-03-26T03:50:45.593554+0000","last_active":"2024-03-26T03:51:12.420390+0000","last_peered":"2024-03-26T03:51:12.420390+0000","last_clean":"2024-03-26T03:51:12.420390+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:51:12.420390+0000","last_undegraded":"2024-03-26T03:51:12.420390+0000","last_fullsized":"2024-03-26T03:51:12.420390+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:25.754709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:25.754709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:40.306509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424145+0000","last_change":"2024-03-26T03:50:08.135619+0000","last_active":"2024-03-26T03:51:12.424145+0000","last_peered":"2024-03-26T03:51:12.424145+0000","last_clean":"2024-03-26T03:51:12.424145+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:51:12.424145+0000","last_undegraded":"2024-03-26T03:51:12.424145+0000","last_fullsized":"2024-03-26T03:51:12.424145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.135537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.135537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:22.675656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":151,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426891+0000","last_change":"2024-03-26T03:50:45.612604+0000","last_active":"2024-03-26T03:51:12.426891+0000","last_peered":"2024-03-26T03:51:12.426891+0000","last_clean":"2024-03-26T03:51:12.426891+0000","last_became_active":"2024-03-26T03:50:45.595599+0000","last_became_peered":"2024-03-26T03:50:45.595599+0000","last_unstale":"2024-03-26T03:51:12.426891+0000","last_undegraded":"2024-03-26T03:51:12.426891+0000","last_fullsized":"2024-03-26T03:51:12.426891+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.135174+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:43.067050+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426043+0000","last_change":"2024-03-26T03:50:34.154191+0000","last_active":"2024-03-26T03:51:12.426043+0000","last_peered":"2024-03-26T03:51:12.426043+0000","last_clean":"2024-03-26T03:51:12.426043+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:51:12.426043+0000","last_undegraded":"2024-03-26T03:51:12.426043+0000","last_fullsized":"2024-03-26T03:51:12.426043+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:34.154097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:34.154097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:41.821674+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423915+0000","last_change":"2024-03-26T03:49:56.980611+0000","last_active":"2024-03-26T03:51:12.423915+0000","last_peered":"2024-03-26T03:51:12.423915+0000","last_clean":"2024-03-26T03:51:12.423915+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:51:12.423915+0000","last_undegraded":"2024-03-26T03:51:12.423915+0000","last_fullsized":"2024-03-26T03:51:12.423915+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.980524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.980524+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.615228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420708+0000","last_change":"2024-03-26T03:49:50.773333+0000","last_active":"2024-03-26T03:51:12.420708+0000","last_peered":"2024-03-26T03:51:12.420708+0000","last_clean":"2024-03-26T03:51:12.420708+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:51:12.420708+0000","last_undegraded":"2024-03-26T03:51:12.420708+0000","last_fullsized":"2024-03-26T03:51:12.420708+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:50.773202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:50.773202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.022373+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":125,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432427+0000","last_change":"2024-03-26T03:50:45.605065+0000","last_active":"2024-03-26T03:51:12.432427+0000","last_peered":"2024-03-26T03:51:12.432427+0000","last_clean":"2024-03-26T03:51:12.432427+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:51:12.432427+0000","last_undegraded":"2024-03-26T03:51:12.432427+0000","last_fullsized":"2024-03-26T03:51:12.432427+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:49:51.011226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.011226+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426583+0000","last_change":"2024-03-26T03:50:02.986818+0000","last_active":"2024-03-26T03:51:12.426583+0000","last_peered":"2024-03-26T03:51:12.426583+0000","last_clean":"2024-03-26T03:51:12.426583+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:51:12.426583+0000","last_undegraded":"2024-03-26T03:51:12.426583+0000","last_fullsized":"2024-03-26T03:51:12.426583+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.986736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.986736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.565698+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432150+0000","last_change":"2024-03-26T03:50:16.056409+0000","last_active":"2024-03-26T03:51:12.432150+0000","last_peered":"2024-03-26T03:51:12.432150+0000","last_clean":"2024-03-26T03:51:12.432150+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:51:12.432150+0000","last_undegraded":"2024-03-26T03:51:12.432150+0000","last_fullsized":"2024-03-26T03:51:12.432150+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:16.056326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.056326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:42.021645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415123+0000","last_change":"2024-03-26T03:50:45.594303+0000","last_active":"2024-03-26T03:51:12.415123+0000","last_peered":"2024-03-26T03:51:12.415123+0000","last_clean":"2024-03-26T03:51:12.415123+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:51:12.415123+0000","last_undegraded":"2024-03-26T03:51:12.415123+0000","last_fullsized":"2024-03-26T03:51:12.415123+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.925725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.925725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.810018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415014+0000","last_change":"2024-03-26T03:50:45.624493+0000","last_active":"2024-03-26T03:51:12.415014+0000","last_peered":"2024-03-26T03:51:12.415014+0000","last_clean":"2024-03-26T03:51:12.415014+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:51:12.415014+0000","last_undegraded":"2024-03-26T03:51:12.415014+0000","last_fullsized":"2024-03-26T03:51:12.415014+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.321517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.428930+0000","last_change":"2024-03-26T03:48:50.046189+0000","last_active":"2024-03-26T03:51:12.428930+0000","last_peered":"2024-03-26T03:51:12.428930+0000","last_clean":"2024-03-26T03:51:12.428930+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:51:12.428930+0000","last_undegraded":"2024-03-26T03:51:12.428930+0000","last_fullsized":"2024-03-26T03:51:12.428930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.088436+0000","last_change":"2024-03-26T03:49:55.131855+0000","last_active":"2024-03-26T03:51:14.088436+0000","last_peered":"2024-03-26T03:51:14.088436+0000","last_clean":"2024-03-26T03:51:14.088436+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:51:14.088436+0000","last_undegraded":"2024-03-26T03:51:14.088436+0000","last_fullsized":"2024-03-26T03:51:14.088436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:55.131726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:55.131726+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420342+0000","last_change":"2024-03-26T03:49:48.735316+0000","last_active":"2024-03-26T03:51:12.420342+0000","last_peered":"2024-03-26T03:51:12.420342+0000","last_clean":"2024-03-26T03:51:12.420342+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:51:12.420342+0000","last_undegraded":"2024-03-26T03:51:12.420342+0000","last_fullsized":"2024-03-26T03:51:12.420342+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.735213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.735213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.022355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":125,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.435313+0000","last_change":"2024-03-26T03:48:46.209727+0000","last_active":"2024-03-26T03:51:12.435313+0000","last_peered":"2024-03-26T03:51:12.435313+0000","last_clean":"2024-03-26T03:51:12.435313+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:51:12.435313+0000","last_undegraded":"2024-03-26T03:51:12.435313+0000","last_fullsized":"2024-03-26T03:51:12.435313+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:48:45.002460+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":85,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415811+0000","last_change":"2024-03-26T03:50:12.892351+0000","last_active":"2024-03-26T03:51:12.415811+0000","last_peered":"2024-03-26T03:51:12.415811+0000","last_clean":"2024-03-26T03:51:12.415811+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:51:12.415811+0000","last_undegraded":"2024-03-26T03:51:12.415811+0000","last_fullsized":"2024-03-26T03:51:12.415811+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.892200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.892200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.447366+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.095377+0000","last_change":"2024-03-26T03:48:48.029400+0000","last_active":"2024-03-26T03:51:14.095377+0000","last_peered":"2024-03-26T03:51:14.095377+0000","last_clean":"2024-03-26T03:51:14.095377+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:51:14.095377+0000","last_undegraded":"2024-03-26T03:51:14.095377+0000","last_fullsized":"2024-03-26T03:51:14.095377+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426505+0000","last_change":"2024-03-26T03:50:35.194524+0000","last_active":"2024-03-26T03:51:12.426505+0000","last_peered":"2024-03-26T03:51:12.426505+0000","last_clean":"2024-03-26T03:51:12.426505+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:51:12.426505+0000","last_undegraded":"2024-03-26T03:51:12.426505+0000","last_fullsized":"2024-03-26T03:51:12.426505+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.194368+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:04.533915+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":127,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424252+0000","last_change":"2024-03-26T03:50:20.001811+0000","last_active":"2024-03-26T03:51:12.424252+0000","last_peered":"2024-03-26T03:51:12.424252+0000","last_clean":"2024-03-26T03:51:12.424252+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:51:12.424252+0000","last_undegraded":"2024-03-26T03:51:12.424252+0000","last_fullsized":"2024-03-26T03:51:12.424252+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:50:20.001688+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:50:20.001688+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.001688+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.152761+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417546+0000","last_change":"2024-03-26T03:50:16.038078+0000","last_active":"2024-03-26T03:51:12.417546+0000","last_peered":"2024-03-26T03:51:12.417546+0000","last_clean":"2024-03-26T03:51:12.417546+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:51:12.417546+0000","last_undegraded":"2024-03-26T03:51:12.417546+0000","last_fullsized":"2024-03-26T03:51:12.417546+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.037958+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.602912+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.439445+0000","last_change":"2024-03-26T03:50:24.093555+0000","last_active":"2024-03-26T03:51:12.439445+0000","last_peered":"2024-03-26T03:51:12.439445+0000","last_clean":"2024-03-26T03:51:12.439445+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:51:12.439445+0000","last_undegraded":"2024-03-26T03:51:12.439445+0000","last_fullsized":"2024-03-26T03:51:12.439445+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:24.093421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:24.093421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:50.167793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420257+0000","last_change":"2024-03-26T03:50:45.594200+0000","last_active":"2024-03-26T03:51:12.420257+0000","last_peered":"2024-03-26T03:51:12.420257+0000","last_clean":"2024-03-26T03:51:12.420257+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:51:12.420257+0000","last_undegraded":"2024-03-26T03:51:12.420257+0000","last_fullsized":"2024-03-26T03:51:12.420257+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:56.749811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:56.749811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.022407+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":134,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.429386+0000","last_change":"2024-03-26T03:50:31.209937+0000","last_active":"2024-03-26T03:51:12.429386+0000","last_peered":"2024-03-26T03:51:12.429386+0000","last_clean":"2024-03-26T03:51:12.429386+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:51:12.429386+0000","last_undegraded":"2024-03-26T03:51:12.429386+0000","last_fullsized":"2024-03-26T03:51:12.429386+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:50:31.209845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:31.209845+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:48.988259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420645+0000","last_change":"2024-03-26T03:50:45.595675+0000","last_active":"2024-03-26T03:51:12.420645+0000","last_peered":"2024-03-26T03:51:12.420645+0000","last_clean":"2024-03-26T03:51:12.420645+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:51:12.420645+0000","last_undegraded":"2024-03-26T03:51:12.420645+0000","last_fullsized":"2024-03-26T03:51:12.420645+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:28.691326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:28.691326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:56.068391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424267+0000","last_change":"2024-03-26T03:50:14.077503+0000","last_active":"2024-03-26T03:51:12.424267+0000","last_peered":"2024-03-26T03:51:12.424267+0000","last_clean":"2024-03-26T03:51:12.424267+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:51:12.424267+0000","last_undegraded":"2024-03-26T03:51:12.424267+0000","last_fullsized":"2024-03-26T03:51:12.424267+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:14.077398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:14.077398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.279309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":28,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426404+0000","last_change":"2024-03-26T03:50:45.593789+0000","last_active":"2024-03-26T03:51:12.426404+0000","last_peered":"2024-03-26T03:51:12.426404+0000","last_clean":"2024-03-26T03:51:12.426404+0000","last_became_active":"2024-03-26T03:50:45.593133+0000","last_became_peered":"2024-03-26T03:50:45.593133+0000","last_unstale":"2024-03-26T03:51:12.426404+0000","last_undegraded":"2024-03-26T03:51:12.426404+0000","last_fullsized":"2024-03-26T03:51:12.426404+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:49:52.564972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:15.598821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":132,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.439671+0000","last_change":"2024-03-26T03:50:24.537221+0000","last_active":"2024-03-26T03:51:12.439671+0000","last_peered":"2024-03-26T03:51:12.439671+0000","last_clean":"2024-03-26T03:51:12.439671+0000","last_became_active":"2024-03-26T03:50:24.531528+0000","last_became_peered":"2024-03-26T03:50:24.531528+0000","last_unstale":"2024-03-26T03:51:12.439671+0000","last_undegraded":"2024-03-26T03:51:12.439671+0000","last_fullsized":"2024-03-26T03:51:12.439671+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:50:02.025987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.025987+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:18.093196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"105'2125","reported_seq":2154,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.156230+0000","last_change":"2024-03-26T03:50:45.595591+0000","last_active":"2024-03-26T03:51:14.156230+0000","last_peered":"2024-03-26T03:51:14.156230+0000","last_clean":"2024-03-26T03:51:14.156230+0000","last_became_active":"2024-03-26T03:50:45.595232+0000","last_became_peered":"2024-03-26T03:50:45.595232+0000","last_unstale":"2024-03-26T03:51:14.156230+0000","last_undegraded":"2024-03-26T03:51:14.156230+0000","last_fullsized":"2024-03-26T03:51:14.156230+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2125,"log_dups_size":0,"ondisk_log_size":2125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:44.911133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139264000,"num_objects":2125,"num_object_clones":0,"num_object_copies":6375,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2125,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2125,"num_write_kb":136000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":100,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:13.568088+0000","last_change":"2024-03-26T03:51:13.568129+0000","last_active":"2024-03-26T03:51:13.568088+0000","last_peered":"2024-03-26T03:51:13.568088+0000","last_clean":"2024-03-26T03:51:13.568088+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:51:13.568088+0000","last_undegraded":"2024-03-26T03:51:13.568088+0000","last_fullsized":"2024-03-26T03:51:13.568088+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:13.568014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:13.568014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:39.170779+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":110,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420886+0000","last_change":"2024-03-26T03:50:04.799748+0000","last_active":"2024-03-26T03:51:12.420886+0000","last_peered":"2024-03-26T03:51:12.420886+0000","last_clean":"2024-03-26T03:51:12.420886+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:51:12.420886+0000","last_undegraded":"2024-03-26T03:51:12.420886+0000","last_fullsized":"2024-03-26T03:51:12.420886+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:50:04.799655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:04.799655+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.356778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":169,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423278+0000","last_change":"2024-03-26T03:50:45.619933+0000","last_active":"2024-03-26T03:51:12.423278+0000","last_peered":"2024-03-26T03:51:12.423278+0000","last_clean":"2024-03-26T03:51:12.423278+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:51:12.423278+0000","last_undegraded":"2024-03-26T03:51:12.423278+0000","last_fullsized":"2024-03-26T03:51:12.423278+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:50:29.915179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:29.915179+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:38.671280+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":139,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420402+0000","last_change":"2024-03-26T03:50:45.606977+0000","last_active":"2024-03-26T03:51:12.420402+0000","last_peered":"2024-03-26T03:51:12.420402+0000","last_clean":"2024-03-26T03:51:12.420402+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:51:12.420402+0000","last_undegraded":"2024-03-26T03:51:12.420402+0000","last_fullsized":"2024-03-26T03:51:12.420402+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:49:54.753680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.753680+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"105'2060","reported_seq":2090,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.177707+0000","last_change":"2024-03-26T03:50:44.592931+0000","last_active":"2024-03-26T03:51:14.177707+0000","last_peered":"2024-03-26T03:51:14.177707+0000","last_clean":"2024-03-26T03:51:14.177707+0000","last_became_active":"2024-03-26T03:50:44.592128+0000","last_became_peered":"2024-03-26T03:50:44.592128+0000","last_unstale":"2024-03-26T03:51:14.177707+0000","last_undegraded":"2024-03-26T03:51:14.177707+0000","last_fullsized":"2024-03-26T03:51:14.177707+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2060,"log_dups_size":0,"ondisk_log_size":2060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:05.497615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135004160,"num_objects":2060,"num_object_clones":0,"num_object_copies":6180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2060,"num_write_kb":131840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424106+0000","last_change":"2024-03-26T03:48:50.043541+0000","last_active":"2024-03-26T03:51:12.424106+0000","last_peered":"2024-03-26T03:51:12.424106+0000","last_clean":"2024-03-26T03:51:12.424106+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:51:12.424106+0000","last_undegraded":"2024-03-26T03:51:12.424106+0000","last_fullsized":"2024-03-26T03:51:12.424106+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420776+0000","last_change":"2024-03-26T03:50:19.811313+0000","last_active":"2024-03-26T03:51:12.420776+0000","last_peered":"2024-03-26T03:51:12.420776+0000","last_clean":"2024-03-26T03:51:12.420776+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:51:12.420776+0000","last_undegraded":"2024-03-26T03:51:12.420776+0000","last_fullsized":"2024-03-26T03:51:12.420776+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:19.811211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:19.811211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:27.929136+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":141,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415554+0000","last_change":"2024-03-26T03:50:13.913444+0000","last_active":"2024-03-26T03:51:12.415554+0000","last_peered":"2024-03-26T03:51:12.415554+0000","last_clean":"2024-03-26T03:51:12.415554+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:51:12.415554+0000","last_undegraded":"2024-03-26T03:51:12.415554+0000","last_fullsized":"2024-03-26T03:51:12.415554+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:50:13.913348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.913348+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:33.729626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420238+0000","last_change":"2024-03-26T03:50:13.104332+0000","last_active":"2024-03-26T03:51:12.420238+0000","last_peered":"2024-03-26T03:51:12.420238+0000","last_clean":"2024-03-26T03:51:12.420238+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:51:12.420238+0000","last_undegraded":"2024-03-26T03:51:12.420238+0000","last_fullsized":"2024-03-26T03:51:12.420238+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:13.104219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.104219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:29.804503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"105'2277","reported_seq":2307,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.170585+0000","last_change":"2024-03-26T03:50:44.586358+0000","last_active":"2024-03-26T03:51:14.170585+0000","last_peered":"2024-03-26T03:51:14.170585+0000","last_clean":"2024-03-26T03:51:14.170585+0000","last_became_active":"2024-03-26T03:50:44.585858+0000","last_became_peered":"2024-03-26T03:50:44.585858+0000","last_unstale":"2024-03-26T03:51:14.170585+0000","last_undegraded":"2024-03-26T03:51:14.170585+0000","last_fullsized":"2024-03-26T03:51:14.170585+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2277,"log_dups_size":0,"ondisk_log_size":2277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:48.984060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149225472,"num_objects":2277,"num_object_clones":0,"num_object_copies":6831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2277,"num_write_kb":145728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420113+0000","last_change":"2024-03-26T03:49:58.807686+0000","last_active":"2024-03-26T03:51:12.420113+0000","last_peered":"2024-03-26T03:51:12.420113+0000","last_clean":"2024-03-26T03:51:12.420113+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:51:12.420113+0000","last_undegraded":"2024-03-26T03:51:12.420113+0000","last_fullsized":"2024-03-26T03:51:12.420113+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:58.807606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:58.807606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.419836+0000","last_change":"2024-03-26T03:50:17.011382+0000","last_active":"2024-03-26T03:51:12.419836+0000","last_peered":"2024-03-26T03:51:12.419836+0000","last_clean":"2024-03-26T03:51:12.419836+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:51:12.419836+0000","last_undegraded":"2024-03-26T03:51:12.419836+0000","last_fullsized":"2024-03-26T03:51:12.419836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:17.011284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:17.011284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:46.537867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420822+0000","last_change":"2024-03-26T03:50:45.595419+0000","last_active":"2024-03-26T03:51:12.420822+0000","last_peered":"2024-03-26T03:51:12.420822+0000","last_clean":"2024-03-26T03:51:12.420822+0000","last_became_active":"2024-03-26T03:50:45.594682+0000","last_became_peered":"2024-03-26T03:50:45.594682+0000","last_unstale":"2024-03-26T03:51:12.420822+0000","last_undegraded":"2024-03-26T03:51:12.420822+0000","last_fullsized":"2024-03-26T03:51:12.420822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:08.854874+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:08.854874+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:36.966586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432000+0000","last_change":"2024-03-26T03:50:45.628592+0000","last_active":"2024-03-26T03:51:12.432000+0000","last_peered":"2024-03-26T03:51:12.432000+0000","last_clean":"2024-03-26T03:51:12.432000+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:51:12.432000+0000","last_undegraded":"2024-03-26T03:51:12.432000+0000","last_fullsized":"2024-03-26T03:51:12.432000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:24.571279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:24.571279+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.462806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"105'2221","reported_seq":2251,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.149852+0000","last_change":"2024-03-26T03:50:44.589891+0000","last_active":"2024-03-26T03:51:14.149852+0000","last_peered":"2024-03-26T03:51:14.149852+0000","last_clean":"2024-03-26T03:51:14.149852+0000","last_became_active":"2024-03-26T03:50:44.589615+0000","last_became_peered":"2024-03-26T03:50:44.589615+0000","last_unstale":"2024-03-26T03:51:14.149852+0000","last_undegraded":"2024-03-26T03:51:14.149852+0000","last_fullsized":"2024-03-26T03:51:14.149852+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2221,"log_dups_size":0,"ondisk_log_size":2221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:57.518059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145555456,"num_objects":2221,"num_object_clones":0,"num_object_copies":6663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2221,"num_write_kb":142144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.428311+0000","last_change":"2024-03-26T03:50:39.104958+0000","last_active":"2024-03-26T03:51:12.428311+0000","last_peered":"2024-03-26T03:51:12.428311+0000","last_clean":"2024-03-26T03:51:12.428311+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:51:12.428311+0000","last_undegraded":"2024-03-26T03:51:12.428311+0000","last_fullsized":"2024-03-26T03:51:12.428311+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:39.104865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:39.104865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:50.826191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417917+0000","last_change":"2024-03-26T03:49:59.255850+0000","last_active":"2024-03-26T03:51:12.417917+0000","last_peered":"2024-03-26T03:51:12.417917+0000","last_clean":"2024-03-26T03:51:12.417917+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:51:12.417917+0000","last_undegraded":"2024-03-26T03:51:12.417917+0000","last_fullsized":"2024-03-26T03:51:12.417917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:49:59.255770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:16.644194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.563055+0000","last_change":"2024-03-26T03:51:12.563077+0000","last_active":"2024-03-26T03:51:12.563055+0000","last_peered":"2024-03-26T03:51:12.563055+0000","last_clean":"2024-03-26T03:51:12.563055+0000","last_became_active":"2024-03-26T03:50:45.592202+0000","last_became_peered":"2024-03-26T03:50:45.592202+0000","last_unstale":"2024-03-26T03:51:12.563055+0000","last_undegraded":"2024-03-26T03:51:12.563055+0000","last_fullsized":"2024-03-26T03:51:12.563055+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:12.562993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:51:12.562993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:30.451054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":140,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427258+0000","last_change":"2024-03-26T03:50:24.536964+0000","last_active":"2024-03-26T03:51:12.427258+0000","last_peered":"2024-03-26T03:51:12.427258+0000","last_clean":"2024-03-26T03:51:12.427258+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:51:12.427258+0000","last_undegraded":"2024-03-26T03:51:12.427258+0000","last_fullsized":"2024-03-26T03:51:12.427258+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:11.191227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.191227+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"105'2155","reported_seq":2181,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.170237+0000","last_change":"2024-03-26T03:50:45.590377+0000","last_active":"2024-03-26T03:51:14.170237+0000","last_peered":"2024-03-26T03:51:14.170237+0000","last_clean":"2024-03-26T03:51:14.170237+0000","last_became_active":"2024-03-26T03:50:45.590035+0000","last_became_peered":"2024-03-26T03:50:45.590035+0000","last_unstale":"2024-03-26T03:51:14.170237+0000","last_undegraded":"2024-03-26T03:51:14.170237+0000","last_fullsized":"2024-03-26T03:51:14.170237+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:01.830862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141230080,"num_objects":2155,"num_object_clones":0,"num_object_copies":6465,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2155,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2155,"num_write_kb":137920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424423+0000","last_change":"2024-03-26T03:48:50.044572+0000","last_active":"2024-03-26T03:51:12.424423+0000","last_peered":"2024-03-26T03:51:12.424423+0000","last_clean":"2024-03-26T03:51:12.424423+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:51:12.424423+0000","last_undegraded":"2024-03-26T03:51:12.424423+0000","last_fullsized":"2024-03-26T03:51:12.424423+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":119,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423515+0000","last_change":"2024-03-26T03:50:45.608620+0000","last_active":"2024-03-26T03:51:12.423515+0000","last_peered":"2024-03-26T03:51:12.423515+0000","last_clean":"2024-03-26T03:51:12.423515+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:51:12.423515+0000","last_undegraded":"2024-03-26T03:51:12.423515+0000","last_fullsized":"2024-03-26T03:51:12.423515+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:50:22.981640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:50:22.981640+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:38.098121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414717+0000","last_change":"2024-03-26T03:50:45.594223+0000","last_active":"2024-03-26T03:51:12.414717+0000","last_peered":"2024-03-26T03:51:12.414717+0000","last_clean":"2024-03-26T03:51:12.414717+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:51:12.414717+0000","last_undegraded":"2024-03-26T03:51:12.414717+0000","last_fullsized":"2024-03-26T03:51:12.414717+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:50:22.903348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:49.281550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":184,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432025+0000","last_change":"2024-03-26T03:50:13.007007+0000","last_active":"2024-03-26T03:51:12.432025+0000","last_peered":"2024-03-26T03:51:12.432025+0000","last_clean":"2024-03-26T03:51:12.432025+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:51:12.432025+0000","last_undegraded":"2024-03-26T03:51:12.432025+0000","last_fullsized":"2024-03-26T03:51:12.432025+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:50:13.006886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.006886+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.897597+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.428555+0000","last_change":"2024-03-26T03:50:14.195816+0000","last_active":"2024-03-26T03:51:12.428555+0000","last_peered":"2024-03-26T03:51:12.428555+0000","last_clean":"2024-03-26T03:51:12.428555+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:51:12.428555+0000","last_undegraded":"2024-03-26T03:51:12.428555+0000","last_fullsized":"2024-03-26T03:51:12.428555+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:14.195713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:14.195713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.803072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"105'2147","reported_seq":2177,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.157440+0000","last_change":"2024-03-26T03:50:44.585186+0000","last_active":"2024-03-26T03:51:14.157440+0000","last_peered":"2024-03-26T03:51:14.157440+0000","last_clean":"2024-03-26T03:51:14.157440+0000","last_became_active":"2024-03-26T03:50:44.584865+0000","last_became_peered":"2024-03-26T03:50:44.584865+0000","last_unstale":"2024-03-26T03:51:14.157440+0000","last_undegraded":"2024-03-26T03:51:14.157440+0000","last_fullsized":"2024-03-26T03:51:14.157440+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2147,"log_dups_size":0,"ondisk_log_size":2147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:48.255633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140705792,"num_objects":2147,"num_object_clones":0,"num_object_copies":6441,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2147,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":137408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420935+0000","last_change":"2024-03-26T03:50:45.584445+0000","last_active":"2024-03-26T03:51:12.420935+0000","last_peered":"2024-03-26T03:51:12.420935+0000","last_clean":"2024-03-26T03:51:12.420935+0000","last_became_active":"2024-03-26T03:50:45.583936+0000","last_became_peered":"2024-03-26T03:50:45.583936+0000","last_unstale":"2024-03-26T03:51:12.420935+0000","last_undegraded":"2024-03-26T03:51:12.420935+0000","last_fullsized":"2024-03-26T03:51:12.420935+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:11.829105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.829105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:37.326238+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":103,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.431062+0000","last_change":"2024-03-26T03:50:45.591776+0000","last_active":"2024-03-26T03:51:12.431062+0000","last_peered":"2024-03-26T03:51:12.431062+0000","last_clean":"2024-03-26T03:51:12.431062+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:51:12.431062+0000","last_undegraded":"2024-03-26T03:51:12.431062+0000","last_fullsized":"2024-03-26T03:51:12.431062+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:05.944040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.944040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.099077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417021+0000","last_change":"2024-03-26T03:49:48.295417+0000","last_active":"2024-03-26T03:51:12.417021+0000","last_peered":"2024-03-26T03:51:12.417021+0000","last_clean":"2024-03-26T03:51:12.417021+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:51:12.417021+0000","last_undegraded":"2024-03-26T03:51:12.417021+0000","last_fullsized":"2024-03-26T03:51:12.417021+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.295312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.295312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:15.017876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":132,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.433210+0000","last_change":"2024-03-26T03:50:13.971916+0000","last_active":"2024-03-26T03:51:12.433210+0000","last_peered":"2024-03-26T03:51:12.433210+0000","last_clean":"2024-03-26T03:51:12.433210+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:51:12.433210+0000","last_undegraded":"2024-03-26T03:51:12.433210+0000","last_fullsized":"2024-03-26T03:51:12.433210+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:50:13.971792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:13.971792+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:34.626065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.418571+0000","last_change":"2024-03-26T03:50:20.896944+0000","last_active":"2024-03-26T03:51:12.418571+0000","last_peered":"2024-03-26T03:51:12.418571+0000","last_clean":"2024-03-26T03:51:12.418571+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:51:12.418571+0000","last_undegraded":"2024-03-26T03:51:12.418571+0000","last_fullsized":"2024-03-26T03:51:12.418571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:20.896856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.896856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:37.752730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"105'2187","reported_seq":2217,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.150755+0000","last_change":"2024-03-26T03:50:44.588711+0000","last_active":"2024-03-26T03:51:14.150755+0000","last_peered":"2024-03-26T03:51:14.150755+0000","last_clean":"2024-03-26T03:51:14.150755+0000","last_became_active":"2024-03-26T03:50:44.588356+0000","last_became_peered":"2024-03-26T03:50:44.588356+0000","last_unstale":"2024-03-26T03:51:14.150755+0000","last_undegraded":"2024-03-26T03:51:14.150755+0000","last_fullsized":"2024-03-26T03:51:14.150755+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2187,"log_dups_size":0,"ondisk_log_size":2187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:02.102519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143327232,"num_objects":2187,"num_object_clones":0,"num_object_copies":6561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2187,"num_write_kb":139968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423943+0000","last_change":"2024-03-26T03:50:17.931877+0000","last_active":"2024-03-26T03:51:12.423943+0000","last_peered":"2024-03-26T03:51:12.423943+0000","last_clean":"2024-03-26T03:51:12.423943+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:51:12.423943+0000","last_undegraded":"2024-03-26T03:51:12.423943+0000","last_fullsized":"2024-03-26T03:51:12.423943+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:17.931766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:17.931766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:24.013525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":361,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420202+0000","last_change":"2024-03-26T03:50:48.723614+0000","last_active":"2024-03-26T03:51:12.420202+0000","last_peered":"2024-03-26T03:51:12.420202+0000","last_clean":"2024-03-26T03:51:12.420202+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:51:12.420202+0000","last_undegraded":"2024-03-26T03:51:12.420202+0000","last_fullsized":"2024-03-26T03:51:12.420202+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:50:36.776668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:50:36.776668+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:44.825169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426573+0000","last_change":"2024-03-26T03:50:45.592650+0000","last_active":"2024-03-26T03:51:12.426573+0000","last_peered":"2024-03-26T03:51:12.426573+0000","last_clean":"2024-03-26T03:51:12.426573+0000","last_became_active":"2024-03-26T03:50:45.592343+0000","last_became_peered":"2024-03-26T03:50:45.592343+0000","last_unstale":"2024-03-26T03:51:12.426573+0000","last_undegraded":"2024-03-26T03:51:12.426573+0000","last_fullsized":"2024-03-26T03:51:12.426573+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415479+0000","last_change":"2024-03-26T03:50:45.615243+0000","last_active":"2024-03-26T03:51:12.415479+0000","last_peered":"2024-03-26T03:51:12.415479+0000","last_clean":"2024-03-26T03:51:12.415479+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:51:12.415479+0000","last_undegraded":"2024-03-26T03:51:12.415479+0000","last_fullsized":"2024-03-26T03:51:12.415479+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:50:06.585054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.585054+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:31.752895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420663+0000","last_change":"2024-03-26T03:50:00.772766+0000","last_active":"2024-03-26T03:51:12.420663+0000","last_peered":"2024-03-26T03:51:12.420663+0000","last_clean":"2024-03-26T03:51:12.420663+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:51:12.420663+0000","last_undegraded":"2024-03-26T03:51:12.420663+0000","last_fullsized":"2024-03-26T03:51:12.420663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.772625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.974412+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"105'2214","reported_seq":2243,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.161714+0000","last_change":"2024-03-26T03:50:44.586811+0000","last_active":"2024-03-26T03:51:14.161714+0000","last_peered":"2024-03-26T03:51:14.161714+0000","last_clean":"2024-03-26T03:51:14.161714+0000","last_became_active":"2024-03-26T03:50:44.586504+0000","last_became_peered":"2024-03-26T03:50:44.586504+0000","last_unstale":"2024-03-26T03:51:14.161714+0000","last_undegraded":"2024-03-26T03:51:14.161714+0000","last_fullsized":"2024-03-26T03:51:14.161714+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:05.437934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145096704,"num_objects":2214,"num_object_clones":0,"num_object_copies":6642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2214,"num_write_kb":141696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":85,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.431639+0000","last_change":"2024-03-26T03:50:37.125164+0000","last_active":"2024-03-26T03:51:12.431639+0000","last_peered":"2024-03-26T03:51:12.431639+0000","last_clean":"2024-03-26T03:51:12.431639+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:51:12.431639+0000","last_undegraded":"2024-03-26T03:51:12.431639+0000","last_fullsized":"2024-03-26T03:51:12.431639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:37.125085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:37.125085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:57.979773+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.428009+0000","last_change":"2024-03-26T03:50:18.248516+0000","last_active":"2024-03-26T03:51:12.428009+0000","last_peered":"2024-03-26T03:51:12.428009+0000","last_clean":"2024-03-26T03:51:12.428009+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:51:12.428009+0000","last_undegraded":"2024-03-26T03:51:12.428009+0000","last_fullsized":"2024-03-26T03:51:12.428009+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:18.248434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.248434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.519794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":253,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:13.011194+0000","last_change":"2024-03-26T03:49:45.618585+0000","last_active":"2024-03-26T03:51:13.011194+0000","last_peered":"2024-03-26T03:51:13.011194+0000","last_clean":"2024-03-26T03:51:13.011194+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:51:13.011194+0000","last_undegraded":"2024-03-26T03:51:13.011194+0000","last_fullsized":"2024-03-26T03:51:13.011194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:49:45.618470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:49:45.618470+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:14.022391+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.428975+0000","last_change":"2024-03-26T03:50:20.200717+0000","last_active":"2024-03-26T03:51:12.428975+0000","last_peered":"2024-03-26T03:51:12.428975+0000","last_clean":"2024-03-26T03:51:12.428975+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:51:12.428975+0000","last_undegraded":"2024-03-26T03:51:12.428975+0000","last_fullsized":"2024-03-26T03:51:12.428975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:50:20.200603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.200603+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:45.989520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424188+0000","last_change":"2024-03-26T03:50:18.981723+0000","last_active":"2024-03-26T03:51:12.424188+0000","last_peered":"2024-03-26T03:51:12.424188+0000","last_clean":"2024-03-26T03:51:12.424188+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:51:12.424188+0000","last_undegraded":"2024-03-26T03:51:12.424188+0000","last_fullsized":"2024-03-26T03:51:12.424188+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:18.981529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:18.981529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:41.258512+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"105'2250","reported_seq":2280,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.123147+0000","last_change":"2024-03-26T03:50:44.594189+0000","last_active":"2024-03-26T03:51:14.123147+0000","last_peered":"2024-03-26T03:51:14.123147+0000","last_clean":"2024-03-26T03:51:14.123147+0000","last_became_active":"2024-03-26T03:50:44.593854+0000","last_became_peered":"2024-03-26T03:50:44.593854+0000","last_unstale":"2024-03-26T03:51:14.123147+0000","last_undegraded":"2024-03-26T03:51:14.123147+0000","last_fullsized":"2024-03-26T03:51:14.123147+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2250,"log_dups_size":0,"ondisk_log_size":2250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:51.591379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147456000,"num_objects":2250,"num_object_clones":0,"num_object_copies":6750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2250,"num_write_kb":144000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":91,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.437525+0000","last_change":"2024-03-26T03:50:45.586477+0000","last_active":"2024-03-26T03:51:12.437525+0000","last_peered":"2024-03-26T03:51:12.437525+0000","last_clean":"2024-03-26T03:51:12.437525+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:51:12.437525+0000","last_undegraded":"2024-03-26T03:51:12.437525+0000","last_fullsized":"2024-03-26T03:51:12.437525+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:22.076195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:22.076195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:51.672317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":97,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.431883+0000","last_change":"2024-03-26T03:50:00.032886+0000","last_active":"2024-03-26T03:51:12.431883+0000","last_peered":"2024-03-26T03:51:12.431883+0000","last_clean":"2024-03-26T03:51:12.431883+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:51:12.431883+0000","last_undegraded":"2024-03-26T03:51:12.431883+0000","last_fullsized":"2024-03-26T03:51:12.431883+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:00.032763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:00.032763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":142,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.427120+0000","last_change":"2024-03-26T03:50:03.113111+0000","last_active":"2024-03-26T03:51:12.427120+0000","last_peered":"2024-03-26T03:51:12.427120+0000","last_clean":"2024-03-26T03:51:12.427120+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:51:12.427120+0000","last_undegraded":"2024-03-26T03:51:12.427120+0000","last_fullsized":"2024-03-26T03:51:12.427120+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:50:03.112983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.112983+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":28,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426403+0000","last_change":"2024-03-26T03:50:45.593522+0000","last_active":"2024-03-26T03:51:12.426403+0000","last_peered":"2024-03-26T03:51:12.426403+0000","last_clean":"2024-03-26T03:51:12.426403+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:51:12.426403+0000","last_undegraded":"2024-03-26T03:51:12.426403+0000","last_fullsized":"2024-03-26T03:51:12.426403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:54.590915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:49:54.590915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:23.871802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"105'2179","reported_seq":2208,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.161823+0000","last_change":"2024-03-26T03:50:45.597482+0000","last_active":"2024-03-26T03:51:14.161823+0000","last_peered":"2024-03-26T03:51:14.161823+0000","last_clean":"2024-03-26T03:51:14.161823+0000","last_became_active":"2024-03-26T03:50:45.596978+0000","last_became_peered":"2024-03-26T03:50:45.596978+0000","last_unstale":"2024-03-26T03:51:14.161823+0000","last_undegraded":"2024-03-26T03:51:14.161823+0000","last_fullsized":"2024-03-26T03:51:14.161823+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:56.536736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142802944,"num_objects":2179,"num_object_clones":0,"num_object_copies":6537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":139456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426164+0000","last_change":"2024-03-26T03:48:50.045296+0000","last_active":"2024-03-26T03:51:12.426164+0000","last_peered":"2024-03-26T03:51:12.426164+0000","last_clean":"2024-03-26T03:51:12.426164+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:51:12.426164+0000","last_undegraded":"2024-03-26T03:51:12.426164+0000","last_fullsized":"2024-03-26T03:51:12.426164+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424121+0000","last_change":"2024-03-26T03:50:21.036093+0000","last_active":"2024-03-26T03:51:12.424121+0000","last_peered":"2024-03-26T03:51:12.424121+0000","last_clean":"2024-03-26T03:51:12.424121+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:51:12.424121+0000","last_undegraded":"2024-03-26T03:51:12.424121+0000","last_fullsized":"2024-03-26T03:51:12.424121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:21.036015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.036015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:50.546522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":120,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432235+0000","last_change":"2024-03-26T03:49:46.927333+0000","last_active":"2024-03-26T03:51:12.432235+0000","last_peered":"2024-03-26T03:51:12.432235+0000","last_clean":"2024-03-26T03:51:12.432235+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:51:12.432235+0000","last_undegraded":"2024-03-26T03:51:12.432235+0000","last_fullsized":"2024-03-26T03:51:12.432235+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:49:46.927230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:49:46.927230+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":131,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.426937+0000","last_change":"2024-03-26T03:50:45.614456+0000","last_active":"2024-03-26T03:51:12.426937+0000","last_peered":"2024-03-26T03:51:12.426937+0000","last_clean":"2024-03-26T03:51:12.426937+0000","last_became_active":"2024-03-26T03:50:45.602990+0000","last_became_peered":"2024-03-26T03:50:45.602990+0000","last_unstale":"2024-03-26T03:51:12.426937+0000","last_undegraded":"2024-03-26T03:51:12.426937+0000","last_fullsized":"2024-03-26T03:51:12.426937+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:48:47.006138+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"105'2192","reported_seq":2222,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.177350+0000","last_change":"2024-03-26T03:50:44.586742+0000","last_active":"2024-03-26T03:51:14.177350+0000","last_peered":"2024-03-26T03:51:14.177350+0000","last_clean":"2024-03-26T03:51:14.177350+0000","last_became_active":"2024-03-26T03:50:44.586424+0000","last_became_peered":"2024-03-26T03:50:44.586424+0000","last_unstale":"2024-03-26T03:51:14.177350+0000","last_undegraded":"2024-03-26T03:51:14.177350+0000","last_fullsized":"2024-03-26T03:51:14.177350+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:53.050397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143654912,"num_objects":2192,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2192,"num_write_kb":140288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":92,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.420574+0000","last_change":"2024-03-26T03:50:45.590037+0000","last_active":"2024-03-26T03:51:12.420574+0000","last_peered":"2024-03-26T03:51:12.420574+0000","last_clean":"2024-03-26T03:51:12.420574+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:51:12.420574+0000","last_undegraded":"2024-03-26T03:51:12.420574+0000","last_fullsized":"2024-03-26T03:51:12.420574+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:57.800113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:49:57.800113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.763204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":99,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.429033+0000","last_change":"2024-03-26T03:50:02.092039+0000","last_active":"2024-03-26T03:51:12.429033+0000","last_peered":"2024-03-26T03:51:12.429033+0000","last_clean":"2024-03-26T03:51:12.429033+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:51:12.429033+0000","last_undegraded":"2024-03-26T03:51:12.429033+0000","last_fullsized":"2024-03-26T03:51:12.429033+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:02.091909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.091909+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":151,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415452+0000","last_change":"2024-03-26T03:50:45.616485+0000","last_active":"2024-03-26T03:51:12.415452+0000","last_peered":"2024-03-26T03:51:12.415452+0000","last_clean":"2024-03-26T03:51:12.415452+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:51:12.415452+0000","last_undegraded":"2024-03-26T03:51:12.415452+0000","last_fullsized":"2024-03-26T03:51:12.415452+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:50:06.823723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:06.823723+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:19.457392+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424672+0000","last_change":"2024-03-26T03:50:10.088926+0000","last_active":"2024-03-26T03:51:12.424672+0000","last_peered":"2024-03-26T03:51:12.424672+0000","last_clean":"2024-03-26T03:51:12.424672+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:51:12.424672+0000","last_undegraded":"2024-03-26T03:51:12.424672+0000","last_fullsized":"2024-03-26T03:51:12.424672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:10.088836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:10.088836+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.361285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"105'2173","reported_seq":2203,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.157551+0000","last_change":"2024-03-26T03:50:44.593372+0000","last_active":"2024-03-26T03:51:14.157551+0000","last_peered":"2024-03-26T03:51:14.157551+0000","last_clean":"2024-03-26T03:51:14.157551+0000","last_became_active":"2024-03-26T03:50:44.592595+0000","last_became_peered":"2024-03-26T03:50:44.592595+0000","last_unstale":"2024-03-26T03:51:14.157551+0000","last_undegraded":"2024-03-26T03:51:14.157551+0000","last_fullsized":"2024-03-26T03:51:14.157551+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2173,"log_dups_size":0,"ondisk_log_size":2173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:55.681010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142409728,"num_objects":2173,"num_object_clones":0,"num_object_copies":6519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2173,"num_write_kb":139072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":78,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414712+0000","last_change":"2024-03-26T03:48:50.043708+0000","last_active":"2024-03-26T03:51:12.414712+0000","last_peered":"2024-03-26T03:51:12.414712+0000","last_clean":"2024-03-26T03:51:12.414712+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:51:12.414712+0000","last_undegraded":"2024-03-26T03:51:12.414712+0000","last_fullsized":"2024-03-26T03:51:12.414712+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.321546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.439195+0000","last_change":"2024-03-26T03:50:22.092536+0000","last_active":"2024-03-26T03:51:12.439195+0000","last_peered":"2024-03-26T03:51:12.439195+0000","last_clean":"2024-03-26T03:51:12.439195+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:51:12.439195+0000","last_undegraded":"2024-03-26T03:51:12.439195+0000","last_fullsized":"2024-03-26T03:51:12.439195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:22.092433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:22.092433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:39.989653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":160,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423716+0000","last_change":"2024-03-26T03:49:51.111438+0000","last_active":"2024-03-26T03:51:12.423716+0000","last_peered":"2024-03-26T03:51:12.423716+0000","last_clean":"2024-03-26T03:51:12.423716+0000","last_became_active":"2024-03-26T03:48:46.209319+0000","last_became_peered":"2024-03-26T03:48:46.209319+0000","last_unstale":"2024-03-26T03:51:12.423716+0000","last_undegraded":"2024-03-26T03:51:12.423716+0000","last_fullsized":"2024-03-26T03:51:12.423716+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.111316+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.311163+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414658+0000","last_change":"2024-03-26T03:50:16.977948+0000","last_active":"2024-03-26T03:51:12.414658+0000","last_peered":"2024-03-26T03:51:12.414658+0000","last_clean":"2024-03-26T03:51:12.414658+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:51:12.414658+0000","last_undegraded":"2024-03-26T03:51:12.414658+0000","last_fullsized":"2024-03-26T03:51:12.414658+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:50:16.977862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:35.490601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"105'2184","reported_seq":2212,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.190263+0000","last_change":"2024-03-26T03:50:45.583826+0000","last_active":"2024-03-26T03:51:14.190263+0000","last_peered":"2024-03-26T03:51:14.190263+0000","last_clean":"2024-03-26T03:51:14.190263+0000","last_became_active":"2024-03-26T03:50:45.583350+0000","last_became_peered":"2024-03-26T03:50:45.583350+0000","last_unstale":"2024-03-26T03:51:14.190263+0000","last_undegraded":"2024-03-26T03:51:14.190263+0000","last_fullsized":"2024-03-26T03:51:14.190263+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2184,"log_dups_size":0,"ondisk_log_size":2184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:09.613801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143130624,"num_objects":2184,"num_object_clones":0,"num_object_copies":6552,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2184,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2184,"num_write_kb":139776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423830+0000","last_change":"2024-03-26T03:50:21.989706+0000","last_active":"2024-03-26T03:51:12.423830+0000","last_peered":"2024-03-26T03:51:12.423830+0000","last_clean":"2024-03-26T03:51:12.423830+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:51:12.423830+0000","last_undegraded":"2024-03-26T03:51:12.423830+0000","last_fullsized":"2024-03-26T03:51:12.423830+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:21.989636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.989636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.038618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.416212+0000","last_change":"2024-03-26T03:50:05.869446+0000","last_active":"2024-03-26T03:51:12.416212+0000","last_peered":"2024-03-26T03:51:12.416212+0000","last_clean":"2024-03-26T03:51:12.416212+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:51:12.416212+0000","last_undegraded":"2024-03-26T03:51:12.416212+0000","last_fullsized":"2024-03-26T03:51:12.416212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:05.869359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:05.869359+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.254418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423617+0000","last_change":"2024-03-26T03:48:44.017015+0000","last_active":"2024-03-26T03:51:12.423617+0000","last_peered":"2024-03-26T03:51:12.423617+0000","last_clean":"2024-03-26T03:51:12.423617+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:51:12.423617+0000","last_undegraded":"2024-03-26T03:51:12.423617+0000","last_fullsized":"2024-03-26T03:51:12.423617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":118,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.433110+0000","last_change":"2024-03-26T03:50:09.997564+0000","last_active":"2024-03-26T03:51:12.433110+0000","last_peered":"2024-03-26T03:51:12.433110+0000","last_clean":"2024-03-26T03:51:12.433110+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:51:12.433110+0000","last_undegraded":"2024-03-26T03:51:12.433110+0000","last_fullsized":"2024-03-26T03:51:12.433110+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:50:09.997449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:09.997449+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:28.384605+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"105'2150","reported_seq":2178,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.141962+0000","last_change":"2024-03-26T03:50:45.591630+0000","last_active":"2024-03-26T03:51:14.141962+0000","last_peered":"2024-03-26T03:51:14.141962+0000","last_clean":"2024-03-26T03:51:14.141962+0000","last_became_active":"2024-03-26T03:50:45.591306+0000","last_became_peered":"2024-03-26T03:50:45.591306+0000","last_unstale":"2024-03-26T03:51:14.141962+0000","last_undegraded":"2024-03-26T03:51:14.141962+0000","last_fullsized":"2024-03-26T03:51:14.141962+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2150,"log_dups_size":0,"ondisk_log_size":2150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:04.071310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":140902400,"num_objects":2150,"num_object_clones":0,"num_object_copies":6450,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2150,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2150,"num_write_kb":137600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.435177+0000","last_change":"2024-03-26T03:50:45.595338+0000","last_active":"2024-03-26T03:51:12.435177+0000","last_peered":"2024-03-26T03:51:12.435177+0000","last_clean":"2024-03-26T03:51:12.435177+0000","last_became_active":"2024-03-26T03:50:45.595009+0000","last_became_peered":"2024-03-26T03:50:45.595009+0000","last_unstale":"2024-03-26T03:51:12.435177+0000","last_undegraded":"2024-03-26T03:51:12.435177+0000","last_fullsized":"2024-03-26T03:51:12.435177+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:48:49.010280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.424093+0000","last_change":"2024-03-26T03:50:45.595938+0000","last_active":"2024-03-26T03:51:12.424093+0000","last_peered":"2024-03-26T03:51:12.424093+0000","last_clean":"2024-03-26T03:51:12.424093+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:51:12.424093+0000","last_undegraded":"2024-03-26T03:51:12.424093+0000","last_fullsized":"2024-03-26T03:51:12.424093+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:02.188975+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.188975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.697455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414677+0000","last_change":"2024-03-26T03:49:48.865134+0000","last_active":"2024-03-26T03:51:12.414677+0000","last_peered":"2024-03-26T03:51:12.414677+0000","last_clean":"2024-03-26T03:51:12.414677+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:51:12.414677+0000","last_undegraded":"2024-03-26T03:51:12.414677+0000","last_fullsized":"2024-03-26T03:51:12.414677+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:49:48.864997+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.321601+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":145,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415974+0000","last_change":"2024-03-26T03:50:45.624031+0000","last_active":"2024-03-26T03:51:12.415974+0000","last_peered":"2024-03-26T03:51:12.415974+0000","last_clean":"2024-03-26T03:51:12.415974+0000","last_became_active":"2024-03-26T03:50:45.596131+0000","last_became_peered":"2024-03-26T03:50:45.596131+0000","last_unstale":"2024-03-26T03:51:12.415974+0000","last_undegraded":"2024-03-26T03:51:12.415974+0000","last_fullsized":"2024-03-26T03:51:12.415974+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:11.925762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:11.925762+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:25.897606+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"105'2111","reported_seq":2141,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.175819+0000","last_change":"2024-03-26T03:50:44.591236+0000","last_active":"2024-03-26T03:51:14.175819+0000","last_peered":"2024-03-26T03:51:14.175819+0000","last_clean":"2024-03-26T03:51:14.175819+0000","last_became_active":"2024-03-26T03:50:44.590889+0000","last_became_peered":"2024-03-26T03:50:44.590889+0000","last_unstale":"2024-03-26T03:51:14.175819+0000","last_undegraded":"2024-03-26T03:51:14.175819+0000","last_fullsized":"2024-03-26T03:51:14.175819+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2111,"log_dups_size":0,"ondisk_log_size":2111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:06.737640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138346496,"num_objects":2111,"num_object_clones":0,"num_object_copies":6333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2111,"num_write_kb":135104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417647+0000","last_change":"2024-03-26T03:50:07.213828+0000","last_active":"2024-03-26T03:51:12.417647+0000","last_peered":"2024-03-26T03:51:12.417647+0000","last_clean":"2024-03-26T03:51:12.417647+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:51:12.417647+0000","last_undegraded":"2024-03-26T03:51:12.417647+0000","last_fullsized":"2024-03-26T03:51:12.417647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:07.213705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:07.213705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:19.297785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":89,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.432137+0000","last_change":"2024-03-26T03:50:12.050853+0000","last_active":"2024-03-26T03:51:12.432137+0000","last_peered":"2024-03-26T03:51:12.432137+0000","last_clean":"2024-03-26T03:51:12.432137+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:51:12.432137+0000","last_undegraded":"2024-03-26T03:51:12.432137+0000","last_fullsized":"2024-03-26T03:51:12.432137+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:12.050772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:12.050772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:32.794737+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":28,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.414777+0000","last_change":"2024-03-26T03:50:45.595904+0000","last_active":"2024-03-26T03:51:12.414777+0000","last_peered":"2024-03-26T03:51:12.414777+0000","last_clean":"2024-03-26T03:51:12.414777+0000","last_became_active":"2024-03-26T03:50:45.593583+0000","last_became_peered":"2024-03-26T03:50:45.593583+0000","last_unstale":"2024-03-26T03:51:12.414777+0000","last_undegraded":"2024-03-26T03:51:12.414777+0000","last_fullsized":"2024-03-26T03:51:12.414777+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:49:51.516891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:49:51.516891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:17.321574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":122,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415762+0000","last_change":"2024-03-26T03:50:03.880855+0000","last_active":"2024-03-26T03:51:12.415762+0000","last_peered":"2024-03-26T03:51:12.415762+0000","last_clean":"2024-03-26T03:51:12.415762+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:51:12.415762+0000","last_undegraded":"2024-03-26T03:51:12.415762+0000","last_fullsized":"2024-03-26T03:51:12.415762+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:50:03.880711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:03.880711+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:26.791446+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"105'2171","reported_seq":2200,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:14.169955+0000","last_change":"2024-03-26T03:50:44.587761+0000","last_active":"2024-03-26T03:51:14.169955+0000","last_peered":"2024-03-26T03:51:14.169955+0000","last_clean":"2024-03-26T03:51:14.169955+0000","last_became_active":"2024-03-26T03:50:44.587505+0000","last_became_peered":"2024-03-26T03:50:44.587505+0000","last_unstale":"2024-03-26T03:51:14.169955+0000","last_undegraded":"2024-03-26T03:51:14.169955+0000","last_fullsized":"2024-03-26T03:51:14.169955+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":2171,"log_dups_size":0,"ondisk_log_size":2171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:52:01.428135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142278656,"num_objects":2171,"num_object_clones":0,"num_object_copies":6513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2171,"num_write_kb":138944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.416265+0000","last_change":"2024-03-26T03:50:02.877487+0000","last_active":"2024-03-26T03:51:12.416265+0000","last_peered":"2024-03-26T03:51:12.416265+0000","last_clean":"2024-03-26T03:51:12.416265+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:51:12.416265+0000","last_undegraded":"2024-03-26T03:51:12.416265+0000","last_fullsized":"2024-03-26T03:51:12.416265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:02.877373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.877373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:30.008619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":129,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.417076+0000","last_change":"2024-03-26T03:50:02.300700+0000","last_active":"2024-03-26T03:51:12.417076+0000","last_peered":"2024-03-26T03:51:12.417076+0000","last_clean":"2024-03-26T03:51:12.417076+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:51:12.417076+0000","last_undegraded":"2024-03-26T03:51:12.417076+0000","last_fullsized":"2024-03-26T03:51:12.417076+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:50:02.300594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:02.300594+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:21.021817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":90,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.423826+0000","last_change":"2024-03-26T03:48:44.015913+0000","last_active":"2024-03-26T03:51:12.423826+0000","last_peered":"2024-03-26T03:51:12.423826+0000","last_clean":"2024-03-26T03:51:12.423826+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:51:12.423826+0000","last_undegraded":"2024-03-26T03:51:12.423826+0000","last_fullsized":"2024-03-26T03:51:12.423826+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:48:42.999718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:13.310980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":127,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-26T03:51:12.415212+0000","last_change":"2024-03-26T03:50:45.629456+0000","last_active":"2024-03-26T03:51:12.415212+0000","last_peered":"2024-03-26T03:51:12.415212+0000","last_clean":"2024-03-26T03:51:12.415212+0000","last_became_active":"2024-03-26T03:50:45.598403+0000","last_became_peered":"2024-03-26T03:50:45.598403+0000","last_unstale":"2024-03-26T03:51:12.415212+0000","last_undegraded":"2024-03-26T03:51:12.415212+0000","last_fullsized":"2024-03-26T03:51:12.415212+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:50:33.010493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:33.010493+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:51:33.955311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":2280390656,"num_objects":34796,"num_object_clones":0,"num_object_copies":104388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34796,"num_write_kb":2226944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6840909824,"data_stored":6840909824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34796,"ondisk_log_size":34796,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":143,"num_read_kb":138,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":1144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":611,"ondisk_log_size":611,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070498,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":737244,"kb_used_data":697748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93007908,"statfs":{"total":95995035648,"available":95240097792,"internally_reserved":0,"allocated":714493952,"data_stored":714070694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171380,"kb_used_data":1117996,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92573772,"statfs":{"total":95995035648,"available":94795542528,"internally_reserved":0,"allocated":1144827904,"data_stored":1144379223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1122668,"kb_used_data":1069220,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92622484,"statfs":{"total":95995035648,"available":94845423616,"internally_reserved":0,"allocated":1094881280,"data_stored":1094441760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411388,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154812,"kb_used_data":1101428,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92590340,"statfs":{"total":95995035648,"available":94812508160,"internally_reserved":0,"allocated":1127862272,"data_stored":1127469064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182454,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1399596,"kb_used_data":1342052,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":92345556,"statfs":{"total":95995035648,"available":94561849344,"internally_reserved":0,"allocated":1374261248,"data_stored":1373821108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":58915271},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953521,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717028,"kb_used_data":677524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93028124,"statfs":{"total":95995035648,"available":95260798976,"internally_reserved":0,"allocated":693784576,"data_stored":693359090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659179,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":559636,"kb_used_data":532612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93185516,"statfs":{"total":95995035648,"available":95421968384,"internally_reserved":0,"allocated":545394688,"data_stored":544985624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1141112832,"data_stored":1141112832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1145438208,"data_stored":1145438208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1132265472,"data_stored":1132265472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1422721024,"data_stored":1422721024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":720502784,"data_stored":720502784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":559742976,"data_stored":559742976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":719126528,"data_stored":719126528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:51:25.178 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-26T03:51:25.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:25.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:25.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 29 16 49476 49460 106.574 81.6875 0.0134174 0.00937825 2024-03-26T03:51:25.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:25.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:25.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:25.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:25.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:25.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:25.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:25.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: osdmap e111: 8 total, 8 up, 7 in 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: osdmap e111: 8 total, 8 up, 7 in 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T03:51:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:25 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T03:51:25.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:25.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:25.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:25.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:25.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:25.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:26.031 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:26.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: osdmap e111: 8 total, 8 up, 7 in 2024-03-26T03:51:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T03:51:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T03:51:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T03:51:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:25 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T03:51:26.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:26.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:26.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:26.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:26.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 30 16 50827 50811 105.834 84.4375 0.00958549 0.00944188 2024-03-26T03:51:26.309 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:26.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:26.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:26.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:26.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:26.583 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:26.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:26.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:26.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:26.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:26.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: pgmap v97: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: 6.6 scrub starts 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: 6.6 scrub ok 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:51:26.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: pgmap v97: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: 6.6 scrub starts 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: 6.6 scrub ok 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:51:26.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:26 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:51:26.910 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:26.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.001 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:27.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: pgmap v97: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T03:51:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: 6.6 scrub starts 2024-03-26T03:51:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: 6.6 scrub ok 2024-03-26T03:51:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:51:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:51:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:26 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:51:27.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:27.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:27.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:27.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 31 16 52108 52092 105.002 80.0625 0.018656 0.00951802 2024-03-26T03:51:27.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:27.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:27.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:27.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:27.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:27.564 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:27.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:27.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:27.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:27.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:27.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:27.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:27.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[37501]: 3.8 scrub starts 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[37501]: 3.8 scrub ok 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[29446]: 3.8 scrub starts 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[29446]: 3.8 scrub ok 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T03:51:27.928 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:27 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T03:51:27.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:27.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:28.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:28.052 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:51:28.052 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:51:28.052 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:51:28.053 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:51:28.053 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:51:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:27 smithi086 ceph-mon[32965]: 3.8 scrub starts 2024-03-26T03:51:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:27 smithi086 ceph-mon[32965]: 3.8 scrub ok 2024-03-26T03:51:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:27 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T03:51:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:27 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T03:51:28.092 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:28.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:28.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:28.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:28.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 32 16 53326 53310 104.1 76.125 0.0310482 0.00959834 2024-03-26T03:51:28.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:28.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:28.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:28.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:28.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:28.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:28.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:28.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[29446]: pgmap v98: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[37501]: pgmap v98: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T03:51:28.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:28 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T03:51:28.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:28.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:28.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:28.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:28.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:28 smithi086 ceph-mon[32965]: pgmap v98: 148 pgs: 1 peering, 147 active+clean; 2.9 GiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 852 B/s rd, 61 MiB/s wr, 977 op/s 2024-03-26T03:51:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:28 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T03:51:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:28 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T03:51:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:28 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T03:51:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:28 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T03:51:28.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:29.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.046 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:29.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:29.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:29.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:29.200 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:51:29.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 33 16 54523 54507 103.212 74.8125 0.0184792 0.00968321 2024-03-26T03:51:29.275 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:29.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:29.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:29.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:29.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:29.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:29.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:29.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:29.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:29.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:29.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:29.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:29.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:29.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:30.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T03:51:30.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1926840243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: osdmap e112: 8 total, 8 up, 7 in 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T03:51:30.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1926840243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: osdmap e112: 8 total, 8 up, 7 in 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T03:51:30.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:29 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T03:51:30.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T03:51:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T03:51:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1926840243' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:51:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:51:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:51:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: osdmap e112: 8 total, 8 up, 7 in 2024-03-26T03:51:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T03:51:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:29 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T03:51:30.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:30.179 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:51:30.179 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:51:30.179 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-26T03:51:30.179 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T03:51:30.179 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T03:51:30.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-26T03:51:30.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:30.267 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 34 16 55837 55821 102.592 82.125 0.00979437 0.00974105 2024-03-26T03:51:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:30.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:30.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:30.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:30.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:30.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:30.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:30.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:30.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:30.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:30.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:30.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: pgmap v100: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T03:51:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T03:51:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T03:51:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T03:51:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:51:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T03:51:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:30 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T03:51:30.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:30.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:30.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:30.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:31.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:31.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:31.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: pgmap v100: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: pgmap v100: 148 pgs: 148 active+clean; 3.1 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T03:51:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T03:51:31.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-26T03:51:31.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:51:31.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T03:51:31.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:30 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T03:51:31.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:31.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:31.265 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 35 16 56925 56909 101.603 68 0.00691115 0.0098371 2024-03-26T03:51:31.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:31.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:31.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:31.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:31.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:31.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:31.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:31.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:31.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:31.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:31.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:51:31.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:31.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:31.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:32.041 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T03:51:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T03:51:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T03:51:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T03:51:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T03:51:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T03:51:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:31 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T03:51:32.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T03:51:32.143 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:31 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T03:51:32.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:32.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.206 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:32.266 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 36 16 58140 58124 100.889 75.9375 0.00675446 0.00990644 2024-03-26T03:51:32.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:32.354 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:32.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:32.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:32.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:32.623 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:32.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:32.787 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:32.800 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:32.810 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:32.881 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:51:32.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:32.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:51:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:32.947 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:32.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:32.971 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:51:32.988 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[37501]: pgmap v101: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 29 B/s, 1 objects/s recovering 2024-03-26T03:51:32.988 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T03:51:32.988 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T03:51:32.990 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[29446]: pgmap v101: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 29 B/s, 1 objects/s recovering 2024-03-26T03:51:32.990 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T03:51:32.990 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:32 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T03:51:33.020 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:51:33.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:33.062 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:51:33.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:32 smithi086 ceph-mon[32965]: pgmap v101: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 101 MiB/s wr, 1.61k op/s; 29 B/s, 1 objects/s recovering 2024-03-26T03:51:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:32 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T03:51:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:32 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T03:51:33.099 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:51:33.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:51:33.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:33.186 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:51:33.200 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:51:33.201 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:51:33.212 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:33.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:33.266 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 37 16 59243 59227 100.026 68.9375 0.00680272 0.0099918 2024-03-26T03:51:33.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:33.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:33.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:33.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:33.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:33.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:33.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:33.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:33.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:33.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:33.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:33.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:33.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:33.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:33.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:34.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:34.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:34.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T03:51:34.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T03:51:34.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3476523622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T03:51:34.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T03:51:34.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3476523622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:34.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:34.030 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:33 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3476523622' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:33 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:34.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:34.129 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:34.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:51:34.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:34.267 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 38 16 60482 60466 99.4275 77.4375 0.0251427 0.0100524 2024-03-26T03:51:34.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:34.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:34.464 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:34.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:34.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:34.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:34.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:34.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:34.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:34.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:34.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:34.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:34.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:34.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:35.014 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: pgmap v102: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 596 B/s rd, 46 MiB/s wr, 743 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 5.1e deep-scrub starts 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 5.1e deep-scrub ok 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: osdmap e113: 8 total, 8 up, 7 in 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 6.3 deep-scrub starts 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 6.3 deep-scrub ok 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: osdmap e114: 8 total, 8 up, 7 in 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: pgmap v102: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 596 B/s rd, 46 MiB/s wr, 743 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:51:35.024 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 5.1e deep-scrub starts 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 5.1e deep-scrub ok 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: osdmap e113: 8 total, 8 up, 7 in 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 6.3 deep-scrub starts 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 6.3 deep-scrub ok 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: osdmap e114: 8 total, 8 up, 7 in 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T03:51:35.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:34 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T03:51:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: pgmap v102: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 596 B/s rd, 46 MiB/s wr, 743 op/s; 25 B/s, 1 objects/s recovering 2024-03-26T03:51:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 5.1e deep-scrub starts 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 5.1e deep-scrub ok 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: osdmap e113: 8 total, 8 up, 7 in 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 6.3 deep-scrub starts 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 6.3 deep-scrub ok 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: osdmap e114: 8 total, 8 up, 7 in 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T03:51:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:34 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T03:51:35.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:35.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:35.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:35.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 39 16 61670 61654 98.7818 74.25 0.00960964 0.0101183 2024-03-26T03:51:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.306 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:35.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:35.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:35.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:35.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:35.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:35.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:35.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.723 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:35.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:35.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:35.833 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T03:51:35.833 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T03:51:35.833 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 5.14 scrub starts 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 5.14 scrub ok 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 5.14 scrub starts 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 5.14 scrub ok 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:35.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T03:51:35.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T03:51:35.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T03:51:35.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:35 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T03:51:35.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:35.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:35.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:35.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:36.069 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 5.14 scrub starts 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 5.14 scrub ok 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T03:51:36.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:35 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T03:51:36.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.141 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:36.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:36.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:36.267 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:51:36.268201+0000 min lat: 0.00129465 max lat: 0.166341 avg lat: 0.0101855 2024-03-26T03:51:36.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:51:36.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 40 16 62830 62814 98.1247 72.5 0.00885752 0.0101855 2024-03-26T03:51:36.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:36.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:36.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:36.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:36.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:36.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:36.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:36.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:36.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:36.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:36.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:36.911 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:51:36.911 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T03:51:36.911 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:51:36.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:36.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: pgmap v105: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 850 B/s rd, 116 MiB/s wr, 1.85k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: osdmap e115: 8 total, 8 up, 7 in 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:51:37.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: pgmap v105: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 850 B/s rd, 116 MiB/s wr, 1.85k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: osdmap e115: 8 total, 8 up, 7 in 2024-03-26T03:51:37.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:51:37.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:36 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:51:37.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:51:37.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: pgmap v105: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 850 B/s rd, 116 MiB/s wr, 1.85k op/s; 36 B/s, 1 objects/s recovering 2024-03-26T03:51:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T03:51:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T03:51:37.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T03:51:37.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T03:51:37.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: osdmap e115: 8 total, 8 up, 7 in 2024-03-26T03:51:37.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:51:37.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:36 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:51:37.127 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:51:37.127 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:51:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:37.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:37.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:37.267 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 41 16 63719 63703 97.0866 55.5625 0.0187606 0.0102943 2024-03-26T03:51:37.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:37.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:37.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:37.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:37.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:37.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:37.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.655 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:37.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:37.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:37.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:37.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:37 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T03:51:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:37 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T03:51:37.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:37 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T03:51:37.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:37 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T03:51:37.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:37.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:37.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:38.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:38.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:37 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T03:51:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:37 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T03:51:38.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:38.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:38.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 42 16 64734 64718 96.2854 63.4375 0.0249113 0.01038 2024-03-26T03:51:38.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:38.354 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:38.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:38.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:38.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:38.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:38.639 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:38.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:38.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:51:38.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:38.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:38.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:38.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:38.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:38.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:38.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:38.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[37501]: pgmap v107: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 992 op/s 2024-03-26T03:51:38.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T03:51:38.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T03:51:38.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:38.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:38.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[29446]: pgmap v107: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 992 op/s 2024-03-26T03:51:38.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T03:51:38.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T03:51:38.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:38.958 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:38.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:39.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:39.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:38 smithi086 ceph-mon[32965]: pgmap v107: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 852 B/s rd, 62 MiB/s wr, 992 op/s 2024-03-26T03:51:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:38 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T03:51:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:38 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T03:51:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:38 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:39.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:39.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:39.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:39.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 43 16 65852 65836 95.6711 69.875 0.0106328 0.010447 2024-03-26T03:51:39.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:39.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:39.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:39.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:39.474 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:39.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:39.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:39.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:39.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:39.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:39.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:39.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:39.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:39.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 6.1f deep-scrub starts 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 6.1f deep-scrub ok 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: osdmap e116: 8 total, 8 up, 7 in 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T03:51:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 6.1f deep-scrub starts 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 6.1f deep-scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: osdmap e116: 8 total, 8 up, 7 in 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T03:51:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T03:51:39.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:39 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T03:51:39.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:40.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 6.1f deep-scrub starts 2024-03-26T03:51:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 6.1f deep-scrub ok 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: osdmap e116: 8 total, 8 up, 7 in 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T03:51:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:39 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T03:51:40.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:40.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:40.140 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:40.215 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:40.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:40.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:40.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 44 16 66980 66964 95.0982 70.5 0.0133326 0.0105096 2024-03-26T03:51:40.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:40.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:40.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:40.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:40.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:40.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:40.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:40.815 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:40.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:40 smithi086 ceph-mon[32965]: pgmap v109: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.8 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T03:51:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:40 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T03:51:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:40 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T03:51:40.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:40.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:40.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:41.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[29446]: pgmap v109: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.8 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T03:51:41.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T03:51:41.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T03:51:41.041 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[37501]: pgmap v109: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.8 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T03:51:41.041 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T03:51:41.041 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:40 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T03:51:41.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:41.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:41.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:41.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:41.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 45 16 68130 68114 94.5821 71.875 0.0151128 0.0105674 2024-03-26T03:51:41.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:41.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:41.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:41.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:41.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:41.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:41.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.725 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:41.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:41.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:41.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:41.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:41.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T03:51:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:41 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T03:51:41.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:42.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:41 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T03:51:42.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:41 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T03:51:42.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:41 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T03:51:42.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:41 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T03:51:42.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:42.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:42.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:42.268 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 46 16 69180 69164 93.9525 65.625 0.0215098 0.0106376 2024-03-26T03:51:42.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:42.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:42.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:42.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:42.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:42.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:42.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:42.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:42.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:42.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:42.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:42.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:42.812 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:51:42.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:42.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:42.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:42.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:42.944 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:51:42.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T03:51:43.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: pgmap v110: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 730 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:51:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T03:51:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T03:51:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T03:51:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T03:51:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3681720406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:42 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.098 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:43.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: pgmap v110: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 730 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3681720406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: pgmap v110: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 730 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:51:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T03:51:43.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T03:51:43.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T03:51:43.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T03:51:43.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3681720406' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:42 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:51:43.172 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:51:43.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:43.216 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:51:43.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:43.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:43.266 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:51:43.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 47 16 70221 70205 93.3363 65.0625 0.0071359 0.0107083 2024-03-26T03:51:43.309 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:51:43.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:43.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:43.353 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:51:43.399 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:51:43.441 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:51:43.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:43.481 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:51:43.491 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:51:43.492 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:51:43.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:43.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:43.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:43.654 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:43.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.805 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:43.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:43.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:43.948 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:43.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:44.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:44.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: 4.14 scrub starts 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: 4.14 scrub ok 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: osdmap e117: 8 total, 8 up, 7 in 2024-03-26T03:51:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1162589416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: 4.14 scrub starts 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: 4.14 scrub ok 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: osdmap e117: 8 total, 8 up, 7 in 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1162589416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: 4.14 scrub starts 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: 4.14 scrub ok 2024-03-26T03:51:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:51:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:51:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: osdmap e117: 8 total, 8 up, 7 in 2024-03-26T03:51:44.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1162589416' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:44.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:44.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:44.269 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 48 16 71186 71170 92.6483 60.3125 0.00790706 0.010787 2024-03-26T03:51:44.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:44.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:44.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:44.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:44.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:44.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:44.621 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:44.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:44.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:44.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:44.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:44.894 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:44.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:44.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:45.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:44 smithi086 ceph-mon[32965]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 687 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-26T03:51:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:44 smithi086 ceph-mon[32965]: osdmap e118: 8 total, 8 up, 7 in 2024-03-26T03:51:45.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:45.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:44 smithi006 ceph-mon[37501]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 687 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-26T03:51:45.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:44 smithi006 ceph-mon[37501]: osdmap e118: 8 total, 8 up, 7 in 2024-03-26T03:51:45.134 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:44 smithi006 ceph-mon[29446]: pgmap v112: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 687 B/s rd, 45 MiB/s wr, 722 op/s 2024-03-26T03:51:45.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:44 smithi006 ceph-mon[29446]: osdmap e118: 8 total, 8 up, 7 in 2024-03-26T03:51:45.179 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:45.269 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 49 16 72148 72132 91.9846 60.125 0.0101132 0.0108656 2024-03-26T03:51:45.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:45.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:45.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:45.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:45.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:45.580 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:45.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:45.725 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:45.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:45.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:45.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:45.998 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:45.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:45 smithi086 ceph-mon[32965]: pgmap v114: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 850 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T03:51:46.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:46.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:45 smithi006 ceph-mon[29446]: pgmap v114: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 850 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T03:51:46.141 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:46.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:46.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:45 smithi006 ceph-mon[37501]: pgmap v114: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 850 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-03-26T03:51:46.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:46.269 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 50 16 73111 73095 91.3481 60.1875 0.00795857 0.0109408 2024-03-26T03:51:46.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:46.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:46.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:46.427 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:46.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:46.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:46.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:46.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:46.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:46.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:46.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:46.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:46.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:46.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:46.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:47.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:47.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:47.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:47.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:47.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:47.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:47.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 51 16 74151 74135 90.8314 65 0.0195971 0.0110041 2024-03-26T03:51:47.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:47.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:47.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.481 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:47.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:47.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:47.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:47.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:47.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:47.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:47.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:47.884 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:47.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:47.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:47.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:48.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:48.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:48.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:48.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:48.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 52 16 75129 75113 90.2601 61.125 0.0119542 0.0110734 2024-03-26T03:51:48.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:48.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:48.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:48.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:48.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:48.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:48.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:48 smithi086 ceph-mon[32965]: pgmap v115: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T03:51:48.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:48 smithi006 ceph-mon[37501]: pgmap v115: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T03:51:48.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:48 smithi006 ceph-mon[29446]: pgmap v115: 148 pgs: 148 active+clean; 4.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T03:51:48.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:48.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:48.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:48.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:48.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:48.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:48.976 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:48.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:48.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:49.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:49.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:49.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:49.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:49.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:76180 7.c e118/96 7:35dfe238:::benchmark_data_smithi006_77260_object76179:head v 118'4745, mlcod=118'4743)", 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.246859+0000", 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.012842572, 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012872883, 2024-03-26T03:51:49.261 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246859+0000", 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246859+0000", 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246860+0000", 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8999999999999996e-07 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246892+0000", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2233999999999997e-05 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246899+0000", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9269999999999999e-06 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246905+0000", 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2439999999999998e-06 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.247320+0000", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00041531700000000001 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.247377+0000", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6524999999999998e-05 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:76184 7.2 e118/95 7:4d830e34:::benchmark_data_smithi006_77260_object76183:head v 118'4703, mlcod=118'4698)", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.246926+0000", 2024-03-26T03:51:49.265 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.01277579, 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012875775000000001, 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246926+0000", 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246926+0000", 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246926+0000", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1699999999999999e-07 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246946+0000", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0191e-05 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246953+0000", 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1419999999999998e-06 2024-03-26T03:51:49.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.246958+0000", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1290000000000004e-06 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.249448+0000", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002490286 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.249467+0000", 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9304e-05 2024-03-26T03:51:49.268 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:76192 7.8 7:1dc89560:::benchmark_data_smithi006_77260_object76191:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.248462+0000", 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.011240004, 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011408288000000001, 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:51:49.269 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 76192 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248462+0000", 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248462+0000", 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248463+0000", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3510000000000001e-06 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248488+0000", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4956e-05 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248502+0000", 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4184e-05 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.248510+0000", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0970000000000006e-06 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.251725+0000", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003214284 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.251864+0000", 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013911300000000001 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.272 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.252027+0000", 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016365800000000001 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.254799+0000", 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027711789999999999 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.273 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:76194 7.b e118/96 7:d8473ef9:::benchmark_data_smithi006_77260_object76193:head v 118'4631, mlcod=118'4629)", 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.253558+0000", 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.006143684, 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006421403, 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253558+0000", 2024-03-26T03:51:49.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253558+0000", 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253559+0000", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7499999999999999e-07 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253583+0000", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3609e-05 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253592+0000", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5449999999999992e-06 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253600+0000", 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5229999999999998e-06 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253689+0000", 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9555000000000006e-05 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.253709+0000", 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9389e-05 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:76186 7.2 e118/95 7:4d1e82ad:::benchmark_data_smithi006_77260_object76185:head v 118'4704, mlcod=118'4699)", 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.259282+0000", 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00041999499999999999, 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00076978799999999998, 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259282+0000", 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259282+0000", 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.279 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259283+0000", 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1739999999999999e-06 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259313+0000", 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0284e-05 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259333+0000", 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9397000000000001e-05 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259342+0000", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0839999999999998e-06 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259472+0000", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000129949 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259498+0000", 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6418999999999999e-05 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.281 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:76198 7.3 e118/95 7:c049fe90:::benchmark_data_smithi006_77260_object76197:head v 118'4757, mlcod=118'4755)", 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.259364+0000", 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00033767600000000002, 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00074755199999999996, 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259364+0000", 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259364+0000", 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259364+0000", 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9700000000000003e-07 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259387+0000", 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2472000000000001e-05 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259393+0000", 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2920000000000001e-06 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259398+0000", 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0860000000000001e-06 2024-03-26T03:51:49.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259867+0000", 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00046930399999999998 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259887+0000", 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9417999999999999e-05 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:76192 7.8 e118/95)", 2024-03-26T03:51:49.285 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:49.259725+0000", 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 4294967295.9999771, 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044509399999999999, 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259725+0000", 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:51:49.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259725+0000", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259725+0000", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5499999999999998e-07 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259734+0000", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0940000000000008e-06 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:51:49.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259744+0000", 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0446e-05 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:49.259751+0000", 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0299999999999996e-06 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 7 2024-03-26T03:51:49.288 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:51:49.288 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 53 16 76208 76192 89.8294 67.4375 0.0146994 0.0111255 2024-03-26T03:51:49.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:49.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:49.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:49.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:49.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:49.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:49.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:49.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:49.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:49.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:49.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:49.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:49.960 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:50.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:50.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:50.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:50.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:50.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 54 16 77278 77262 89.4043 66.875 0.0083202 0.0111799 2024-03-26T03:51:50.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:50.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:50.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:50 smithi006 ceph-mon[37501]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 790 op/s 2024-03-26T03:51:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:50 smithi006 ceph-mon[29446]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 790 op/s 2024-03-26T03:51:50.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:50.478 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:50.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:51:50.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:51:50.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:50.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:50 smithi086 ceph-mon[32965]: pgmap v116: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 790 op/s 2024-03-26T03:51:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.655 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:50.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:50.836 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:50.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:50.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:51.003 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:51.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:51.150 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:51.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:51.270 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 55 16 78242 78226 88.8742 60.25 0.0163012 0.0112467 2024-03-26T03:51:51.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:51.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:51.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:51.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:51.556 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:51.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:51.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:51.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:51.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:51:51.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:51.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:51.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:51.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:51.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:52.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:52.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:52.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:52.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.224 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:52.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:52.271 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 56 16 79255 79239 88.4155 63.3125 0.0250313 0.011305 2024-03-26T03:51:52.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.381 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:52.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:52.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:52 smithi006 ceph-mon[37501]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:52.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:52 smithi006 ceph-mon[29446]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:52.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:52.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:52.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:52.526 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:52.534 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:52.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:52 smithi086 ceph-mon[32965]: pgmap v117: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:51:52.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:52.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.668 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:52.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:52.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:52.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:52.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:52.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:52.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:53.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:53.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:53.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:53.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:53.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 57 16 80196 80180 87.8961 58.8125 0.0162013 0.0113709 2024-03-26T03:51:53.301 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T03:51:53.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:53.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:53.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:53.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:53.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:53.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:51:53.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:53.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:51:53.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T03:51:53.641 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:53.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:53.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:53.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:53.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:53.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:53.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:53.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:54.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:54.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:54.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:54.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:54.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:54.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:54.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 58 16 81181 81165 87.4421 61.5625 0.0194532 0.0114309 2024-03-26T03:51:54.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:54.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:51:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: pgmap v118: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: osdmap e119: 8 total, 8 up, 7 in 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:54.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[37501]: osdmap e120: 8 total, 8 up, 7 in 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: pgmap v118: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: osdmap e119: 8 total, 8 up, 7 in 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:54.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:54 smithi006 ceph-mon[29446]: osdmap e120: 8 total, 8 up, 7 in 2024-03-26T03:51:54.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:54.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:54.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:54.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:54.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:54.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:51:54.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: pgmap v118: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-26T03:51:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2482549533' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T03:51:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: osdmap e119: 8 total, 8 up, 7 in 2024-03-26T03:51:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:51:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:51:54.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:54 smithi086 ceph-mon[32965]: osdmap e120: 8 total, 8 up, 7 in 2024-03-26T03:51:54.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:54.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:54.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:54.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:54.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:54.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:54.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:54.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:54.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:55.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:55.106 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:55.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:55.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:55.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.255 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:55.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:55.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 59 16 82186 82170 87.0246 62.8125 0.0122781 0.0114855 2024-03-26T03:51:55.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:55.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:55.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:55.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:55.537 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:51:55.562 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:55.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:55 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:51:55.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:55 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:51:55.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:51:55.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:51:55.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:55.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:51:55.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:55.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:55.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:55.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:55.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:55.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:55.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:56.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:56.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:56.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:56.220 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:56.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:56.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:51:56.272735+0000 min lat: 0.00129465 max lat: 0.166341 avg lat: 0.0115324 2024-03-26T03:51:56.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:51:56.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 60 16 83242 83226 86.6741 66 0.0112238 0.0115324 2024-03-26T03:51:56.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:56.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:56.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:56.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:56.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:56.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:56.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:56 smithi086 ceph-mon[32965]: pgmap v121: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:51:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.618 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:56.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:51:56.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:56 smithi006 ceph-mon[37501]: pgmap v121: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:51:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:56 smithi006 ceph-mon[29446]: pgmap v121: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:51:56.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:56.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:56.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:56.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:56.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:56.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:57.038 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:57.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:57.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:57.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:57.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:57.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:57.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 61 16 84287 84271 86.3236 65.3125 0.0146564 0.0115794 2024-03-26T03:51:57.314 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:57.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:57.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:57.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:57.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:57.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:57.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:57.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:57.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:57.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:57.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:57.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T03:51:57.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:51:58.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:51:58.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:58.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:58.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:51:58.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 62 16 85274 85258 85.9263 61.6875 0.0129481 0.0116312 2024-03-26T03:51:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:58.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:58.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:58.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:58.496 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:58 smithi006 ceph-mon[37501]: pgmap v122: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T03:51:58.496 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:58 smithi006 ceph-mon[29446]: pgmap v122: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T03:51:58.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:58 smithi086 ceph-mon[32965]: pgmap v122: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-26T03:51:58.590 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:58.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:58.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:58.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:58.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:58.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:58.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:58.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:58.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:59.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:59.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:59.087 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:51:59.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:51:59.129 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:51:59.172 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:51:59.176 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:51:59.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:51:59.270 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:51:59.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 63 16 86216 86200 85.4968 58.875 0.0178974 0.011691 2024-03-26T03:51:59.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:59.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:59.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:59.313 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:51:59.356 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:51:59.398 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:51:59.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:51:59.419 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:51:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:51:59.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:51:59.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:51:59.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:51:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4003985693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:51:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4003985693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:59.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:51:59.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:51:59.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:51:59.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:51:59.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:51:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:51:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4003985693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:51:59.930 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:51:59.930 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:51:59.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:51:59.976 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:51:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:51:59.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:00.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:00.120 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:00.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:00.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:00.272 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 64 16 87237 87221 85.1579 63.8125 0.00830941 0.0117373 2024-03-26T03:52:00.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:00.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:00.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:00.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:00.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:00.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:00.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:00.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:00.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:00.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:00 smithi006 ceph-mon[37501]: pgmap v123: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 738 op/s 2024-03-26T03:52:00.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:00 smithi006 ceph-mon[29446]: pgmap v123: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 738 op/s 2024-03-26T03:52:00.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:00.777 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:00.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:00 smithi086 ceph-mon[32965]: pgmap v123: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 46 MiB/s wr, 738 op/s 2024-03-26T03:52:00.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:00.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:00.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:01.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:01.070 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:01.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:52:01.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:01.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:01.274 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 65 16 88300 88284 84.8678 66.4375 0.0101147 0.0117777 2024-03-26T03:52:01.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:01.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:01.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:01 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:01 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:01 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:01 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:01.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:01.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:01.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:01.560 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:01.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:01.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:01.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:01.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:01.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:01.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:01 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:01 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:01.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:01.886 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:01.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:01.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:01.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:02.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:02.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:02.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:02.171 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:02.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:02.274 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 66 16 89290 89274 84.5195 61.875 0.00699218 0.0118266 2024-03-26T03:52:02.311 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:02.412 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:02.456 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:02.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:52:02.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:02 smithi006 ceph-mon[37501]: pgmap v124: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:52:02.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:02 smithi006 ceph-mon[37501]: osdmap e121: 8 total, 8 up, 7 in 2024-03-26T03:52:02.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:02 smithi006 ceph-mon[29446]: pgmap v124: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:52:02.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:02 smithi006 ceph-mon[29446]: osdmap e121: 8 total, 8 up, 7 in 2024-03-26T03:52:02.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:02.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:02.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.774 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:02.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:02 smithi086 ceph-mon[32965]: pgmap v124: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T03:52:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:02 smithi086 ceph-mon[32965]: osdmap e121: 8 total, 8 up, 7 in 2024-03-26T03:52:02.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:02.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:02.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:03.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:03.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:03.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:03.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:03.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:03.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:03.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 67 16 90247 90231 84.1495 59.8125 0.0347054 0.0118778 2024-03-26T03:52:03.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.298 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:03.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:03.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:03.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:03.457 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:03.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:03.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:03.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:03.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:03.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:03.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:03.845 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:03.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:03.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:03.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:03.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:04.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:04.124 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:04.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:04.227 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:52:04.277 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 68 16 91228 91212 83.8137 61.3125 0.017075 0.0119257 2024-03-26T03:52:04.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:04.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:04.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:04.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:04.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:04.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: pgmap v126: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 567 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2970720657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:04.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:04 smithi086 ceph-mon[32965]: osdmap e122: 8 total, 8 up, 7 in 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: pgmap v126: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 567 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2970720657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[37501]: osdmap e122: 8 total, 8 up, 7 in 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: pgmap v126: 148 pgs: 148 active+clean; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 567 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2970720657' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:04.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:04 smithi006 ceph-mon[29446]: osdmap e122: 8 total, 8 up, 7 in 2024-03-26T03:52:04.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:04.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:04.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:04.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:04.846 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:04.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:04.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:04.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:04.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:05.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:05.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:05.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.263 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:05.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:05.275 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 69 16 92261 92245 83.5346 64.5625 0.0264514 0.0119654 2024-03-26T03:52:05.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:05.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:05.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:05.507 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:05.507 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:05.535 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:05.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:05.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:05.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:05.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:05.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:05.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:05.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:05.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:05.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:06.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:06.090 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:06.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:06.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:06.275 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 70 16 93252 93236 83.2261 61.9375 0.0159066 0.0120103 2024-03-26T03:52:06.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:06.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:06.405 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T03:52:06.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:06.462 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:06.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:06.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:06.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:06.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:06.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:06.788 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[37501]: pgmap v128: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:52:06.788 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2788148546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.788 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[29446]: pgmap v128: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:52:06.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2788148546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.789 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:06 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:06 smithi086 ceph-mon[32965]: pgmap v128: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-26T03:52:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2788148546' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:06 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T03:52:06.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:06.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:06.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:06.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:06.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:07.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:07.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.138 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:07.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:07.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:07.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 71 16 94262 94246 82.9429 63.125 0.0125187 0.0120511 2024-03-26T03:52:07.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:07.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:07.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:07.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T03:52:07.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:07.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.612 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:07.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:07.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:07.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:07.758 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:52:07.758 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:52:07.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:07 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:52:07.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:07 smithi006 ceph-mon[29446]: osdmap e123: 8 total, 8 up, 7 in 2024-03-26T03:52:07.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:07 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:52:07.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:07 smithi006 ceph-mon[37501]: osdmap e123: 8 total, 8 up, 7 in 2024-03-26T03:52:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:07 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T03:52:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:07 smithi086 ceph-mon[32965]: osdmap e123: 8 total, 8 up, 7 in 2024-03-26T03:52:07.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:07.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:07.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:08.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.055 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:08.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:08.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:52:08.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:08.208 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:08.275 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 72 16 95047 95031 82.4724 49.0625 0.019946 0.0121194 2024-03-26T03:52:08.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:08.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:08.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:08.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:08.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:08.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:08.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:08.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:08.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:52:08.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:08.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:08 smithi006 ceph-mon[29446]: pgmap v130: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s 2024-03-26T03:52:08.704 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:08 smithi006 ceph-mon[37501]: pgmap v130: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s 2024-03-26T03:52:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:08.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:08 smithi086 ceph-mon[32965]: pgmap v130: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s 2024-03-26T03:52:08.857 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:08.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:08.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:09.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:09.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:09.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:09.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:09.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 73 16 95860 95844 82.0386 50.8125 0.0098862 0.0121833 2024-03-26T03:52:09.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:09.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:09.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:09.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:09.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:09.589 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:09.590 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:09 smithi086 ceph-mon[32965]: osdmap e124: 8 total, 8 up, 7 in 2024-03-26T03:52:09.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:09.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:09.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:09.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:09 smithi006 ceph-mon[37501]: osdmap e124: 8 total, 8 up, 7 in 2024-03-26T03:52:09.715 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:09.715 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:09 smithi006 ceph-mon[29446]: osdmap e124: 8 total, 8 up, 7 in 2024-03-26T03:52:09.741 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:09.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:09.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:09.857 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:09.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:09.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:09.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:09.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:10.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:10.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:10.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:10.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 74 16 96902 96886 81.81 65.125 0.0123163 0.0122189 2024-03-26T03:52:10.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:10.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:10.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:10.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:10.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:10.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:10.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:10.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:10.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.824 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:10.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:10 smithi086 ceph-mon[32965]: pgmap v132: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-26T03:52:10.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:10 smithi006 ceph-mon[37501]: pgmap v132: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-26T03:52:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:10 smithi006 ceph-mon[29446]: pgmap v132: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-26T03:52:10.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:10.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:10.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:10.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:11.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:11.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:11.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:11.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:11.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 75 16 97849 97833 81.5084 59.1875 0.00326848 0.0122621 2024-03-26T03:52:11.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:11.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:11.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:11.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:11.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:11.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:11.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:11.616 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:11.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:11.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:11.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:11.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:11.888 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:11.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:11.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:11.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.035 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:12.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:12.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:12.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:12.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:12.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:12.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 76 16 98851 98835 81.2599 62.625 0.00724248 0.0123005 2024-03-26T03:52:12.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:12.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:12.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:12.431 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:12 smithi006 ceph-mon[29446]: pgmap v133: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 729 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:52:12.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:12.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:12.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:12.577 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:12.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:12.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:12.762 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:12 smithi006 ceph-mon[37501]: pgmap v133: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 729 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:52:12.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:12.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:12.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:12 smithi086 ceph-mon[32965]: pgmap v133: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 729 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:52:12.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:12.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:12.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:13.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:13.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:13.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:13.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:13.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 77 16 99908 99892 81.0625 66.0625 0.00614592 0.0123305 2024-03-26T03:52:13.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:13.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:13.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:13.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:13.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:13.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:13.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:13.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:13 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:13 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:13 smithi086 ceph-mon[32965]: osdmap e125: 8 total, 8 up, 7 in 2024-03-26T03:52:13.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:13.871 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:13.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:13.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:13.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[37501]: osdmap e125: 8 total, 8 up, 7 in 2024-03-26T03:52:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:13 smithi006 ceph-mon[29446]: osdmap e125: 8 total, 8 up, 7 in 2024-03-26T03:52:13.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:14.018 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:14.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:14.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:14.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:14.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:14.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:14.277 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 78 16 100979 100963 80.8813 66.9375 0.020578 0.0123589 2024-03-26T03:52:14.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:14.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:14.455 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:14.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:14.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:14.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:14 smithi086 ceph-mon[32965]: pgmap v135: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 748 B/s rd, 42 MiB/s wr, 678 op/s 2024-03-26T03:52:14.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:14 smithi086 ceph-mon[32965]: osdmap e126: 8 total, 8 up, 7 in 2024-03-26T03:52:14.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:14.607 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:52:14.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:14.651 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:52:14.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:14.697 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:52:14.741 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:52:14.742 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:14 smithi006 ceph-mon[37501]: pgmap v135: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 748 B/s rd, 42 MiB/s wr, 678 op/s 2024-03-26T03:52:14.742 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:14 smithi006 ceph-mon[37501]: osdmap e126: 8 total, 8 up, 7 in 2024-03-26T03:52:14.742 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:14 smithi006 ceph-mon[29446]: pgmap v135: 148 pgs: 148 active+clean; 5.9 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 748 B/s rd, 42 MiB/s wr, 678 op/s 2024-03-26T03:52:14.742 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:14 smithi006 ceph-mon[29446]: osdmap e126: 8 total, 8 up, 7 in 2024-03-26T03:52:14.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:14.790 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:52:14.801 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:14.832 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:52:14.873 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:52:14.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:14.915 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:52:14.917 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:52:14.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:14.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:15.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:15.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:15.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:15.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:15.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:15.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 79 16 101859 101843 80.5537 55 0.0175311 0.0124095 2024-03-26T03:52:15.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:15.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:15.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:15.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:15.543 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:102054 7.2 7:4ba1060d:::benchmark_data_smithi006_77260_object102053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.515223+0000", 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.022141717000000002, 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022185779999999999, 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:15.544 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 102054 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515223+0000", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515223+0000", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515224+0000", 2024-03-26T03:52:15.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2799999999999996e-07 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515243+0000", 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8573999999999999e-05 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515250+0000", 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0269999999999998e-06 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515256+0000", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.122e-06 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515462+0000", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020632499999999999 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515575+0000", 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113303 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515709+0000", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013378899999999999 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.531527+0000", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015817716999999999 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.531755+0000", 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022868699999999999 2024-03-26T03:52:15.548 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:102055 7.c 7:3a9320c6:::benchmark_data_smithi006_77260_object102054:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.515272+0000", 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.022093435000000002, 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022245438999999999, 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:15.549 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 102055 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515272+0000", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515272+0000", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515272+0000", 2024-03-26T03:52:15.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6499999999999999e-07 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515292+0000", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9545e-05 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515298+0000", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5649999999999998e-06 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.515304+0000", 2024-03-26T03:52:15.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6509999999999998e-06 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.521529+0000", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0062255640000000003 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.529686+0000", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081563179999999992 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.529870+0000", 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018457099999999999 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533699+0000", 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038281589999999998 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:102067 7.a 7:5fa0b036:::benchmark_data_smithi006_77260_object102066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.526855+0000", 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.010509760999999999, 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010758772, 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.553 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 102067 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526855+0000", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526855+0000", 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526857+0000", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5409999999999999e-06 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526881+0000", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3900000000000002e-05 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526891+0000", 2024-03-26T03:52:15.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9620000000000005e-06 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.526898+0000", 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3880000000000002e-06 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.527069+0000", 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000170779 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.527186+0000", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011657700000000001 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.527319+0000", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013361100000000001 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.531723+0000", 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044037709999999999 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.536715+0000", 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049920119999999997 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:102069 7.2 7:48f1e31f:::benchmark_data_smithi006_77260_object102068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.530789+0000", 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0065764489999999998, 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0069227610000000004, 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.558 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 102069 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530789+0000", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530789+0000", 2024-03-26T03:52:15.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530790+0000", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4410000000000001e-06 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530812+0000", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.192e-05 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530822+0000", 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0145e-05 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.530830+0000", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.576e-06 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.531783+0000", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095275299999999998 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.531922+0000", 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013923000000000001 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.532082+0000", 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016032500000000001 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:102065 7.6 e126/95 7:6de249f7:::benchmark_data_smithi006_77260_object102064:head v 126'6334, mlcod=126'6332)", 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.533462+0000", 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0039026489999999998, 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043263379999999999, 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533462+0000", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533462+0000", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533463+0000", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2500000000000005e-07 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533486+0000", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3071999999999999e-05 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533495+0000", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0669999999999998e-06 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533502+0000", 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0219999999999999e-06 2024-03-26T03:52:15.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533581+0000", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8848000000000001e-05 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533600+0000", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8756000000000001e-05 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:102070 7.0 e126/95 7:02173585:::benchmark_data_smithi006_77260_object102069:head v 126'6438, mlcod=126'6436)", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.533661+0000", 2024-03-26T03:52:15.565 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0037037060000000002, 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041918329999999998, 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533661+0000", 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533661+0000", 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533662+0000", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5199999999999996e-07 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533721+0000", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8882000000000003e-05 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533731+0000", 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5570000000000001e-06 2024-03-26T03:52:15.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533738+0000", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0079999999999996e-06 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533807+0000", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9382999999999996e-05 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.533826+0000", 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9422000000000001e-05 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.568 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:102075 7.a 7:5aac609e:::benchmark_data_smithi006_77260_object102074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.535229+0000", 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0021360749999999999, 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026945379999999998, 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 102075 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.569 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535229+0000", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535229+0000", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535231+0000", 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8989999999999999e-06 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535265+0000", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4501000000000001e-05 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535279+0000", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3219e-05 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535288+0000", 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0569999999999999e-06 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.536746+0000", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014586 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.536891+0000", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014426599999999999 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.537054+0000", 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016298199999999999 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.572 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:102074 7.1 e126/96 7:8a1354d6:::benchmark_data_smithi006_77260_object102073:head v 126'6388, mlcod=126'6386)", 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:52:15.535758+0000", 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.001607444, 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002254327, 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535758+0000", 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535758+0000", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535758+0000", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.99e-07 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535824+0000", 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5649999999999997e-05 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535833+0000", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8480000000000007e-06 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535839+0000", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0310000000000004e-06 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535902+0000", 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2891999999999993e-05 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:52:15.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:52:15.535922+0000", 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0333e-05 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 8 2024-03-26T03:52:15.576 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3013472228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3013472228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:15.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:15 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:52:15.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:15.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:15.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:15.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:15.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:15 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3013472228' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:15 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:52:15.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:15.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:15.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:15.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:16.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:16.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:16.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:16.164 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:16.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:16.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:16.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:52:16.277185+0000 min lat: 0.00129465 max lat: 0.166341 avg lat: 0.0124468 2024-03-26T03:52:16.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:52:16.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 80 16 102837 102821 80.3108 61.125 0.0182133 0.0124468 2024-03-26T03:52:16.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:16.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:16.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:16.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:16.574 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:16.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:16.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:16.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:16.829 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:16.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:16 smithi086 ceph-mon[32965]: pgmap v137: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 850 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T03:52:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:16 smithi006 ceph-mon[37501]: pgmap v137: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 850 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T03:52:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:16 smithi006 ceph-mon[29446]: pgmap v137: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 850 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-26T03:52:16.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:16.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:16.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:17.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:17.103 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:17.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:17.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:17.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:17.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 81 16 103838 103822 80.0916 62.5625 0.00788229 0.0124804 2024-03-26T03:52:17.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:17.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:17.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:17.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:17.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:17.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:17.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:17.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:17.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:17.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:52:17.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:17.914 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:17.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:18.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:18.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:18.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:18.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:18.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:18.276 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 82 15 104822 104807 79.8657 61.5625 0.015225 0.0125154 2024-03-26T03:52:18.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:18.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:18.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:18.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:18.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:18.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:18.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:18 smithi006 ceph-mon[29446]: pgmap v138: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-26T03:52:18.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:18 smithi006 ceph-mon[37501]: pgmap v138: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-26T03:52:18.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:18.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:18 smithi086 ceph-mon[32965]: pgmap v138: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.11k op/s 2024-03-26T03:52:18.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:18.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:18.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:18.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:18.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:19.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:19.133 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:19.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:19.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:19.276 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:19.277 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 83 16 105698 105682 79.5623 54.6875 0.0170038 0.0125634 2024-03-26T03:52:19.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:19.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:19.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:19.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:19.520 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T03:52:19.520 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:52:19.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:19.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:19.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:19.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:19.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:19.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:19.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:52:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3758632305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:19 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3758632305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3758632305' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:19 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:52:19.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:19.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:20.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:20.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:20.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:20.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:20.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:20.277 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 84 16 106663 106647 79.3331 60.3125 0.0201224 0.0125992 2024-03-26T03:52:20.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:20.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:20.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:20.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:20.489 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:20.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:20.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:20.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: pgmap v139: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: osdmap e127: 8 total, 8 up, 7 in 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: pgmap v139: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:52:20.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:52:20.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: osdmap e127: 8 total, 8 up, 7 in 2024-03-26T03:52:20.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:20 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:20.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.762 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:20.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: pgmap v139: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 753 op/s 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3943746985' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: osdmap e127: 8 total, 8 up, 7 in 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:20 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:20.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:20.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:20.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:21.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:21.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:21.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:52:21.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:21.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:21.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:21.278 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 85 16 107574 107558 79.0682 56.9375 0.0217429 0.012642 2024-03-26T03:52:21.311 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:21.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:21.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:21.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:21.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:21.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:21.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:21.770 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:21.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:21.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:21.898 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:21.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:22.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:22.038 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:22.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T03:52:22.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:22.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:22.278 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 86 16 108584 108568 78.8827 63.125 0.0218153 0.0126715 2024-03-26T03:52:22.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:22.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:52:22.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:22.365 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:22.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:22.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T03:52:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T03:52:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[29446]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:52:22.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T03:52:22.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T03:52:22.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:22 smithi006 ceph-mon[37501]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:52:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.683 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:22.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:22.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:22 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T03:52:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:22 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T03:52:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:22 smithi086 ceph-mon[32965]: pgmap v141: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T03:52:22.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:22.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:22.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:23.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:23.098 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:23.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:23.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:23.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:23.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:23.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 87 16 109483 109467 78.6218 56.1875 0.0336052 0.0127134 2024-03-26T03:52:23.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:23.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:23.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:23.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:23.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:23.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:23.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:23.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:23.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:23.739 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:23.808 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:23 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T03:52:23.808 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:23 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T03:52:23.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:23.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:23 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T03:52:23.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:23 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T03:52:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:23 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T03:52:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:23 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T03:52:23.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:23.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:23.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:23.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:24.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:24.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:24.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:24.208 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:24.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 88 16 110478 110462 78.4351 62.1875 0.0275995 0.0127447 2024-03-26T03:52:24.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:24.334 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:24.348 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:24.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:24.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:24.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:24.626 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:24.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:24.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:24.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:24.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 567 B/s rd, 62 MiB/s wr, 986 op/s 2024-03-26T03:52:24.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 567 B/s rd, 62 MiB/s wr, 986 op/s 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T03:52:24.761 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:24 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T03:52:24.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: pgmap v142: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 567 B/s rd, 62 MiB/s wr, 986 op/s 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T03:52:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:24 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T03:52:24.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:24.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:24.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:24.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:25.022 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:25.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:25.165 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:25.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:25.278 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 89 16 111360 111344 78.1731 55.125 0.0135636 0.0127875 2024-03-26T03:52:25.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.306 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:25.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:25.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:25.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:25.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:25.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:25.722 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:25.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T03:52:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:25 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T03:52:25.863 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:25.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T03:52:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T03:52:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T03:52:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T03:52:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:25 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T03:52:25.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:26.005 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:26.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:26.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:26.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:26.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:26.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total time run: 90.0085 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total writes made: 112351 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Write size: 65536 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object size: 65536 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Bandwidth (MB/sec): 78.0142 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Bandwidth: 26.6991 2024-03-26T03:52:26.279 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max bandwidth (MB/sec): 178.125 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min bandwidth (MB/sec): 49.0625 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average IOPS: 1248 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev IOPS: 427.186 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max IOPS: 2850 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min IOPS: 785 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average Latency(s): 0.0128141 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Latency(s): 0.00910074 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max latency(s): 0.166341 2024-03-26T03:52:26.280 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min latency(s): 0.00129465 2024-03-26T03:52:26.282 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:52:26.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:26.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:26.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:26.606 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:52:26.606 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:52:26.606 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:52:26.607 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:52:26.607 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:52:26.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:26.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[37501]: pgmap v143: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[29446]: pgmap v143: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T03:52:26.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:26 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T03:52:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:26.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:26.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:26 smithi086 ceph-mon[32965]: pgmap v143: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:26 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T03:52:26.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:26 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T03:52:26.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:26 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T03:52:26.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:26 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T03:52:26.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:26.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:26.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:26.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:26.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:27.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:27.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:27.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:27.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:27.303 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:27.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:27.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:27.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:27.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T03:52:27.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T03:52:27.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T03:52:27.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T03:52:27.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T03:52:27.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T03:52:27.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T03:52:27.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:27 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T03:52:27.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:27.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:27.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:27 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T03:52:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:27 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T03:52:27.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:27 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T03:52:27.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:27 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T03:52:27.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:27.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:27.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:27.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:28.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:28.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:28.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:28.268 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:28.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:28.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:28.484 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:28.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:28.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:52:28.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:28.566 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:28.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:28.727 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:28.727 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:28.727 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:28.727 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: osdmap e128: 8 total, 8 up, 7 in 2024-03-26T03:52:28.727 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: osdmap e128: 8 total, 8 up, 7 in 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T03:52:28.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:28 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T03:52:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.735 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:28.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:28.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: pgmap v144: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 952 op/s 2024-03-26T03:52:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: osdmap e128: 8 total, 8 up, 7 in 2024-03-26T03:52:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T03:52:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:28 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T03:52:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:28.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:28.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:28.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:29.031 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:29.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:29.179 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:29.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:29.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:29.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:29.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:29.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:29.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:29.577 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:29.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:29 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T03:52:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:29 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T03:52:29.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:29.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:29.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:29.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:29.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:29 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T03:52:29.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:29 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T03:52:29.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:29 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T03:52:29.846 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:29 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T03:52:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:29.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:29.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:30.015 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:30.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:30.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:30.257 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:52:30.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:30.263 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:52:30.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:30.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:30.315 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:52:30.356 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:52:30.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:30.397 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:52:30.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:30.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:30.444 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:52:30.501 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:52:30.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:30.557 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:52:30.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:52:30.570 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:52:30.571 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:52:30.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:30.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 593 op/s 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T03:52:30.746 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/58258863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 593 op/s 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T03:52:30.747 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:30 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/58258863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:30.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:30.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:30.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: pgmap v146: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.0 KiB/s rd, 33 MiB/s wr, 593 op/s 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T03:52:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:30 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/58258863' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:30.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:30.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:30.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:31.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:31.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:31.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:31.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:31.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:31.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:31.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:31.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:31.405 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:31.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.528 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:31.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T03:52:31.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:31.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:31.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:31.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:31.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.810 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:31.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:31 smithi086 ceph-mon[32965]: 7.0 scrub starts 2024-03-26T03:52:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:31 smithi086 ceph-mon[32965]: 7.4 scrub starts 2024-03-26T03:52:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:31 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:52:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:31 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T03:52:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:31 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[37501]: 7.0 scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[37501]: 7.4 scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[29446]: 7.0 scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[29446]: 7.4 scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T03:52:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:31 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T03:52:31.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:31.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:31.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:32.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:32.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:32.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:32.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:32.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T03:52:32.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:32.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:32.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:32.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:32.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:32.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:32.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:32.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:32.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:32.753 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:32.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T03:52:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T03:52:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: pgmap v147: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 981 op/s 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 7.8 scrub starts 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: osdmap e129: 8 total, 8 up, 7 in 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T03:52:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:32 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T03:52:32.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:32.879 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:32.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: pgmap v147: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 981 op/s 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 7.8 scrub starts 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: osdmap e129: 8 total, 8 up, 7 in 2024-03-26T03:52:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: pgmap v147: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.1 KiB/s rd, 42 MiB/s wr, 981 op/s 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 7.8 scrub starts 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T03:52:32.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T03:52:32.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T03:52:32.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/15734293' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T03:52:32.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: osdmap e129: 8 total, 8 up, 7 in 2024-03-26T03:52:32.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T03:52:32.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:32 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T03:52:32.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:33.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:33.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:33.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:33.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:52:33.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:33.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:33.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:33.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:33.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:33.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:33.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:33.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:33.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:33.804 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 7.0 scrub ok 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T03:52:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:33 smithi086 ceph-mon[32965]: 7.4 scrub ok 2024-03-26T03:52:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 7.0 scrub ok 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[37501]: 7.4 scrub ok 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T03:52:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 7.0 scrub ok 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T03:52:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:33 smithi006 ceph-mon[29446]: 7.4 scrub ok 2024-03-26T03:52:33.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:33.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:33.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:33.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:34.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:34.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:34.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:34.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:34.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:34.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:34.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:34.455 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:34.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:34.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:34.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: pgmap v149: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 15 MiB/s wr, 626 op/s 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 7.9 scrub starts 2024-03-26T03:52:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[29446]: 7.8 scrub ok 2024-03-26T03:52:34.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:34.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:34.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:34.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:34.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:34.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: pgmap v149: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 15 MiB/s wr, 626 op/s 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 7.9 scrub starts 2024-03-26T03:52:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:34 smithi086 ceph-mon[32965]: 7.8 scrub ok 2024-03-26T03:52:34.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:34.863 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:34.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:34.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: pgmap v149: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 15 MiB/s wr, 626 op/s 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 7.9 scrub starts 2024-03-26T03:52:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:34 smithi006 ceph-mon[37501]: 7.8 scrub ok 2024-03-26T03:52:34.965 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:35.017 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:35.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:35.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.202 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:35.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:35.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:35.353 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:35.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:35.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.497 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:35.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:35.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:35.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:35.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:35.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:35.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:35.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: 4.18 scrub starts 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: 4.18 scrub ok 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: osdmap e130: 8 total, 8 up, 7 in 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T03:52:35.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: 4.18 scrub starts 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: 4.18 scrub ok 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: osdmap e130: 8 total, 8 up, 7 in 2024-03-26T03:52:35.859 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:35 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T03:52:35.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:35.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:35.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:36.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:36.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: 4.18 scrub starts 2024-03-26T03:52:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: 4.18 scrub ok 2024-03-26T03:52:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T03:52:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: osdmap e130: 8 total, 8 up, 7 in 2024-03-26T03:52:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:35 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T03:52:36.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:36.195 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:36.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.345 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:36.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:36.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:36.461 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:52:36.461 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:52:36.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:36.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:36.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:36.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:36.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:36.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:36.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:36.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.24k op/s 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: 4.1d scrub starts 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: 4.1d scrub ok 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[37501]: 7.9 scrub ok 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.24k op/s 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: 4.1d scrub starts 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: 4.1d scrub ok 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T03:52:36.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:36 smithi006 ceph-mon[29446]: 7.9 scrub ok 2024-03-26T03:52:36.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:36.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:36.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:36.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:36.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:37.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: pgmap v151: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.4 KiB/s rd, 15 MiB/s wr, 1.24k op/s 2024-03-26T03:52:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: 4.1d scrub starts 2024-03-26T03:52:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: 4.1d scrub ok 2024-03-26T03:52:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T03:52:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T03:52:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:36 smithi086 ceph-mon[32965]: 7.9 scrub ok 2024-03-26T03:52:37.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:37.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:37.300 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:37.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:37.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:37.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:37.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:37.516 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:37.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:52:37.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:37.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:37.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:37.648 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:52:37.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:37.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:37 smithi006 ceph-mon[29446]: 7.e scrub starts 2024-03-26T03:52:37.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:37.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:37 smithi006 ceph-mon[37501]: 7.e scrub starts 2024-03-26T03:52:37.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:37.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:37.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:37.813 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:37 smithi086 ceph-mon[32965]: 7.e scrub starts 2024-03-26T03:52:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:52:37.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:37.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:37.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:37.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:38.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:38.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:38.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:38.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:38.295 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:38.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:38.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:38.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:38.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:38.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:38.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:38.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:38.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:38.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:38.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 12 MiB/s wr, 1.07k op/s 2024-03-26T03:52:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: osdmap e131: 8 total, 8 up, 7 in 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: 5.1 scrub starts 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: 5.1 scrub ok 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 12 MiB/s wr, 1.07k op/s 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: osdmap e131: 8 total, 8 up, 7 in 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: 5.1 scrub starts 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: 5.1 scrub ok 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:38.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:38.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:38.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:38.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: pgmap v152: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 767 B/s rd, 12 MiB/s wr, 1.07k op/s 2024-03-26T03:52:38.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T03:52:38.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:52:38.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:52:38.978 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2398554603' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: osdmap e131: 8 total, 8 up, 7 in 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: 5.1 scrub starts 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: 5.1 scrub ok 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:38.979 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:39.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:39.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:39.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:39.249 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:39.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:39.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:39.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:39.522 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:39.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:39.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:39.716 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:52:39.716 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:52:39.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:39.788 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:39.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:39 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T03:52:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:39 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T03:52:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:39 smithi086 ceph-mon[32965]: 4.1b deep-scrub starts 2024-03-26T03:52:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:39 smithi086 ceph-mon[32965]: 4.1b deep-scrub ok 2024-03-26T03:52:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:39 smithi086 ceph-mon[32965]: 7.e scrub ok 2024-03-26T03:52:39.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[37501]: 4.1b deep-scrub starts 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[37501]: 4.1b deep-scrub ok 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[37501]: 7.e scrub ok 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[29446]: 4.1b deep-scrub starts 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[29446]: 4.1b deep-scrub ok 2024-03-26T03:52:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:39 smithi006 ceph-mon[29446]: 7.e scrub ok 2024-03-26T03:52:39.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:39.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:39.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:40.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:40.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:40.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:40.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:40.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:40.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:40.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:40.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:40.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:52:40.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:40.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:40.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:40.767 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:40.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:40.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:40.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: pgmap v154: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 702 op/s 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:40.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:41.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:41.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:41.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:41.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:41.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:41.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:41.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:41.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:41.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:41.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:41.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:41.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:41.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:41.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:52:41.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:41.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:41.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T03:52:41.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T03:52:41.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:41 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T03:52:41.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T03:52:41.984 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:41 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T03:52:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:42.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:42.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:42.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:42.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:42.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:42.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:42.543 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:52:42.543 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:52:42.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:42.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:42.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:42.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:42.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:42.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: pgmap v155: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:52:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T03:52:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T03:52:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: 7.b scrub starts 2024-03-26T03:52:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:52:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:52:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:42 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:42.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:42.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:42.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:42.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:42.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:43.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:43.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: pgmap v155: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: 7.b scrub starts 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: pgmap v155: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: 7.b scrub starts 2024-03-26T03:52:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:52:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:52:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:42 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:43.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:43.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:43.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:52:43.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:43.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:43.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:43.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:43.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:43.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.692 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:43.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:43.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:43.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:43.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:43.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 5.f deep-scrub starts 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 5.f deep-scrub ok 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T03:52:43.923 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 5.f deep-scrub starts 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 5.f deep-scrub ok 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:43.924 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T03:52:43.925 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:43 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T03:52:43.925 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 5.f deep-scrub starts 2024-03-26T03:52:43.925 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 5.f deep-scrub ok 2024-03-26T03:52:43.925 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T03:52:43.926 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:43 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T03:52:43.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:43.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:43.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:44.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:44.065 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:44.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:44.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:44.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:44.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:44.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:44.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:44.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:44.556 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:44.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:44.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:44.663 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:44.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:44.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:44.782 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:52:44.782 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:52:44.817 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: pgmap v156: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 508 op/s 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: 7.f scrub starts 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: osdmap e132: 8 total, 8 up, 7 in 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: 5.19 scrub starts 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: 5.19 scrub ok 2024-03-26T03:52:44.818 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1767240913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: pgmap v156: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 508 op/s 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: 7.f scrub starts 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: osdmap e132: 8 total, 8 up, 7 in 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: 5.19 scrub starts 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: 5.19 scrub ok 2024-03-26T03:52:44.819 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1767240913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:44.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:44.871 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:52:44.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:44.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:44.916 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:52:44.966 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:52:44.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:45.009 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:52:45.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:45.059 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:52:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:52:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:52:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:52:45.065 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:52:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:52:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:52:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:52:45.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: pgmap v156: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 592 B/s rd, 0 B/s wr, 508 op/s 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: 7.f scrub starts 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: osdmap e132: 8 total, 8 up, 7 in 2024-03-26T03:52:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: 5.19 scrub starts 2024-03-26T03:52:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: 5.19 scrub ok 2024-03-26T03:52:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1767240913' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:45.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:45.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.172 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:45.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:45.234 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T03:52:45.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:45.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:45.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:45.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:45.456 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:45.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:45.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:45.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:45.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:45.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:45.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 7.b scrub ok 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 7.7 scrub starts 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T03:52:45.725 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2310804519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:45.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:45.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:52:45.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:45 smithi086 ceph-mon[32965]: osdmap e133: 8 total, 8 up, 7 in 2024-03-26T03:52:45.771 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:44465 7.1 7:83afc1a9:::benchmark_data_smithi006_77260_object44464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 84.941618165999998, 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14001919600000001, 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.772 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 44465 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.773 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827921+0000", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3820000000000001e-06 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827973+0000", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2698999999999998e-05 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827983+0000", 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8139999999999995e-06 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.774 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827991+0000", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4429999999999997e-06 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828043+0000", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2494999999999998e-05 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828177+0000", 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000133635 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.775 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828321+0000", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014464 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.829756+0000", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014350140000000001 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.833618+0000", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038613060000000001 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967795+0000", 2024-03-26T03:52:45.776 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13417760400000001 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967892+0000", 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6203e-05 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967939+0000", 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6961e-05 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.777 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:45151 7.1 7:83b09f16:::benchmark_data_smithi006_77260_object45150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:21.569975+0000", 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 84.199562021000006, 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061131382999999997, 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45151 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.778 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.569975+0000", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.569975+0000", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.569977+0000", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1069999999999999e-06 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570024+0000", 2024-03-26T03:52:45.779 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7307e-05 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570034+0000", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.029e-05 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570040+0000", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9120000000000003e-06 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570094+0000", 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4236e-05 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.780 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570219+0000", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012422799999999999 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.570336+0000", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011716499999999999 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.571996+0000", 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016600790000000001 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.781 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.575135+0000", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031388919999999999 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.631026+0000", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.055890825999999998 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.631071+0000", 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5209e-05 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.782 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.631107+0000", 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6131999999999999e-05 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:45952 7.1 7:88aa6d4d:::benchmark_data_smithi006_77260_object45951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.580185116999999, 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13436482599999999, 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.783 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45952 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.784 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189353+0000", 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0470000000000001e-06 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.785 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189407+0000", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3606999999999997e-05 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189417+0000", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431999999999999e-05 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189424+0000", 2024-03-26T03:52:45.786 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8499999999999996e-06 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189497+0000", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2306999999999997e-05 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189606+0000", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109842 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189710+0000", 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103675 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.787 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.191064+0000", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013539629999999999 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.192039+0000", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097487699999999997 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323621+0000", 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13158181599999999 2024-03-26T03:52:45.788 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323675+0000", 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4469000000000003e-05 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323717+0000", 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1940999999999997e-05 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46143 7.1 7:81fb4aa4:::benchmark_data_smithi006_77260_object46142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:52:45.789 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 83.330939104999999, 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10240083799999999, 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46143 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.790 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438600+0000", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2619999999999999e-06 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438643+0000", 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3565e-05 2024-03-26T03:52:45.791 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438654+0000", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431999999999999e-05 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438661+0000", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6469999999999998e-06 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438717+0000", 2024-03-26T03:52:45.792 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5680000000000002e-05 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438859+0000", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014230699999999999 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439003+0000", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014414000000000001 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.440321+0000", 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001317412 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.793 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.441425+0000", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001103726 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540905+0000", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099480856000000006 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540948+0000", 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3044000000000003e-05 2024-03-26T03:52:45.794 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540999+0000", 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0766999999999997e-05 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46492 7.1 7:829c04db:::benchmark_data_smithi006_77260_object46491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 82.978459138000005, 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14271868800000001, 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.795 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46492 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:52:45.796 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791079+0000", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4300000000000002e-07 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791120+0000", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0858000000000001e-05 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791131+0000", 2024-03-26T03:52:45.797 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0658e-05 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791136+0000", 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7690000000000002e-06 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792218+0000", 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010815589999999999 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792297+0000", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9115000000000004e-05 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792397+0000", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9881000000000001e-05 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.793328+0000", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093134399999999997 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.795379+0000", 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002050765 2024-03-26T03:52:45.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933714+0000", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13833505600000001 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933759+0000", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5204999999999998e-05 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933797+0000", 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7734999999999997e-05 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.800 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:48062 7.1 7:8a572633:::benchmark_data_smithi006_77260_object48061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e110)", 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:24.124491+0000", 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 81.645046868999998, 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.075596016000000002, 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 48062 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.801 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124491+0000", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124491+0000", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124492+0000", 2024-03-26T03:52:45.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1599999999999999e-06 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124539+0000", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6749999999999998e-05 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124549+0000", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0438999999999999e-05 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124555+0000", 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1340000000000001e-06 2024-03-26T03:52:45.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124602+0000", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6795000000000002e-05 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124698+0000", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6047999999999999e-05 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.124804+0000", 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106436 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.126096+0000", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012911940000000001 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.126982+0000", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00088616599999999997 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.199950+0000", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072968174999999996 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.200022+0000", 2024-03-26T03:52:45.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1938000000000001e-05 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:24.200087+0000", 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4781000000000002e-05 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:70989 7.1 7:8446efc3:::benchmark_data_smithi006_77260_object70988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.767870914, 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080342884000000003, 2024-03-26T03:52:45.806 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70989 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:52:45.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001668+0000", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.398e-06 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001716+0000", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8364e-05 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001729+0000", 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2765000000000001e-05 2024-03-26T03:52:45.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001737+0000", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2150000000000001e-06 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001793+0000", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5238000000000002e-05 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001957+0000", 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016436399999999999 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.002145+0000", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000188569 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.004409+0000", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022632310000000001 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.005835+0000", 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001426278 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.081913+0000", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076077991999999997 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.081958+0000", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5354e-05 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.082009+0000", 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1116000000000003e-05 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:72265 7.1 7:8380fa30:::benchmark_data_smithi006_77260_object72264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:45.368477+0000", 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.401060149000003, 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061948688000000002, 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 72265 2024-03-26T03:52:45.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368477+0000", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368477+0000", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368479+0000", 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.19e-06 2024-03-26T03:52:45.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368525+0000", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6519e-05 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368537+0000", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2140999999999999e-05 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368543+0000", 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2910000000000004e-06 2024-03-26T03:52:45.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368586+0000", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2929000000000002e-05 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368713+0000", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000126916 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.368837+0000", 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012356499999999999 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.370811+0000", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019742929999999998 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.372976+0000", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021648520000000001 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.430280+0000", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057303477999999998 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.430372+0000", 2024-03-26T03:52:45.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2291000000000001e-05 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.430426+0000", 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4222999999999999e-05 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:72435 7.1 7:81a747f2:::benchmark_data_smithi006_77260_object72434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:45.581690+0000", 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.187847011000002, 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.067629520999999998, 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.817 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 72435 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581690+0000", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581690+0000", 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581691+0000", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.023e-06 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581749+0000", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7064000000000001e-05 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581759+0000", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0095000000000001e-05 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581764+0000", 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.626e-06 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581805+0000", 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0388e-05 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.581910+0000", 2024-03-26T03:52:45.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000105759 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.582020+0000", 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109442 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.583934+0000", 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019136800000000001 2024-03-26T03:52:45.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.586198+0000", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002264528 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.649230+0000", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063031459999999997 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.649274+0000", 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4042000000000003e-05 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.649320+0000", 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6414e-05 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:72440 7.1 7:844e2348:::benchmark_data_smithi006_77260_object72439:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:45.589090+0000", 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.180447702000002, 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.064767452000000003, 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.824 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 72440 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589090+0000", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589090+0000", 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589091+0000", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3880000000000001e-06 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589141+0000", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9561000000000002e-05 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589151+0000", 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0720000000000001e-05 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589159+0000", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1760000000000004e-06 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589205+0000", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6533999999999999e-05 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589301+0000", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5524000000000002e-05 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.589393+0000", 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2435999999999994e-05 2024-03-26T03:52:45.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591323+0000", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00192958 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.592043+0000", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00071999800000000001 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.653758+0000", 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061715349000000003 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.653800+0000", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1894999999999998e-05 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.653857+0000", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7290999999999997e-05 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:72442 7.1 7:8a4df9ad:::benchmark_data_smithi006_77260_object72441:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:45.591914+0000", 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.177623959000002, 2024-03-26T03:52:45.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063845350999999995, 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 72442 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591914+0000", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591914+0000", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591914+0000", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7599999999999996e-07 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591974+0000", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0123e-05 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591988+0000", 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3217999999999999e-05 2024-03-26T03:52:45.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.591994+0000", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5259999999999999e-06 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.592071+0000", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6380999999999999e-05 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.592157+0000", 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6206999999999996e-05 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.592250+0000", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3682000000000002e-05 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.593534+0000", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012837899999999999 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.593918+0000", 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000383476 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.655690+0000", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061771984000000002 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.655729+0000", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9208000000000001e-05 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:45.655759+0000", 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9980000000000001e-05 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.834 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:74573 7.1 7:845b8743:::benchmark_data_smithi006_77260_object74572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 58.136229692000001, 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10512387099999999, 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74573 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633309+0000", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.313e-06 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633367+0000", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7587000000000003e-05 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633379+0000", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2622e-05 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633386+0000", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3690000000000003e-06 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633439+0000", 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3578000000000001e-05 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633572+0000", 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013307 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633741+0000", 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016872 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.635759+0000", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020182310000000001 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.636437+0000", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00067761600000000003 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738338+0000", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.101901425 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738381+0000", 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2623000000000002e-05 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738432+0000", 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0717000000000002e-05 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:74776 7.1 7:898c7125:::benchmark_data_smithi006_77260_object74775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:47.906384+0000", 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 57.863153738999998, 2024-03-26T03:52:45.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.074268751999999993, 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74776 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906384+0000", 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906384+0000", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906394+0000", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0438999999999999e-05 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906459+0000", 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4721999999999995e-05 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906493+0000", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3593000000000002e-05 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906514+0000", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1963e-05 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906649+0000", 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013427900000000001 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906774+0000", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125458 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.906988+0000", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000213935 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.912125+0000", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051364030000000003 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.912977+0000", 2024-03-26T03:52:45.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00085253399999999995 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980464+0000", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.067487074999999994 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980604+0000", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013971099999999999 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980653+0000", 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8640000000000002e-05 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.845 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:74777 7.1 7:870329a0:::benchmark_data_smithi006_77260_object74776:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:47.907656+0000", 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 57.861881580000002, 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073149500000000006, 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74777 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.846 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907656+0000", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907656+0000", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907657+0000", 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4799999999999997e-07 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907716+0000", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9647e-05 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907726+0000", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.448e-06 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.907732+0000", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9440000000000002e-06 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909093+0000", 2024-03-26T03:52:45.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013611820000000001 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909181+0000", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7808999999999999e-05 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909290+0000", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109782 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.912134+0000", 2024-03-26T03:52:45.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028430970000000002 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.915865+0000", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037312930000000001 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980726+0000", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.064861357999999994 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980775+0000", 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8492999999999998e-05 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.980805+0000", 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0698999999999999e-05 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:74780 7.1 7:871a1a04:::benchmark_data_smithi006_77260_object74779:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:47.909821+0000", 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 57.859716696, 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079794075000000006, 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.851 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74780 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909821+0000", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909821+0000", 2024-03-26T03:52:45.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909821+0000", 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9999999999999997e-07 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909867+0000", 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5133999999999999e-05 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909876+0000", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2620000000000004e-06 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.909883+0000", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5290000000000003e-06 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.910495+0000", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00061202999999999995 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.910590+0000", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4239000000000004e-05 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.910681+0000", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1456e-05 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.912323+0000", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016422450000000001 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.915895+0000", 2024-03-26T03:52:45.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035720019999999999 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.989503+0000", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073607556000000005 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.989568+0000", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4728999999999996e-05 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.989615+0000", 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7193e-05 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.856 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:80429 7.1 7:83134101:::benchmark_data_smithi006_77260_object80428:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:53.500994+0000", 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 52.268543921999999, 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062071283999999997, 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 80429 2024-03-26T03:52:45.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.500994+0000", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.500994+0000", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.500995+0000", 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3340000000000001e-06 2024-03-26T03:52:45.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501199+0000", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020424899999999999 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501213+0000", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3597999999999999e-05 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501222+0000", 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6209999999999999e-06 2024-03-26T03:52:45.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501294+0000", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1250999999999999e-05 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501421+0000", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012747499999999999 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.501564+0000", 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014334100000000001 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.503855+0000", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022908429999999999 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.513021+0000", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0091660419999999992 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.562839+0000", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.049817276000000001 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.563018+0000", 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017974000000000001 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.563065+0000", 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6514000000000002e-05 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:80435 7.1 7:8996d40f:::benchmark_data_smithi006_77260_object80434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:53.510807+0000", 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 52.258730127, 2024-03-26T03:52:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061218036000000003, 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 80435 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510807+0000", 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510807+0000", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510809+0000", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.232e-06 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510876+0000", 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7127999999999998e-05 2024-03-26T03:52:45.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510891+0000", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4806e-05 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510900+0000", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9510000000000001e-06 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.510954+0000", 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3053999999999997e-05 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.511064+0000", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110868 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.511177+0000", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112513 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.513111+0000", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019342490000000001 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.513525+0000", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000414036 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.571951+0000", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.058425904000000001 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.571991+0000", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0074999999999998e-05 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:53.572025+0000", 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4220000000000001e-05 2024-03-26T03:52:45.867 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:84079 7.1 7:87d1b5e4:::benchmark_data_smithi006_77260_object84078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:57.060124+0000", 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 48.709413863999998, 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060178927, 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 84079 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.868 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060124+0000", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060124+0000", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060125+0000", 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.576e-06 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060180+0000", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5142999999999997e-05 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060194+0000", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3213e-05 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060202+0000", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5120000000000008e-06 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060254+0000", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2018999999999998e-05 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060369+0000", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114585 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060470+0000", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101395 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.064101+0000", 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036306620000000002 2024-03-26T03:52:45.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.064935+0000", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00083414899999999996 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120198+0000", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.055262973 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120257+0000", 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8700999999999999e-05 2024-03-26T03:52:45.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120303+0000", 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5998999999999998e-05 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:84080 7.1 7:87c0dd1b:::benchmark_data_smithi006_77260_object84079:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:57.060230+0000", 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 48.709307353, 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.060178976000000002, 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.873 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 84080 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060230+0000", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060230+0000", 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060231+0000", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5400000000000001e-07 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060287+0000", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6242999999999997e-05 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060295+0000", 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9189999999999999e-06 2024-03-26T03:52:45.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.060302+0000", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0740000000000004e-06 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.062295+0000", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019934689999999999 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.062419+0000", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123218 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.062514+0000", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4850000000000002e-05 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.064210+0000", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001696924 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.065029+0000", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00081820099999999995 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120345+0000", 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.055316401000000001 2024-03-26T03:52:45.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120382+0000", 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6525e-05 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:57.120409+0000", 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7498e-05 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.878 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:103403 7.1 7:87b95f78:::benchmark_data_smithi006_77260_object103402:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:16.809571+0000", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 28.959966041000001, 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.069337233999999998, 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 103403 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809571+0000", 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809571+0000", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809572+0000", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.048e-06 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809604+0000", 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1259e-05 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809613+0000", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7440000000000002e-06 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809620+0000", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6669999999999999e-06 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809660+0000", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9957999999999999e-05 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809753+0000", 2024-03-26T03:52:45.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3064000000000002e-05 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.809862+0000", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108465 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.811814+0000", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019520620000000001 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.859464+0000", 2024-03-26T03:52:45.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.047650605999999998 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.878823+0000", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019358354000000001 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.878873+0000", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0207000000000002e-05 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:16.878909+0000", 2024-03-26T03:52:45.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5800000000000003e-05 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:45.884 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:52:45.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:45.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:46.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:46.048 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:46.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 7.b scrub ok 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 7.7 scrub starts 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2310804519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:52:46.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[37501]: osdmap e133: 8 total, 8 up, 7 in 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 7.b scrub ok 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 7.7 scrub starts 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2310804519' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T03:52:46.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:45 smithi006 ceph-mon[29446]: osdmap e133: 8 total, 8 up, 7 in 2024-03-26T03:52:46.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:46.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:46.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:46.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:46.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:46.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:46.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T03:52:46.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:46.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:46.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:46.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:46.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:46.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.730 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:46.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:46.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:46.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:46.902 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:46.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:47.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:47.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:47.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:46 smithi086 ceph-mon[32965]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:52:47.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:46 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:52:47.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:46 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:52:47.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:46 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T03:52:47.044 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:46 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T03:52:47.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:52:47.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[37501]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[29446]: pgmap v159: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T03:52:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:46 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T03:52:47.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:47.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:47.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:47.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:47.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:47.344 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:47.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:47.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:47.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:47.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:47.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:47.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:47.712 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:47.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: 7.6 scrub starts 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: 7.f scrub ok 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:52:47.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:47 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:52:47.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:47.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:47.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:47.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:47.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:47.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:47.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:48.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:48.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:48.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: 7.6 scrub starts 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: 7.f scrub ok 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T03:52:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: 7.6 scrub starts 2024-03-26T03:52:48.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: 7.f scrub ok 2024-03-26T03:52:48.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:52:48.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:47 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:52:48.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:48.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:48.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.376 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:48.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:48.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:48.517 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:48.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:48.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:48.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:48.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:48.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:48.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:48 smithi086 ceph-mon[32965]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T03:52:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:48 smithi086 ceph-mon[32965]: osdmap e134: 8 total, 8 up, 7 in 2024-03-26T03:52:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:48 smithi086 ceph-mon[32965]: 7.7 scrub ok 2024-03-26T03:52:48.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:52:48.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:48.974 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:48.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:49.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:49.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:49.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[37501]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T03:52:49.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[37501]: osdmap e134: 8 total, 8 up, 7 in 2024-03-26T03:52:49.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[37501]: 7.7 scrub ok 2024-03-26T03:52:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[29446]: pgmap v160: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T03:52:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[29446]: osdmap e134: 8 total, 8 up, 7 in 2024-03-26T03:52:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:48 smithi006 ceph-mon[29446]: 7.7 scrub ok 2024-03-26T03:52:49.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:49.245 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:49.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:49.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:49.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:49.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:49.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:49.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:49.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:49.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:49.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:49.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:49.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:49 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T03:52:49.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:49 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T03:52:49.772 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:49 smithi086 ceph-mon[32965]: 7.6 scrub ok 2024-03-26T03:52:49.797 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:52:49.797 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:52:49.797 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.797 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133933 7.0 7:06966e64:::benchmark_data_smithi006_77260_object21579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:52:49.797 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.744530+0000", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.027051355999999999, 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027079932000000001, 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133933 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744530+0000", 2024-03-26T03:52:49.798 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744530+0000", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744531+0000", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6199999999999996e-07 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744538+0000", 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.52e-06 2024-03-26T03:52:49.799 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744547+0000", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2070000000000001e-06 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.744553+0000", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0619999999999997e-06 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.746128+0000", 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001574385 2024-03-26T03:52:49.800 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.746247+0000", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118954 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.746385+0000", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013860899999999999 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.751153+0000", 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047671609999999998 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.801 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.756093+0000", 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049401599999999999 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133936 7.0 7:08fc916b:::benchmark_data_smithi006_77260_object21582:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.748820+0000", 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.022761177, 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022847816, 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:49.802 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133936 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748820+0000", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:49.803 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748820+0000", 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748822+0000", 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9190000000000002e-06 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748829+0000", 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.25e-06 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.804 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748847+0000", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7805e-05 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748856+0000", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3749999999999992e-06 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.748895+0000", 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8179999999999997e-05 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.805 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749007+0000", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112909 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749084+0000", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6266999999999993e-05 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.751294+0000", 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022106690000000002 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.806 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.756128+0000", 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048333509999999996 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133941 7.0 7:074a1219:::benchmark_data_smithi006_77260_object21587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.752118+0000", 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.019463009999999999, 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019605602999999999, 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:52:49.807 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133941 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752118+0000", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752118+0000", 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.808 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752120+0000", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4249999999999999e-06 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752126+0000", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3929999999999996e-06 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752139+0000", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2541000000000001e-05 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752145+0000", 2024-03-26T03:52:49.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2580000000000001e-06 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752200+0000", 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5611e-05 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752282+0000", 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2023999999999999e-05 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:52:49.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752372+0000", 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0068999999999994e-05 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.754651+0000", 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002278843 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.756161+0000", 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001509881 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T03:52:49.811 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 3 2024-03-26T03:52:49.812 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:52:49.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:49.899 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:49.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:49.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:50.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:50.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:50.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:50.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T03:52:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T03:52:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[29446]: 7.6 scrub ok 2024-03-26T03:52:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T03:52:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T03:52:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:49 smithi006 ceph-mon[37501]: 7.6 scrub ok 2024-03-26T03:52:50.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:50.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:50.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:50.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:50.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:50.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:50.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:50.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:50.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:50.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:50.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:50.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:50.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:50.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:50.878 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:50.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[29446]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[37501]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-26T03:52:50.969 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:52:50.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:52:50.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T03:52:50.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:50 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T03:52:51.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:50 smithi086 ceph-mon[32965]: pgmap v162: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 952 op/s 2024-03-26T03:52:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:50 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:52:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:50 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:52:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:50 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T03:52:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:50 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T03:52:51.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:51.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:51.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:51.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:51.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:52:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:51.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:51.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:51.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:51.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:51.576 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:51.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:51.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:51.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:51.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:51.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:51.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:51.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:51.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:51.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T03:52:51.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T03:52:51.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:51 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T03:52:51.957 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:51 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T03:52:51.957 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:51 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T03:52:51.957 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:51 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T03:52:51.957 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:51 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T03:52:51.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:52.000 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:52.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:52.143 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:52.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:52.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:52.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:52.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:52.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:52.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:52.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:52.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:52.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:52.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:52.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.692 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:52.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:52.742 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:52:52.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:52.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:52.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:52.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:53.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:53.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:53.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:52 smithi086 ceph-mon[32965]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:52:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:52 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T03:52:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:52 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T03:52:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:52:53.095 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[29446]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:52:53.095 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T03:52:53.095 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T03:52:53.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:52:53.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[37501]: pgmap v163: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:52:53.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T03:52:53.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T03:52:53.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:52:53.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:53.231 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:53.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:53.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:53.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:53.503 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:53.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:53.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:53.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:53.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:53.719 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:52:53.719 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:52:53.719 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T03:52:53.719 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T03:52:53.719 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:52:53.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:53.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:53.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:53.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: osdmap e135: 8 total, 8 up, 7 in 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:53.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:53.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:53.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:53.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:53.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:54.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:54.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:54.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: osdmap e135: 8 total, 8 up, 7 in 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:54.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3475800899' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: osdmap e135: 8 total, 8 up, 7 in 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:52:54.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:52:54.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:54.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:54.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:54.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:54.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:54.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:54.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:52:54.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:54.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:54.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:52:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:52:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:52:54.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:54.730 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:54.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:54.732 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:52:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:52:54.819 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:54.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:54.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:54.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:54.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:54.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:54.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:54 smithi086 ceph-mon[32965]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s 2024-03-26T03:52:55.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:55.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:54 smithi006 ceph-mon[37501]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s 2024-03-26T03:52:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:54 smithi006 ceph-mon[29446]: pgmap v165: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s 2024-03-26T03:52:55.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:55.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.280 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:55.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:55.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:55.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:55.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:55.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:55.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:55.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:55.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:55.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:55.821 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:55.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:55.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:55.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:55.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:52:55.993 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:55.993 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:56.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:56.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:52:56.136 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:52:56.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:56.186 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:52:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.225 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:56.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:56.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:52:56.311 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:52:56.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:56.363 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:52:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:56.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:56.420 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:52:56.461 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:52:56.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:56.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:56.507 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:52:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:52:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:52:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:52:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:52:56.518 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:52:56.519 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:52:56.519 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:52:56.519 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:52:56.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:56.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:56.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:56.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:56.829 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:52:56.829 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:52:56.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:56.870 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T03:52:56.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:56.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:56.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:57.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.045 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:57.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:56 smithi086 ceph-mon[32965]: pgmap v166: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T03:52:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:56 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1368127212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:57.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:56 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3354152544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:56 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[37501]: pgmap v166: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1368127212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3354152544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[29446]: pgmap v166: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1368127212' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3354152544' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:56 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T03:52:57.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:57.181 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:57.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:57.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:52:57.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:52:57.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:57.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:52:57.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.450 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:57.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:57.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:57.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:57.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:52:57.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:57.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:57.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:57.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:57.882 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:57.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:57.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:52:57.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:58.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:58.076 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:57 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:52:58.076 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:57 smithi086 ceph-mon[32965]: osdmap e136: 8 total, 8 up, 7 in 2024-03-26T03:52:58.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:58.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:57 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:52:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:57 smithi006 ceph-mon[37501]: osdmap e136: 8 total, 8 up, 7 in 2024-03-26T03:52:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:57 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T03:52:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:57 smithi006 ceph-mon[29446]: osdmap e136: 8 total, 8 up, 7 in 2024-03-26T03:52:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:58.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:58.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:52:58.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:52:58.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:58.579 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:58.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T03:52:58.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:52:58.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:52:58.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:52:58 smithi006 ceph-mon[37501]: pgmap v168: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T03:52:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:52:58 smithi006 ceph-mon[29446]: pgmap v168: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T03:52:58.893 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:58.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:58.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:58.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:59.042 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:59.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:52:58 smithi086 ceph-mon[32965]: pgmap v168: 148 pgs: 148 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-26T03:52:59.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:59.150 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:59.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:59.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:59.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:59.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:52:59.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:52:59.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:52:59.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:52:59.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:52:59.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:52:59.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:52:59.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:52:59.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:52:59.934 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:52:59.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:52:59.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:52:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:52:59.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:00.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:00.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:00.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:00.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:00.355 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:00 smithi006 ceph-mon[29446]: pgmap v169: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 717 op/s 2024-03-26T03:53:00.355 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:00 smithi006 ceph-mon[37501]: pgmap v169: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 717 op/s 2024-03-26T03:53:00.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:00.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:00.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:00.477 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:53:00.477 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:53:00.478 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:00 smithi086 ceph-mon[32965]: pgmap v169: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 717 op/s 2024-03-26T03:53:00.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:00.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:00.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:00.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:00.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:00.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.829 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:00.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:00.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:00.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:00.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:00.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:01.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:01.096 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:01.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:53:01.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:01.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:01.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:01.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:53:01.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:01.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:01.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:01.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:01.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:01.585 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:53:01.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:01.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:01.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:01.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:01.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:01.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:01.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:01.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:01.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:02.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:02.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:02.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:02.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:02.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:02.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:02.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:02.353 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:02.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:02.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:02.504 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:02 smithi006 ceph-mon[29446]: pgmap v170: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T03:53:02.504 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:02 smithi006 ceph-mon[37501]: pgmap v170: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T03:53:02.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:02 smithi086 ceph-mon[32965]: pgmap v170: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-26T03:53:02.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:02.609 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:02.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:02.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:02.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:02.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:02.893 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:02.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:02.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:03.035 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:03.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:03.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:03.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:03.273 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4171231648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4171231648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.288 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:53:03.289 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:53:03.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4171231648' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.290 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:03 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:03.291 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T03:53:03.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:03.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:03.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:03.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:03.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:03.586 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:53:03.587 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:53:03.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:03.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:03.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:03.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.832 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:03.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:03.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:03.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:03.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:04.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:04.118 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:04.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:04.263 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:04.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:53:04.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:04.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[37501]: pgmap v171: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:53:04.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[37501]: osdmap e137: 8 total, 8 up, 7 in 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[29446]: pgmap v171: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:53:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-mon[29446]: osdmap e137: 8 total, 8 up, 7 in 2024-03-26T03:53:04.391 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:04.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.404 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:04.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:04.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:04.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-mon[32965]: pgmap v171: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:53:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/793965682' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T03:53:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-mon[32965]: osdmap e137: 8 total, 8 up, 7 in 2024-03-26T03:53:04.588 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:04.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:04.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:04.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:04.663 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T03:53:04.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:53:04.674 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:53:04.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:04.822 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:04.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:04.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:04.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:05.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:05.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:05.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:05.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:05.245 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:05.277 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:05.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:05.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:05.349 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:05.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.427 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:05.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:05.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:05.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:05.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:05.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:05.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:05.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:05.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:05.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:05.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:06.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:06.044 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:53:06.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:06.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:53:06.148 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:53:06.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:06.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:53:06.197 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:53:06.224 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:06.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:06.259 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:53:06.305 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:53:06.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:06.347 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:53:06.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:06 smithi086 ceph-mon[32965]: pgmap v173: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:53:06.355 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/394269609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:06.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.382 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:06.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:06.393 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:53:06.402 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:53:06.406 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:06 smithi006 ceph-mon[37501]: pgmap v173: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:53:06.406 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/394269609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:06.406 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:06 smithi006 ceph-mon[29446]: pgmap v173: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:53:06.407 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/394269609' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:06.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:06.498 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.24367.0:4 2.2 e68/64 2:5cea7035:::rbd_info:head v 68'1, mlcod=0'0)", 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:46.018153+0000", 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 260.478039196, 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015816999999999999, 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018153+0000", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018153+0000", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018153+0000", 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-07 2024-03-26T03:53:06.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018158+0000", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2449999999999998e-06 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018167+0000", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6049999999999998e-06 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018177+0000", 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.04e-05 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018770+0000", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000593145 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.018790+0000", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9998999999999999e-05 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.019675+0000", 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00088507899999999999 2024-03-26T03:53:06.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:46.019734+0000", 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8882999999999998e-05 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:48.015313+0000", 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 258.480879066, 2024-03-26T03:53:06.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012253356, 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015313+0000", 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015313+0000", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015314+0000", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2389999999999999e-06 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015320+0000", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4799999999999998e-06 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015333+0000", 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2262999999999999e-05 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015351+0000", 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8790000000000001e-05 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015440+0000", 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8795000000000006e-05 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.015451+0000", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0427e-05 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.016385+0000", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093465099999999999 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.016393+0000", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1529999999999997e-06 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.022944+0000", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065507969999999997 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023065+0000", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120874 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023200+0000", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134454 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023858+0000", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00065860199999999999 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.025051+0000", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001192371 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027464+0000", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002413583 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027515+0000", 2024-03-26T03:53:06.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1069000000000002e-05 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027566+0000", 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0807999999999999e-05 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.14478.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:48.017052+0000", 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 258.47914024300002, 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010708377999999999, 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.510 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.14478", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3868950920", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017052+0000", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017052+0000", 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017053+0000", 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1340000000000001e-06 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017059+0000", 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9900000000000002e-06 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017069+0000", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0309000000000001e-05 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017078+0000", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8389999999999996e-06 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017163+0000", 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5513999999999995e-05 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.017171+0000", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3010000000000001e-06 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023456+0000", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0062857679999999997 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023528+0000", 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1824999999999998e-05 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.023638+0000", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109404 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.025063+0000", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001425789 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.025935+0000", 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000871838 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027668+0000", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001733011 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027716+0000", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8239999999999999e-05 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027760+0000", 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3415999999999999e-05 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94232629230336] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:48.027866+0000", 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 258.46832546299999, 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023792549999999998, 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 47 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.518 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027866+0000", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027866+0000", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027868+0000", 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.477e-06 2024-03-26T03:53:06.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027880+0000", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2547e-05 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027896+0000", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5452e-05 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027905+0000", 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8789999999999992e-06 2024-03-26T03:53:06.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.027969+0000", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4562999999999998e-05 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028065+0000", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6094000000000005e-05 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028215+0000", 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015014 2024-03-26T03:53:06.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028594+0000", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00037840900000000002 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.029582+0000", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00098791799999999996 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.029980+0000", 2024-03-26T03:53:06.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00039844300000000001 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.030042+0000", 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1597e-05 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.030246+0000", 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020373599999999999 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.14478.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94815156262416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:48.028148+0000", 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 258.46804360200002, 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039018999999999998, 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.14478", 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3868950920", 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 47 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028148+0000", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028148+0000", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028149+0000", 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4099999999999997e-07 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028153+0000", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9659999999999998e-06 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028162+0000", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7020000000000007e-06 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028169+0000", 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6399999999999997e-06 2024-03-26T03:53:06.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028433+0000", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00026338300000000001 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.028505+0000", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2447999999999995e-05 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.030279+0000", 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017738770000000001 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.030345+0000", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6254999999999995e-05 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.030484+0000", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013853800000000001 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.031409+0000", 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092529999999999999 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.528 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.031586+0000", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017722699999999999 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.031924+0000", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00033813799999999998 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.031978+0000", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3547000000000001e-05 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.032050+0000", 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2038000000000004e-05 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.14478.0:191 4.5 4:a2ace8b3:::data_log.0:head [call fifo.create_meta in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:48.078353+0000", 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 258.41783878899997, 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014736479999999999, 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.530 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.14478", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3868950920", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 191 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078353+0000", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078353+0000", 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078354+0000", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9699999999999996e-07 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078357+0000", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5719999999999999e-06 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078364+0000", 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.058e-06 2024-03-26T03:53:06.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078369+0000", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7350000000000004e-06 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078652+0000", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028320899999999998 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.078700+0000", 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7435000000000002e-05 2024-03-26T03:53:06.533 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:48.079827+0000", 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001127042 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.14478.0:208 5.1f 5:f95f44c2:::notify.0:head [watch unwatch cookie 94815156262416] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.049012+0000", 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.44717998100001, 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014389680000000001, 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.534 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.14478", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3868950920", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 208 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049012+0000", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049012+0000", 2024-03-26T03:53:06.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049014+0000", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.103e-06 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049023+0000", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4059999999999994e-06 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049049+0000", 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5497e-05 2024-03-26T03:53:06.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049067+0000", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8428999999999999e-05 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049213+0000", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014554999999999999 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049335+0000", 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012193600000000001 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049483+0000", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014827899999999999 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.049822+0000", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00033939799999999998 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.050287+0000", 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00046437099999999999 2024-03-26T03:53:06.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.050325+0000", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8680999999999998e-05 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.050383+0000", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7328999999999999e-05 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.050451+0000", 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7989000000000002e-05 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.539 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:240 4.1f 4:ffe20264:lc::lc.0:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.088350+0000", 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.40784218100001, 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001573884, 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 240 2024-03-26T03:53:06.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088350+0000", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088350+0000", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088351+0000", 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.023e-06 2024-03-26T03:53:06.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088356+0000", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4389999999999999e-06 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088367+0000", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0900000000000001e-05 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088374+0000", 2024-03-26T03:53:06.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0910000000000004e-06 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088445+0000", 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0965999999999998e-05 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088553+0000", 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108343 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088740+0000", 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000186255 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089360+0000", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00061992299999999998 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089598+0000", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023801599999999999 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089815+0000", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021730400000000001 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089877+0000", 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2448999999999998e-05 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089923+0000", 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6174999999999998e-05 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:241 4.1f 4:ffe20264:lc::lc.0:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.546 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.088500+0000", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.40769169999999, 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001884499, 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 241 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088500+0000", 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.547 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088500+0000", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088501+0000", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2900000000000003e-07 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088505+0000", 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9999999999999998e-06 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088512+0000", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2699999999999999e-06 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088516+0000", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4689999999999999e-06 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.088986+0000", 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00046977800000000001 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.549 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.089072+0000", 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5822000000000006e-05 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090385+0000", 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001312531 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.550 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:255 4.1f 4:ffe20264:lc::lc.0:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.090499+0000", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.40569316599999, 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001314536, 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 255 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090499+0000", 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.551 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.552 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.552 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:06.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090499+0000", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090499+0000", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6600000000000002e-07 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090501+0000", 2024-03-26T03:53:06.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4540000000000001e-06 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090514+0000", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2865e-05 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090519+0000", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2100000000000001e-06 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090626+0000", 2024-03-26T03:53:06.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106506 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090685+0000", 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9126000000000003e-05 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.091813+0000", 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011280089999999999 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:259 4.1f 4:ffe20264:lc::lc.0:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.090729+0000", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.40546311, 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001585719, 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 259 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090729+0000", 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090729+0000", 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090729+0000", 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0299999999999998e-07 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090734+0000", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1690000000000002e-06 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090744+0000", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0511e-05 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090753+0000", 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7129999999999994e-06 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.559 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.090932+0000", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000179409 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.091009+0000", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7149000000000003e-05 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.091171+0000", 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000161781 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.091579+0000", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00040759900000000003 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.091939+0000", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036081600000000002 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.092222+0000", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028235000000000001 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.092273+0000", 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0810999999999999e-05 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.092314+0000", 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1708000000000001e-05 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:415 4.a 4:50d64734:::obj_delete_at_hint.0000000009:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.562 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.122828+0000", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.37336348700001, 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0018462509999999999, 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 415 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122828+0000", 2024-03-26T03:53:06.563 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122828+0000", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122831+0000", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2630000000000002e-06 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122837+0000", 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8360000000000003e-06 2024-03-26T03:53:06.564 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122859+0000", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1569e-05 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122868+0000", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8910000000000001e-06 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.122934+0000", 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6167999999999994e-05 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.565 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.123064+0000", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012977299999999999 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.123275+0000", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000211146 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.123684+0000", 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000408799 2024-03-26T03:53:06.566 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.123925+0000", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000241291 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.124575+0000", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00064985399999999995 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.124634+0000", 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9218000000000001e-05 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.567 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.124675+0000", 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0442999999999999e-05 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:439 4.a 4:55b0279d:reshard::reshard.0000000009:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.128385+0000", 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.367806694, 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017012419999999999, 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.568 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 439 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128385+0000", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128385+0000", 2024-03-26T03:53:06.569 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128386+0000", 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7599999999999996e-07 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128393+0000", 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5690000000000001e-06 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.570 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128402+0000", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8749999999999995e-06 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128408+0000", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7100000000000004e-06 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128461+0000", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3133000000000001e-05 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128574+0000", 2024-03-26T03:53:06.571 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112465 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.128755+0000", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000181731 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.129186+0000", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00043051400000000001 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.129411+0000", 2024-03-26T03:53:06.572 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022534700000000001 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.130009+0000", 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059775599999999996 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.130057+0000", 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7691000000000001e-05 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.573 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.130086+0000", 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9575000000000001e-05 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:475 4.5 4:a4a56432:::obj_delete_at_hint.0000000012:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.138607+0000", 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.35758489, 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013696839999999999, 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.574 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 475 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138607+0000", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138607+0000", 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.575 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138608+0000", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2999999999999999e-07 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138612+0000", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3059999999999999e-06 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138621+0000", 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2289999999999992e-06 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.576 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138627+0000", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8499999999999999e-06 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138687+0000", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0202999999999999e-05 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.138813+0000", 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012592300000000001 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.577 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139020+0000", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000206534 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139401+0000", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00038103799999999999 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139821+0000", 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00041990300000000001 2024-03-26T03:53:06.578 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139888+0000", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7222000000000001e-05 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139938+0000", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9820000000000001e-05 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.139977+0000", 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8825999999999997e-05 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.579 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:495 4.5 4:a7f90313:reshard::reshard.0000000012:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.142206+0000", 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.35398612099999, 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001390718, 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.580 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 495 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142206+0000", 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142206+0000", 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.581 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142207+0000", 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.003e-06 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142212+0000", 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3600000000000004e-06 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142227+0000", 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5362999999999998e-05 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.582 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142236+0000", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6400000000000003e-06 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142284+0000", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8439999999999997e-05 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142422+0000", 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013727800000000001 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.583 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4,7", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142613+0000", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019150699999999999 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.142937+0000", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000323503 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.143415+0000", 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000478092 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.584 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.143505+0000", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0427999999999995e-05 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.143559+0000", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3837e-05 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.143596+0000", 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7267000000000001e-05 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.585 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:531 4.a 4:5395e6f3:lc::lc.4:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.149150+0000", 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.34704138699999, 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014000250000000001, 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 531 2024-03-26T03:53:06.586 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149150+0000", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149150+0000", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149151+0000", 2024-03-26T03:53:06.587 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2099999999999996e-07 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149162+0000", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1133e-05 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149171+0000", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0289999999999994e-06 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149176+0000", 2024-03-26T03:53:06.588 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3489999999999999e-06 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149218+0000", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1369000000000003e-05 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149292+0000", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3895999999999999e-05 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,7", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149438+0000", 2024-03-26T03:53:06.589 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0001462 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.149907+0000", 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000469347 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.150317+0000", 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00040980699999999999 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.590 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.150470+0000", 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015278700000000001 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.150515+0000", 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4864000000000003e-05 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.150550+0000", 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5623000000000001e-05 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.591 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:713 4.1f 4:ffe20264:lc::lc.0:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.193931+0000", 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.30226109899999, 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015989070000000001, 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 713 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.592 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193931+0000", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193931+0000", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193931+0000", 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4899999999999995e-07 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.593 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193934+0000", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.762e-06 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193940+0000", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2899999999999999e-06 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193945+0000", 2024-03-26T03:53:06.594 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2769999999999999e-06 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.193973+0000", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8731e-05 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.194017+0000", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3834000000000001e-05 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.194110+0000", 2024-03-26T03:53:06.595 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2429000000000006e-05 2024-03-26T03:53:06.596 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.596 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.596 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.194424+0000", 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031495600000000003 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.194727+0000", 2024-03-26T03:53:06.597 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000302501 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.195460+0000", 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00073289899999999996 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.195496+0000", 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5976999999999998e-05 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.598 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.195530+0000", 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3701999999999997e-05 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:947 4.1f 4:f93251ce:::obj_delete_at_hint.0000000042:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.245763+0000", 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.25042845300001, 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001738627, 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.599 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 947 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245763+0000", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245763+0000", 2024-03-26T03:53:06.600 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245764+0000", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0999999999999997e-07 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245769+0000", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9640000000000002e-06 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245780+0000", 2024-03-26T03:53:06.601 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1328000000000001e-05 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245788+0000", 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7519999999999996e-06 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245846+0000", 2024-03-26T03:53:06.602 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7627000000000002e-05 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.245976+0000", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130617 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,5", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.246173+0000", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019708099999999999 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.246509+0000", 2024-03-26T03:53:06.603 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00033578999999999998 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.246736+0000", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000227149 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.247433+0000", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00069647600000000002 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.247466+0000", 2024-03-26T03:53:06.604 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3201000000000003e-05 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.247502+0000", 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5831999999999998e-05 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.24349.0:1031 4.8 e73/67 4:10010070:lc::lc.29:head v 73'23, mlcod=73'21)", 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:48:51.286801+0000", 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 255.20939059299999, 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001478011, 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.605 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286801+0000", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286801+0000", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286802+0000", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8400000000000004e-07 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286808+0000", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7400000000000001e-06 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286820+0000", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1977999999999999e-05 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286827+0000", 2024-03-26T03:53:06.607 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5340000000000002e-06 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286887+0000", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0353000000000003e-05 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.286908+0000", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0698000000000001e-05 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.288243+0000", 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013351979999999999 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.608 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:48:51.288279+0000", 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5825999999999999e-05 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(6.1e,QREQUEST,e:93,from:4,reservation_nonce:3)", 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:50:35.962092+0000", 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 150.534100096, 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021882260000000001, 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:06.609 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962092+0000", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962092+0000", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962093+0000", 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2669999999999999e-06 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.610 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962099+0000", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8980000000000001e-06 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962113+0000", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4392e-05 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962121+0000", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2330000000000006e-06 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962190+0000", 2024-03-26T03:53:06.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8344999999999997e-05 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.962195+0000", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4750000000000001e-06 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.964211+0000", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020157439999999999 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.964233+0000", 2024-03-26T03:53:06.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1756999999999999e-05 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:35.964280+0000", 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7114999999999999e-05 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:06.613 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:53:06.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:06.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:06.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:06.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:06.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:06.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:06.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:07.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:07.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:07.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:07.237 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:07.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:07.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:07.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:07.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:07.506 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:07.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:07.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:53:07.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[37501]: osdmap e138: 8 total, 8 up, 7 in 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:07.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:07 smithi006 ceph-mon[29446]: osdmap e138: 8 total, 8 up, 7 in 2024-03-26T03:53:07.644 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:07.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:07.789 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:07.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:07 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:07 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:07 smithi086 ceph-mon[32965]: osdmap e138: 8 total, 8 up, 7 in 2024-03-26T03:53:07.848 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:53:07.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:07.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:07.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:07.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:08.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:08.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:08.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:08.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:08.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:08.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.334 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:08.391 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:08.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:08.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:08.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:08.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:08.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:08 smithi086 ceph-mon[32965]: pgmap v175: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:53:08.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:08.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:08.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:08 smithi006 ceph-mon[37501]: pgmap v175: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:53:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:08 smithi006 ceph-mon[29446]: pgmap v175: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.17k op/s 2024-03-26T03:53:08.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:08.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:08.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:08.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:08.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:08.908 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:53:08.916 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:53:08.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:08.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:08.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:09.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:09.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:09.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:09.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:09.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:09.418 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:09.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:09.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:09.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:09.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:09.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:09.664 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:09.704 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:09.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:153892 7.5 7:a7f2ce71:::benchmark_data_smithi006_77260_object41538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:09.795863+0000", 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.017221999000000002, 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017253675, 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:09.815 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 153892 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795863+0000", 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795863+0000", 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795863+0000", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3099999999999998e-07 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795866+0000", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.993e-06 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795873+0000", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.126e-06 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.795880+0000", 2024-03-26T03:53:09.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4139999999999999e-06 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.798103+0000", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022230599999999998 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.798251+0000", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014829399999999999 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.798402+0000", 2024-03-26T03:53:09.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015117200000000001 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.802510+0000", 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041077320000000002 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807770+0000", 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052597759999999999 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:153895 7.5 7:a7f0aba4:::benchmark_data_smithi006_77260_object41541:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:09.797386+0000", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.015699008, 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015829481999999999, 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 153895 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:09.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797386+0000", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797386+0000", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797387+0000", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9800000000000002e-07 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797394+0000", 2024-03-26T03:53:09.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7640000000000005e-06 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797406+0000", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1593000000000001e-05 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.797414+0000", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1210000000000007e-06 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.800581+0000", 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031672549999999999 2024-03-26T03:53:09.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.800711+0000", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012998900000000001 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.800853+0000", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141817 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.802640+0000", 2024-03-26T03:53:09.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017872739999999999 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.24704.0:153906 7.8 e138/95 7:11676f9d:::benchmark_data_smithi006_77260_object41552:head v 138'9664, mlcod=138'9660)", 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:09.807747+0000", 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0053378119999999999, 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055525180000000002, 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807747+0000", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807747+0000", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807747+0000", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.48e-07 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807751+0000", 2024-03-26T03:53:09.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5899999999999999e-06 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807766+0000", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5755000000000001e-05 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807772+0000", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6380000000000001e-06 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810179+0000", 2024-03-26T03:53:09.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024068420000000002 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810216+0000", 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7440000000000001e-05 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:153908 7.5 7:a4a68c0d:::benchmark_data_smithi006_77260_object41554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:09.807838+0000", 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0052465020000000001, 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055267550000000004, 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:09.827 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 153908 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807838+0000", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807838+0000", 2024-03-26T03:53:09.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807846+0000", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0949999999999996e-06 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807852+0000", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9920000000000004e-06 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807870+0000", 2024-03-26T03:53:09.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7901e-05 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807881+0000", 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0743e-05 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.807930+0000", 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8866000000000003e-05 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:09.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.808062+0000", 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000132719 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.808194+0000", 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013116199999999999 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810485+0000", 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022915539999999999 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.24704.0:153907 7.8 e138/95 7:11e8fd02:::benchmark_data_smithi006_77260_object41553:head v 138'9665, mlcod=138'9661)", 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:09.810850+0000", 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.002234205, 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002595998, 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810850+0000", 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810850+0000", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810851+0000", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9999999999999996e-07 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810857+0000", 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2809999999999997e-06 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810866+0000", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4319999999999999e-06 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.810874+0000", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6850000000000001e-06 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.812532+0000", 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016581639999999999 2024-03-26T03:53:09.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:09.812550+0000", 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8491e-05 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 5 2024-03-26T03:53:09.835 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:53:09.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:53:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:09.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:09.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:09.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:09.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:09.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:10.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:10.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:10.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:10.252 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:10.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:10.327 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":175,"stamp":"2024-03-26T03:53:07.244771+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4984609216,"num_objects":76281,"num_object_clones":0,"num_object_copies":228843,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76281,"num_whiteouts":0,"num_read":1473,"num_read_kb":1287,"num_write":149460,"num_write_kb":7192926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2362855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76381,"ondisk_log_size":76381,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15441484,"kb_used_data":14785072,"kb_used_omap":10,"kb_used_meta":655477,"kb_avail":640774580,"statfs":{"total":671965249536,"available":656153169920,"internally_reserved":0,"allocated":15139913728,"data_stored":15137331417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":671208554},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-611909632,"num_objects":-9337,"num_object_clones":0,"num_object_copies":-28011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9337,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":9337,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003305"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279207+0000","last_change":"2024-03-26T03:50:34.035051+0000","last_active":"2024-03-26T03:53:03.279207+0000","last_peered":"2024-03-26T03:53:03.279207+0000","last_clean":"2024-03-26T03:53:03.279207+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:53:03.279207+0000","last_undegraded":"2024-03-26T03:53:03.279207+0000","last_fullsized":"2024-03-26T03:53:03.279207+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:34.034948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:34.034948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497754+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":161,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.282776+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:53:03.282776+0000","last_peered":"2024-03-26T03:53:03.282776+0000","last_clean":"2024-03-26T03:53:03.282776+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:53:03.282776+0000","last_undegraded":"2024-03-26T03:53:03.282776+0000","last_fullsized":"2024-03-26T03:53:03.282776+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.054186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.298805+0000","last_change":"2024-03-26T03:51:39.331101+0000","last_active":"2024-03-26T03:53:03.298805+0000","last_peered":"2024-03-26T03:53:03.298805+0000","last_clean":"2024-03-26T03:53:03.298805+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:53:03.298805+0000","last_undegraded":"2024-03-26T03:53:03.298805+0000","last_fullsized":"2024-03-26T03:53:03.298805+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.330345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729168+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":212,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279004+0000","last_change":"2024-03-26T03:52:33.804342+0000","last_active":"2024-03-26T03:53:03.279004+0000","last_peered":"2024-03-26T03:53:03.279004+0000","last_clean":"2024-03-26T03:53:03.279004+0000","last_became_active":"2024-03-26T03:50:45.598168+0000","last_became_peered":"2024-03-26T03:50:45.598168+0000","last_unstale":"2024-03-26T03:53:03.279004+0000","last_undegraded":"2024-03-26T03:53:03.279004+0000","last_fullsized":"2024-03-26T03:53:03.279004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.024936+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":164,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275433+0000","last_change":"2024-03-26T03:52:43.442740+0000","last_active":"2024-03-26T03:53:03.275433+0000","last_peered":"2024-03-26T03:53:03.275433+0000","last_clean":"2024-03-26T03:53:03.275433+0000","last_became_active":"2024-03-26T03:50:45.597941+0000","last_became_peered":"2024-03-26T03:50:45.597941+0000","last_unstale":"2024-03-26T03:53:03.275433+0000","last_undegraded":"2024-03-26T03:53:03.275433+0000","last_fullsized":"2024-03-26T03:53:03.275433+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.003941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278939+0000","last_change":"2024-03-26T03:50:45.601271+0000","last_active":"2024-03-26T03:53:03.278939+0000","last_peered":"2024-03-26T03:53:03.278939+0000","last_clean":"2024-03-26T03:53:03.278939+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:53:03.278939+0000","last_undegraded":"2024-03-26T03:53:03.278939+0000","last_fullsized":"2024-03-26T03:53:03.278939+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:32.005669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:32.005669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497678+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":169,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.288401+0000","last_change":"2024-03-26T03:51:21.399264+0000","last_active":"2024-03-26T03:53:03.288401+0000","last_peered":"2024-03-26T03:53:03.288401+0000","last_clean":"2024-03-26T03:53:03.288401+0000","last_became_active":"2024-03-26T03:50:45.581989+0000","last_became_peered":"2024-03-26T03:50:45.581989+0000","last_unstale":"2024-03-26T03:53:03.288401+0000","last_undegraded":"2024-03-26T03:53:03.288401+0000","last_fullsized":"2024-03-26T03:53:03.288401+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.398535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.398535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411802+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":231,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275306+0000","last_change":"2024-03-26T03:50:40.221494+0000","last_active":"2024-03-26T03:53:03.275306+0000","last_peered":"2024-03-26T03:53:03.275306+0000","last_clean":"2024-03-26T03:53:03.275306+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:53:03.275306+0000","last_undegraded":"2024-03-26T03:53:03.275306+0000","last_fullsized":"2024-03-26T03:53:03.275306+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:50:40.221401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:40.221401+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411358+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.294990+0000","last_change":"2024-03-26T03:51:17.173504+0000","last_active":"2024-03-26T03:53:03.294990+0000","last_peered":"2024-03-26T03:53:03.294990+0000","last_clean":"2024-03-26T03:53:03.294990+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:53:03.294990+0000","last_undegraded":"2024-03-26T03:53:03.294990+0000","last_fullsized":"2024-03-26T03:53:03.294990+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:17.173394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.173394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729004+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269036+0000","last_change":"2024-03-26T03:51:23.617976+0000","last_active":"2024-03-26T03:53:03.269036+0000","last_peered":"2024-03-26T03:53:03.269036+0000","last_clean":"2024-03-26T03:53:03.269036+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:53:03.269036+0000","last_undegraded":"2024-03-26T03:53:03.269036+0000","last_fullsized":"2024-03-26T03:53:03.269036+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.617899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.617899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446571+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269667+0000","last_change":"2024-03-26T03:51:18.480480+0000","last_active":"2024-03-26T03:53:03.269667+0000","last_peered":"2024-03-26T03:53:03.269667+0000","last_clean":"2024-03-26T03:53:03.269667+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:53:03.269667+0000","last_undegraded":"2024-03-26T03:53:03.269667+0000","last_fullsized":"2024-03-26T03:53:03.269667+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.480360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.480360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":207,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279373+0000","last_change":"2024-03-26T03:51:27.245595+0000","last_active":"2024-03-26T03:53:03.279373+0000","last_peered":"2024-03-26T03:53:03.279373+0000","last_clean":"2024-03-26T03:53:03.279373+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:53:03.279373+0000","last_undegraded":"2024-03-26T03:53:03.279373+0000","last_fullsized":"2024-03-26T03:53:03.279373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:27.245475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.245475+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497892+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.268752+0000","last_change":"2024-03-26T03:52:26.427599+0000","last_active":"2024-03-26T03:53:03.268752+0000","last_peered":"2024-03-26T03:53:03.268752+0000","last_clean":"2024-03-26T03:53:03.268752+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:53:03.268752+0000","last_undegraded":"2024-03-26T03:53:03.268752+0000","last_fullsized":"2024-03-26T03:53:03.268752+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.427501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.427501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.945072+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286261+0000","last_change":"2024-03-26T03:52:52.666181+0000","last_active":"2024-03-26T03:53:03.286261+0000","last_peered":"2024-03-26T03:53:03.286261+0000","last_clean":"2024-03-26T03:53:03.286261+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:53:03.286261+0000","last_undegraded":"2024-03-26T03:53:03.286261+0000","last_fullsized":"2024-03-26T03:53:03.286261+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.666107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.666107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.715049+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269301+0000","last_change":"2024-03-26T03:52:28.376930+0000","last_active":"2024-03-26T03:53:03.269301+0000","last_peered":"2024-03-26T03:53:03.269301+0000","last_clean":"2024-03-26T03:53:03.269301+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:53:03.269301+0000","last_undegraded":"2024-03-26T03:53:03.269301+0000","last_fullsized":"2024-03-26T03:53:03.269301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:28.376798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:28.376798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:53.233172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":191,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.280133+0000","last_change":"2024-03-26T03:52:38.915695+0000","last_active":"2024-03-26T03:53:03.280133+0000","last_peered":"2024-03-26T03:53:03.280133+0000","last_clean":"2024-03-26T03:53:03.280133+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:53:03.280133+0000","last_undegraded":"2024-03-26T03:53:03.280133+0000","last_fullsized":"2024-03-26T03:53:03.280133+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.912180+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":165,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279678+0000","last_change":"2024-03-26T03:52:51.658215+0000","last_active":"2024-03-26T03:53:03.279678+0000","last_peered":"2024-03-26T03:53:03.279678+0000","last_clean":"2024-03-26T03:53:03.279678+0000","last_became_active":"2024-03-26T03:50:45.588396+0000","last_became_peered":"2024-03-26T03:50:45.588396+0000","last_unstale":"2024-03-26T03:53:03.279678+0000","last_undegraded":"2024-03-26T03:53:03.279678+0000","last_fullsized":"2024-03-26T03:53:03.279678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:17.128768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":153,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279514+0000","last_change":"2024-03-26T03:51:15.157751+0000","last_active":"2024-03-26T03:53:03.279514+0000","last_peered":"2024-03-26T03:53:03.279514+0000","last_clean":"2024-03-26T03:53:03.279514+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:53:03.279514+0000","last_undegraded":"2024-03-26T03:53:03.279514+0000","last_fullsized":"2024-03-26T03:53:03.279514+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:15.157477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.157477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686370+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296763+0000","last_change":"2024-03-26T03:51:24.313200+0000","last_active":"2024-03-26T03:53:03.296763+0000","last_peered":"2024-03-26T03:53:03.296763+0000","last_clean":"2024-03-26T03:53:03.296763+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:53:03.296763+0000","last_undegraded":"2024-03-26T03:53:03.296763+0000","last_fullsized":"2024-03-26T03:53:03.296763+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:24.312536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:24.312536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.728943+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":199,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.284954+0000","last_change":"2024-03-26T03:52:45.404305+0000","last_active":"2024-03-26T03:53:03.284954+0000","last_peered":"2024-03-26T03:53:03.284954+0000","last_clean":"2024-03-26T03:53:03.284954+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:53:03.284954+0000","last_undegraded":"2024-03-26T03:53:03.284954+0000","last_fullsized":"2024-03-26T03:53:03.284954+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:45.403401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.403401+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.513250+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":157,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.277982+0000","last_change":"2024-03-26T03:51:38.222500+0000","last_active":"2024-03-26T03:53:03.277982+0000","last_peered":"2024-03-26T03:53:03.277982+0000","last_clean":"2024-03-26T03:53:03.277982+0000","last_became_active":"2024-03-26T03:50:45.591442+0000","last_became_peered":"2024-03-26T03:50:45.591442+0000","last_unstale":"2024-03-26T03:53:03.277982+0000","last_undegraded":"2024-03-26T03:53:03.277982+0000","last_fullsized":"2024-03-26T03:53:03.277982+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.222375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.047554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.281497+0000","last_change":"2024-03-26T03:52:49.690105+0000","last_active":"2024-03-26T03:53:03.281497+0000","last_peered":"2024-03-26T03:53:03.281497+0000","last_clean":"2024-03-26T03:53:03.281497+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:53:03.281497+0000","last_undegraded":"2024-03-26T03:53:03.281497+0000","last_fullsized":"2024-03-26T03:53:03.281497+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:49.690025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.690025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.171459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279392+0000","last_change":"2024-03-26T03:51:13.376075+0000","last_active":"2024-03-26T03:53:03.279392+0000","last_peered":"2024-03-26T03:53:03.279392+0000","last_clean":"2024-03-26T03:53:03.279392+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:53:03.279392+0000","last_undegraded":"2024-03-26T03:53:03.279392+0000","last_fullsized":"2024-03-26T03:53:03.279392+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:13.375964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:13.375964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497786+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":201,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.274984+0000","last_change":"2024-03-26T03:52:35.473032+0000","last_active":"2024-03-26T03:53:03.274984+0000","last_peered":"2024-03-26T03:53:03.274984+0000","last_clean":"2024-03-26T03:53:03.274984+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:53:03.274984+0000","last_undegraded":"2024-03-26T03:53:03.274984+0000","last_fullsized":"2024-03-26T03:53:03.274984+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.269829+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287070+0000","last_change":"2024-03-26T03:52:23.708646+0000","last_active":"2024-03-26T03:53:03.287070+0000","last_peered":"2024-03-26T03:53:03.287070+0000","last_clean":"2024-03-26T03:53:03.287070+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:53:03.287070+0000","last_undegraded":"2024-03-26T03:53:03.287070+0000","last_fullsized":"2024-03-26T03:53:03.287070+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:52:23.708510+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.708510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:25.385430+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":164,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279835+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:53:03.279835+0000","last_peered":"2024-03-26T03:53:03.279835+0000","last_clean":"2024-03-26T03:53:03.279835+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:53:03.279835+0000","last_undegraded":"2024-03-26T03:53:03.279835+0000","last_fullsized":"2024-03-26T03:53:03.279835+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.226521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.282633+0000","last_change":"2024-03-26T03:50:45.116437+0000","last_active":"2024-03-26T03:53:03.282633+0000","last_peered":"2024-03-26T03:53:03.282633+0000","last_clean":"2024-03-26T03:53:03.282633+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:53:03.282633+0000","last_undegraded":"2024-03-26T03:53:03.282633+0000","last_fullsized":"2024-03-26T03:53:03.282633+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:50:45.116355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:50:45.116355+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686283+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":194,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297978+0000","last_change":"2024-03-26T03:52:42.591753+0000","last_active":"2024-03-26T03:53:03.297978+0000","last_peered":"2024-03-26T03:53:03.297978+0000","last_clean":"2024-03-26T03:53:03.297978+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:53:03.297978+0000","last_undegraded":"2024-03-26T03:53:03.297978+0000","last_fullsized":"2024-03-26T03:53:03.297978+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:52:42.591503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.591503+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.339508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":157,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279159+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:53:03.279159+0000","last_peered":"2024-03-26T03:53:03.279159+0000","last_clean":"2024-03-26T03:53:03.279159+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:53:03.279159+0000","last_undegraded":"2024-03-26T03:53:03.279159+0000","last_fullsized":"2024-03-26T03:53:03.279159+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:07.860041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.298339+0000","last_change":"2024-03-26T03:51:33.548319+0000","last_active":"2024-03-26T03:53:03.298339+0000","last_peered":"2024-03-26T03:53:03.298339+0000","last_clean":"2024-03-26T03:53:03.298339+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:53:03.298339+0000","last_undegraded":"2024-03-26T03:53:03.298339+0000","last_fullsized":"2024-03-26T03:53:03.298339+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.548192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729194+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.274596+0000","last_change":"2024-03-26T03:51:35.260411+0000","last_active":"2024-03-26T03:53:03.274596+0000","last_peered":"2024-03-26T03:53:03.274596+0000","last_clean":"2024-03-26T03:53:03.274596+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:53:03.274596+0000","last_undegraded":"2024-03-26T03:53:03.274596+0000","last_fullsized":"2024-03-26T03:53:03.274596+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.258167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.258167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411852+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":175,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270208+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:53:03.270208+0000","last_peered":"2024-03-26T03:53:03.270208+0000","last_clean":"2024-03-26T03:53:03.270208+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:53:03.270208+0000","last_undegraded":"2024-03-26T03:53:03.270208+0000","last_fullsized":"2024-03-26T03:53:03.270208+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.838957+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286156+0000","last_change":"2024-03-26T03:52:34.605587+0000","last_active":"2024-03-26T03:53:03.286156+0000","last_peered":"2024-03-26T03:53:03.286156+0000","last_clean":"2024-03-26T03:53:03.286156+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:53:03.286156+0000","last_undegraded":"2024-03-26T03:53:03.286156+0000","last_fullsized":"2024-03-26T03:53:03.286156+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:34.605509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:34.605509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.539945+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":100,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279639+0000","last_change":"2024-03-26T03:51:39.355828+0000","last_active":"2024-03-26T03:53:03.279639+0000","last_peered":"2024-03-26T03:53:03.279639+0000","last_clean":"2024-03-26T03:53:03.279639+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:53:03.279639+0000","last_undegraded":"2024-03-26T03:53:03.279639+0000","last_fullsized":"2024-03-26T03:53:03.279639+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.355728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.355728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498139+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.299017+0000","last_change":"2024-03-26T03:52:23.578870+0000","last_active":"2024-03-26T03:53:03.299017+0000","last_peered":"2024-03-26T03:53:03.299017+0000","last_clean":"2024-03-26T03:53:03.299017+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:53:03.299017+0000","last_undegraded":"2024-03-26T03:53:03.299017+0000","last_fullsized":"2024-03-26T03:53:03.299017+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.989992+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.274700+0000","last_change":"2024-03-26T03:50:45.606641+0000","last_active":"2024-03-26T03:53:03.274700+0000","last_peered":"2024-03-26T03:53:03.274700+0000","last_clean":"2024-03-26T03:53:03.274700+0000","last_became_active":"2024-03-26T03:50:45.587808+0000","last_became_peered":"2024-03-26T03:50:45.587808+0000","last_unstale":"2024-03-26T03:53:03.274700+0000","last_undegraded":"2024-03-26T03:53:03.274700+0000","last_fullsized":"2024-03-26T03:53:03.274700+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-03-26T03:50:35.236237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.236237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411443+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275626+0000","last_change":"2024-03-26T03:50:45.590379+0000","last_active":"2024-03-26T03:53:03.275626+0000","last_peered":"2024-03-26T03:53:03.275626+0000","last_clean":"2024-03-26T03:53:03.275626+0000","last_became_active":"2024-03-26T03:50:45.589899+0000","last_became_peered":"2024-03-26T03:50:45.589899+0000","last_unstale":"2024-03-26T03:53:03.275626+0000","last_undegraded":"2024-03-26T03:53:03.275626+0000","last_fullsized":"2024-03-26T03:53:03.275626+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.094208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":168,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287641+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:53:03.287641+0000","last_peered":"2024-03-26T03:53:03.287641+0000","last_clean":"2024-03-26T03:53:03.287641+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:53:03.287641+0000","last_undegraded":"2024-03-26T03:53:03.287641+0000","last_fullsized":"2024-03-26T03:53:03.287641+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.913379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":178,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270004+0000","last_change":"2024-03-26T03:52:39.291143+0000","last_active":"2024-03-26T03:53:03.270004+0000","last_peered":"2024-03-26T03:53:03.270004+0000","last_clean":"2024-03-26T03:53:03.270004+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:53:03.270004+0000","last_undegraded":"2024-03-26T03:53:03.270004+0000","last_fullsized":"2024-03-26T03:53:03.270004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:39.291063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.291063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:55.327602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":215,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275203+0000","last_change":"2024-03-26T03:50:45.612604+0000","last_active":"2024-03-26T03:53:03.275203+0000","last_peered":"2024-03-26T03:53:03.275203+0000","last_clean":"2024-03-26T03:53:03.275203+0000","last_became_active":"2024-03-26T03:50:45.595599+0000","last_became_peered":"2024-03-26T03:50:45.595599+0000","last_unstale":"2024-03-26T03:53:03.275203+0000","last_undegraded":"2024-03-26T03:53:03.275203+0000","last_fullsized":"2024-03-26T03:53:03.275203+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.135174+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411295+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.294565+0000","last_change":"2024-03-26T03:52:26.596430+0000","last_active":"2024-03-26T03:53:03.294565+0000","last_peered":"2024-03-26T03:53:03.294565+0000","last_clean":"2024-03-26T03:53:03.294565+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:53:03.294565+0000","last_undegraded":"2024-03-26T03:53:03.294565+0000","last_fullsized":"2024-03-26T03:53:03.294565+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.596311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.596311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.308150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275344+0000","last_change":"2024-03-26T03:52:40.410756+0000","last_active":"2024-03-26T03:53:03.275344+0000","last_peered":"2024-03-26T03:53:03.275344+0000","last_clean":"2024-03-26T03:53:03.275344+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:53:03.275344+0000","last_undegraded":"2024-03-26T03:53:03.275344+0000","last_fullsized":"2024-03-26T03:53:03.275344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.947379+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286728+0000","last_change":"2024-03-26T03:51:15.916808+0000","last_active":"2024-03-26T03:53:03.286728+0000","last_peered":"2024-03-26T03:53:03.286728+0000","last_clean":"2024-03-26T03:53:03.286728+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:53:03.286728+0000","last_undegraded":"2024-03-26T03:53:03.286728+0000","last_fullsized":"2024-03-26T03:53:03.286728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:15.916730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.916730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653079+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":197,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278949+0000","last_change":"2024-03-26T03:51:29.988370+0000","last_active":"2024-03-26T03:53:03.278949+0000","last_peered":"2024-03-26T03:53:03.278949+0000","last_clean":"2024-03-26T03:53:03.278949+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:53:03.278949+0000","last_undegraded":"2024-03-26T03:53:03.278949+0000","last_fullsized":"2024-03-26T03:53:03.278949+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:51:29.988111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.988111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686541+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.274801+0000","last_change":"2024-03-26T03:52:41.432452+0000","last_active":"2024-03-26T03:53:03.274801+0000","last_peered":"2024-03-26T03:53:03.274801+0000","last_clean":"2024-03-26T03:53:03.274801+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:53:03.274801+0000","last_undegraded":"2024-03-26T03:53:03.274801+0000","last_fullsized":"2024-03-26T03:53:03.274801+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:41.432332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:41.432332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.168460+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":161,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278823+0000","last_change":"2024-03-26T03:52:40.835637+0000","last_active":"2024-03-26T03:53:03.278823+0000","last_peered":"2024-03-26T03:53:03.278823+0000","last_clean":"2024-03-26T03:53:03.278823+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:53:03.278823+0000","last_undegraded":"2024-03-26T03:53:03.278823+0000","last_fullsized":"2024-03-26T03:53:03.278823+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.905532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.281407+0000","last_change":"2024-03-26T03:51:30.319510+0000","last_active":"2024-03-26T03:53:03.281407+0000","last_peered":"2024-03-26T03:53:03.281407+0000","last_clean":"2024-03-26T03:53:03.281407+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:53:03.281407+0000","last_undegraded":"2024-03-26T03:53:03.281407+0000","last_fullsized":"2024-03-26T03:53:03.281407+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:30.319423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.319423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497815+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":195,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.280332+0000","last_change":"2024-03-26T03:51:17.380156+0000","last_active":"2024-03-26T03:53:03.280332+0000","last_peered":"2024-03-26T03:53:03.280332+0000","last_clean":"2024-03-26T03:53:03.280332+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:53:03.280332+0000","last_undegraded":"2024-03-26T03:53:03.280332+0000","last_fullsized":"2024-03-26T03:53:03.280332+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:51:17.380041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.380041+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497857+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297484+0000","last_change":"2024-03-26T03:51:19.175698+0000","last_active":"2024-03-26T03:53:03.297484+0000","last_peered":"2024-03-26T03:53:03.297484+0000","last_clean":"2024-03-26T03:53:03.297484+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:53:03.297484+0000","last_undegraded":"2024-03-26T03:53:03.297484+0000","last_fullsized":"2024-03-26T03:53:03.297484+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.175455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.175455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729118+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":231,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270120+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:53:03.270120+0000","last_peered":"2024-03-26T03:53:03.270120+0000","last_clean":"2024-03-26T03:53:03.270120+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:53:03.270120+0000","last_undegraded":"2024-03-26T03:53:03.270120+0000","last_fullsized":"2024-03-26T03:53:03.270120+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.768552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286472+0000","last_change":"2024-03-26T03:51:14.953771+0000","last_active":"2024-03-26T03:53:03.286472+0000","last_peered":"2024-03-26T03:53:03.286472+0000","last_clean":"2024-03-26T03:53:03.286472+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:53:03.286472+0000","last_undegraded":"2024-03-26T03:53:03.286472+0000","last_fullsized":"2024-03-26T03:53:03.286472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:14.953632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:14.953632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653044+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":197,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275609+0000","last_change":"2024-03-26T03:51:42.190694+0000","last_active":"2024-03-26T03:53:03.275609+0000","last_peered":"2024-03-26T03:53:03.275609+0000","last_clean":"2024-03-26T03:53:03.275609+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:53:03.275609+0000","last_undegraded":"2024-03-26T03:53:03.275609+0000","last_fullsized":"2024-03-26T03:53:03.275609+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:42.190557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:42.190557+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.464058+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279893+0000","last_change":"2024-03-26T03:51:34.417542+0000","last_active":"2024-03-26T03:53:03.279893+0000","last_peered":"2024-03-26T03:53:03.279893+0000","last_clean":"2024-03-26T03:53:03.279893+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:53:03.279893+0000","last_undegraded":"2024-03-26T03:53:03.279893+0000","last_fullsized":"2024-03-26T03:53:03.279893+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:34.417463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.417463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":215,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.276285+0000","last_change":"2024-03-26T03:51:34.258348+0000","last_active":"2024-03-26T03:53:03.276285+0000","last_peered":"2024-03-26T03:53:03.276285+0000","last_clean":"2024-03-26T03:53:03.276285+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:53:03.276285+0000","last_undegraded":"2024-03-26T03:53:03.276285+0000","last_fullsized":"2024-03-26T03:53:03.276285+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.464095+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296547+0000","last_change":"2024-03-26T03:50:35.194524+0000","last_active":"2024-03-26T03:53:03.296547+0000","last_peered":"2024-03-26T03:53:03.296547+0000","last_clean":"2024-03-26T03:53:03.296547+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:53:03.296547+0000","last_undegraded":"2024-03-26T03:53:03.296547+0000","last_fullsized":"2024-03-26T03:53:03.296547+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.194368+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.728914+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":199,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269971+0000","last_change":"2024-03-26T03:51:36.601210+0000","last_active":"2024-03-26T03:53:03.269971+0000","last_peered":"2024-03-26T03:53:03.269971+0000","last_clean":"2024-03-26T03:53:03.269971+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:53:03.269971+0000","last_undegraded":"2024-03-26T03:53:03.269971+0000","last_fullsized":"2024-03-26T03:53:03.269971+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:51:36.601123+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:50:20.001688+0000","last_clean_scrub_stamp":"2024-03-26T03:51:36.601123+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446631+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":160,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285589+0000","last_change":"2024-03-26T03:51:34.179163+0000","last_active":"2024-03-26T03:53:03.285589+0000","last_peered":"2024-03-26T03:53:03.285589+0000","last_clean":"2024-03-26T03:53:03.285589+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:53:03.285589+0000","last_undegraded":"2024-03-26T03:53:03.285589+0000","last_fullsized":"2024-03-26T03:53:03.285589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:51:34.179031+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.179031+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":154,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.277840+0000","last_change":"2024-03-26T03:50:24.093555+0000","last_active":"2024-03-26T03:53:03.277840+0000","last_peered":"2024-03-26T03:53:03.277840+0000","last_clean":"2024-03-26T03:53:03.277840+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:53:03.277840+0000","last_undegraded":"2024-03-26T03:53:03.277840+0000","last_fullsized":"2024-03-26T03:53:03.277840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:24.093421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:24.093421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286261+0000","last_change":"2024-03-26T03:51:19.966491+0000","last_active":"2024-03-26T03:53:03.286261+0000","last_peered":"2024-03-26T03:53:03.286261+0000","last_clean":"2024-03-26T03:53:03.286261+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:53:03.286261+0000","last_undegraded":"2024-03-26T03:53:03.286261+0000","last_fullsized":"2024-03-26T03:53:03.286261+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.966387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.966387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653111+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":206,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.298189+0000","last_change":"2024-03-26T03:52:37.650277+0000","last_active":"2024-03-26T03:53:03.298189+0000","last_peered":"2024-03-26T03:53:03.298189+0000","last_clean":"2024-03-26T03:53:03.298189+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:53:03.298189+0000","last_undegraded":"2024-03-26T03:53:03.298189+0000","last_fullsized":"2024-03-26T03:53:03.298189+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.102642+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286645+0000","last_change":"2024-03-26T03:50:45.595675+0000","last_active":"2024-03-26T03:53:03.286645+0000","last_peered":"2024-03-26T03:53:03.286645+0000","last_clean":"2024-03-26T03:53:03.286645+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:53:03.286645+0000","last_undegraded":"2024-03-26T03:53:03.286645+0000","last_fullsized":"2024-03-26T03:53:03.286645+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:28.691326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:28.691326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.652988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270083+0000","last_change":"2024-03-26T03:51:38.612538+0000","last_active":"2024-03-26T03:53:03.270083+0000","last_peered":"2024-03-26T03:53:03.270083+0000","last_clean":"2024-03-26T03:53:03.270083+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:53:03.270083+0000","last_undegraded":"2024-03-26T03:53:03.270083+0000","last_fullsized":"2024-03-26T03:53:03.270083+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.612390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.612390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446682+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":100,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.294866+0000","last_change":"2024-03-26T03:51:38.327055+0000","last_active":"2024-03-26T03:53:03.294866+0000","last_peered":"2024-03-26T03:53:03.294866+0000","last_clean":"2024-03-26T03:53:03.294866+0000","last_became_active":"2024-03-26T03:50:45.593133+0000","last_became_peered":"2024-03-26T03:50:45.593133+0000","last_unstale":"2024-03-26T03:53:03.294866+0000","last_undegraded":"2024-03-26T03:53:03.294866+0000","last_fullsized":"2024-03-26T03:53:03.294866+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.326923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.326923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729143+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":204,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278084+0000","last_change":"2024-03-26T03:52:33.466952+0000","last_active":"2024-03-26T03:53:03.278084+0000","last_peered":"2024-03-26T03:53:03.278084+0000","last_clean":"2024-03-26T03:53:03.278084+0000","last_became_active":"2024-03-26T03:50:24.531528+0000","last_became_peered":"2024-03-26T03:50:24.531528+0000","last_unstale":"2024-03-26T03:53:03.278084+0000","last_undegraded":"2024-03-26T03:53:03.278084+0000","last_fullsized":"2024-03-26T03:53:03.278084+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:48.107634+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"137'9104","reported_seq":9266,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.307139+0000","last_change":"2024-03-26T03:52:44.145650+0000","last_active":"2024-03-26T03:53:04.307139+0000","last_peered":"2024-03-26T03:53:04.307139+0000","last_clean":"2024-03-26T03:53:04.307139+0000","last_became_active":"2024-03-26T03:50:45.595232+0000","last_became_peered":"2024-03-26T03:50:45.595232+0000","last_unstale":"2024-03-26T03:53:04.307139+0000","last_undegraded":"2024-03-26T03:53:04.307139+0000","last_fullsized":"2024-03-26T03:53:04.307139+0000","mapping_epoch":96,"log_start":"118'4900","ondisk_log_start":"118'4900","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'7919","last_scrub_stamp":"2024-03-26T03:52:44.145396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.145396+0000","objects_scrubbed":5918,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.921446+0000","scrub_duration":2302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9104,"num_write_kb":440960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269320+0000","last_change":"2024-03-26T03:52:50.408368+0000","last_active":"2024-03-26T03:53:03.269320+0000","last_peered":"2024-03-26T03:53:03.269320+0000","last_clean":"2024-03-26T03:53:03.269320+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:53:03.269320+0000","last_undegraded":"2024-03-26T03:53:03.269320+0000","last_fullsized":"2024-03-26T03:53:03.269320+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.408254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.408254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.251022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":182,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287669+0000","last_change":"2024-03-26T03:52:32.622645+0000","last_active":"2024-03-26T03:53:03.287669+0000","last_peered":"2024-03-26T03:53:03.287669+0000","last_clean":"2024-03-26T03:53:03.287669+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:53:03.287669+0000","last_undegraded":"2024-03-26T03:53:03.287669+0000","last_fullsized":"2024-03-26T03:53:03.287669+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:52:32.622512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.622512+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.145922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":242,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.268756+0000","last_change":"2024-03-26T03:52:22.438654+0000","last_active":"2024-03-26T03:53:03.268756+0000","last_peered":"2024-03-26T03:53:03.268756+0000","last_clean":"2024-03-26T03:53:03.268756+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:53:03.268756+0000","last_undegraded":"2024-03-26T03:53:03.268756+0000","last_fullsized":"2024-03-26T03:53:03.268756+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:52:22.438565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:22.438565+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.719143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":233,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286499+0000","last_change":"2024-03-26T03:51:20.930418+0000","last_active":"2024-03-26T03:53:03.286499+0000","last_peered":"2024-03-26T03:53:03.286499+0000","last_clean":"2024-03-26T03:53:03.286499+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:53:03.286499+0000","last_undegraded":"2024-03-26T03:53:03.286499+0000","last_fullsized":"2024-03-26T03:53:03.286499+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:51:20.930317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:20.930317+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"137'9152","reported_seq":9244,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.439142+0000","last_change":"2024-03-26T03:50:44.592931+0000","last_active":"2024-03-26T03:53:04.439142+0000","last_peered":"2024-03-26T03:53:04.439142+0000","last_clean":"2024-03-26T03:53:04.439142+0000","last_became_active":"2024-03-26T03:50:44.592128+0000","last_became_peered":"2024-03-26T03:50:44.592128+0000","last_unstale":"2024-03-26T03:53:04.439142+0000","last_undegraded":"2024-03-26T03:53:04.439142+0000","last_fullsized":"2024-03-26T03:53:04.439142+0000","mapping_epoch":95,"log_start":"118'4900","ondisk_log_start":"118'4900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4252,"log_dups_size":0,"ondisk_log_size":4252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314441728,"num_objects":4798,"num_object_clones":0,"num_object_copies":14394,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4798,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9152,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275527+0000","last_change":"2024-03-26T03:51:23.414926+0000","last_active":"2024-03-26T03:53:03.275527+0000","last_peered":"2024-03-26T03:53:03.275527+0000","last_clean":"2024-03-26T03:53:03.275527+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:53:03.275527+0000","last_undegraded":"2024-03-26T03:53:03.275527+0000","last_fullsized":"2024-03-26T03:53:03.275527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287067+0000","last_change":"2024-03-26T03:52:31.638104+0000","last_active":"2024-03-26T03:53:03.287067+0000","last_peered":"2024-03-26T03:53:03.287067+0000","last_clean":"2024-03-26T03:53:03.287067+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:53:03.287067+0000","last_undegraded":"2024-03-26T03:53:03.287067+0000","last_fullsized":"2024-03-26T03:53:03.287067+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.637987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.637987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.891585+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":213,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.280963+0000","last_change":"2024-03-26T03:52:30.697090+0000","last_active":"2024-03-26T03:53:03.280963+0000","last_peered":"2024-03-26T03:53:03.280963+0000","last_clean":"2024-03-26T03:53:03.280963+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:53:03.280963+0000","last_undegraded":"2024-03-26T03:53:03.280963+0000","last_fullsized":"2024-03-26T03:53:03.280963+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.990440+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.284486+0000","last_change":"2024-03-26T03:51:36.110972+0000","last_active":"2024-03-26T03:53:03.284486+0000","last_peered":"2024-03-26T03:53:03.284486+0000","last_clean":"2024-03-26T03:53:03.284486+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:53:03.284486+0000","last_undegraded":"2024-03-26T03:53:03.284486+0000","last_fullsized":"2024-03-26T03:53:03.284486+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:36.110583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:36.110583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"137'9592","reported_seq":9762,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.426638+0000","last_change":"2024-03-26T03:52:36.221752+0000","last_active":"2024-03-26T03:53:04.426638+0000","last_peered":"2024-03-26T03:53:04.426638+0000","last_clean":"2024-03-26T03:53:04.426638+0000","last_became_active":"2024-03-26T03:50:44.585858+0000","last_became_peered":"2024-03-26T03:50:44.585858+0000","last_unstale":"2024-03-26T03:53:04.426638+0000","last_undegraded":"2024-03-26T03:53:04.426638+0000","last_fullsized":"2024-03-26T03:53:04.426638+0000","mapping_epoch":95,"log_start":"118'4800","ondisk_log_start":"118'4800","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7838","last_scrub_stamp":"2024-03-26T03:52:36.221537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:36.221537+0000","objects_scrubbed":6697,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.562362+0000","scrub_duration":2663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317849600,"num_objects":4850,"num_object_clones":0,"num_object_copies":14550,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4850,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9592,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287293+0000","last_change":"2024-03-26T03:51:27.913279+0000","last_active":"2024-03-26T03:53:03.287293+0000","last_peered":"2024-03-26T03:53:03.287293+0000","last_clean":"2024-03-26T03:53:03.287293+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:53:03.287293+0000","last_undegraded":"2024-03-26T03:53:03.287293+0000","last_fullsized":"2024-03-26T03:53:03.287293+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:27.913164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.913164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653142+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287357+0000","last_change":"2024-03-26T03:52:29.251669+0000","last_active":"2024-03-26T03:53:03.287357+0000","last_peered":"2024-03-26T03:53:03.287357+0000","last_clean":"2024-03-26T03:53:03.287357+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:53:03.287357+0000","last_undegraded":"2024-03-26T03:53:03.287357+0000","last_fullsized":"2024-03-26T03:53:03.287357+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.251258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.251258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.082401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286958+0000","last_change":"2024-03-26T03:52:35.600199+0000","last_active":"2024-03-26T03:53:03.286958+0000","last_peered":"2024-03-26T03:53:03.286958+0000","last_clean":"2024-03-26T03:53:03.286958+0000","last_became_active":"2024-03-26T03:50:45.594682+0000","last_became_peered":"2024-03-26T03:50:45.594682+0000","last_unstale":"2024-03-26T03:53:03.286958+0000","last_undegraded":"2024-03-26T03:53:03.286958+0000","last_fullsized":"2024-03-26T03:53:03.286958+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:35.600091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.600091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:57.681359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":185,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278753+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:53:03.278753+0000","last_peered":"2024-03-26T03:53:03.278753+0000","last_clean":"2024-03-26T03:53:03.278753+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:53:03.278753+0000","last_undegraded":"2024-03-26T03:53:03.278753+0000","last_fullsized":"2024-03-26T03:53:03.278753+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.717740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"137'9353","reported_seq":9523,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.328592+0000","last_change":"2024-03-26T03:52:33.625525+0000","last_active":"2024-03-26T03:53:04.328592+0000","last_peered":"2024-03-26T03:53:04.328592+0000","last_clean":"2024-03-26T03:53:04.328592+0000","last_became_active":"2024-03-26T03:50:44.589615+0000","last_became_peered":"2024-03-26T03:50:44.589615+0000","last_unstale":"2024-03-26T03:53:04.328592+0000","last_undegraded":"2024-03-26T03:53:04.328592+0000","last_fullsized":"2024-03-26T03:53:04.328592+0000","mapping_epoch":95,"log_start":"114'3800","ondisk_log_start":"114'3800","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7504","last_scrub_stamp":"2024-03-26T03:52:33.625304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.625304+0000","objects_scrubbed":6650,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.340680+0000","scrub_duration":2355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309526528,"num_objects":4723,"num_object_clones":0,"num_object_copies":14169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9353,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296150+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:53:03.296150+0000","last_peered":"2024-03-26T03:53:03.296150+0000","last_clean":"2024-03-26T03:53:03.296150+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:53:03.296150+0000","last_undegraded":"2024-03-26T03:53:03.296150+0000","last_fullsized":"2024-03-26T03:53:03.296150+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.605441+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285239+0000","last_change":"2024-03-26T03:52:42.382709+0000","last_active":"2024-03-26T03:53:03.285239+0000","last_peered":"2024-03-26T03:53:03.285239+0000","last_clean":"2024-03-26T03:53:03.285239+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:53:03.285239+0000","last_undegraded":"2024-03-26T03:53:03.285239+0000","last_fullsized":"2024-03-26T03:53:03.285239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.382447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.382447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:08.004037+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269401+0000","last_change":"2024-03-26T03:52:31.320265+0000","last_active":"2024-03-26T03:53:03.269401+0000","last_peered":"2024-03-26T03:53:03.269401+0000","last_clean":"2024-03-26T03:53:03.269401+0000","last_became_active":"2024-03-26T03:50:45.592202+0000","last_became_peered":"2024-03-26T03:50:45.592202+0000","last_unstale":"2024-03-26T03:53:03.269401+0000","last_undegraded":"2024-03-26T03:53:03.269401+0000","last_fullsized":"2024-03-26T03:53:03.269401+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.320181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.320181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:37.000450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":212,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297188+0000","last_change":"2024-03-26T03:51:34.486120+0000","last_active":"2024-03-26T03:53:03.297188+0000","last_peered":"2024-03-26T03:53:03.297188+0000","last_clean":"2024-03-26T03:53:03.297188+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:53:03.297188+0000","last_undegraded":"2024-03-26T03:53:03.297188+0000","last_fullsized":"2024-03-26T03:53:03.297188+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:34.485979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.485979+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"137'9352","reported_seq":9513,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.437690+0000","last_change":"2024-03-26T03:52:48.148490+0000","last_active":"2024-03-26T03:53:04.437690+0000","last_peered":"2024-03-26T03:53:04.437690+0000","last_clean":"2024-03-26T03:53:04.437690+0000","last_became_active":"2024-03-26T03:50:45.590035+0000","last_became_peered":"2024-03-26T03:50:45.590035+0000","last_unstale":"2024-03-26T03:53:04.437690+0000","last_undegraded":"2024-03-26T03:53:04.437690+0000","last_fullsized":"2024-03-26T03:53:04.437690+0000","mapping_epoch":96,"log_start":"118'4500","ondisk_log_start":"118'4500","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8409","last_scrub_stamp":"2024-03-26T03:52:48.148171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.148171+0000","objects_scrubbed":5895,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.767336+0000","scrub_duration":3557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319553536,"num_objects":4876,"num_object_clones":0,"num_object_copies":14628,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4876,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9352,"num_write_kb":455296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.288717+0000","last_change":"2024-03-26T03:51:25.463769+0000","last_active":"2024-03-26T03:53:03.288717+0000","last_peered":"2024-03-26T03:53:03.288717+0000","last_clean":"2024-03-26T03:53:03.288717+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:53:03.288717+0000","last_undegraded":"2024-03-26T03:53:03.288717+0000","last_fullsized":"2024-03-26T03:53:03.288717+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.463664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.463664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411777+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":192,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269104+0000","last_change":"2024-03-26T03:51:41.572876+0000","last_active":"2024-03-26T03:53:03.269104+0000","last_peered":"2024-03-26T03:53:03.269104+0000","last_clean":"2024-03-26T03:53:03.269104+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:53:03.269104+0000","last_undegraded":"2024-03-26T03:53:03.269104+0000","last_fullsized":"2024-03-26T03:53:03.269104+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:51:41.572734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.572734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:09.496149+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":177,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279986+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:53:03.279986+0000","last_peered":"2024-03-26T03:53:03.279986+0000","last_clean":"2024-03-26T03:53:03.279986+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:53:03.279986+0000","last_undegraded":"2024-03-26T03:53:03.279986+0000","last_fullsized":"2024-03-26T03:53:03.279986+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.516195+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":279,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285779+0000","last_change":"2024-03-26T03:51:34.973521+0000","last_active":"2024-03-26T03:53:03.285779+0000","last_peered":"2024-03-26T03:53:03.285779+0000","last_clean":"2024-03-26T03:53:03.285779+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:53:03.285779+0000","last_undegraded":"2024-03-26T03:53:03.285779+0000","last_fullsized":"2024-03-26T03:53:03.285779+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:51:34.973366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.973366+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686481+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297740+0000","last_change":"2024-03-26T03:52:25.587745+0000","last_active":"2024-03-26T03:53:03.297740+0000","last_peered":"2024-03-26T03:53:03.297740+0000","last_clean":"2024-03-26T03:53:03.297740+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:53:03.297740+0000","last_undegraded":"2024-03-26T03:53:03.297740+0000","last_fullsized":"2024-03-26T03:53:03.297740+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.587502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.587502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:25.609331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"137'9275","reported_seq":9443,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.455251+0000","last_change":"2024-03-26T03:52:33.050130+0000","last_active":"2024-03-26T03:53:04.455251+0000","last_peered":"2024-03-26T03:53:04.455251+0000","last_clean":"2024-03-26T03:53:04.455251+0000","last_became_active":"2024-03-26T03:50:44.584865+0000","last_became_peered":"2024-03-26T03:50:44.584865+0000","last_unstale":"2024-03-26T03:53:04.455251+0000","last_undegraded":"2024-03-26T03:53:04.455251+0000","last_fullsized":"2024-03-26T03:53:04.455251+0000","mapping_epoch":95,"log_start":"118'4500","ondisk_log_start":"118'4500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7431","last_scrub_stamp":"2024-03-26T03:52:33.049847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.049847+0000","objects_scrubbed":6680,"log_size":4775,"log_dups_size":0,"ondisk_log_size":4775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.994227+0000","scrub_duration":2449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":313065472,"num_objects":4777,"num_object_clones":0,"num_object_copies":14331,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4777,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9275,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":165,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287178+0000","last_change":"2024-03-26T03:52:44.692881+0000","last_active":"2024-03-26T03:53:03.287178+0000","last_peered":"2024-03-26T03:53:03.287178+0000","last_clean":"2024-03-26T03:53:03.287178+0000","last_became_active":"2024-03-26T03:50:45.583936+0000","last_became_peered":"2024-03-26T03:50:45.583936+0000","last_unstale":"2024-03-26T03:53:03.287178+0000","last_undegraded":"2024-03-26T03:53:03.287178+0000","last_fullsized":"2024-03-26T03:53:03.287178+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.026793+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279360+0000","last_change":"2024-03-26T03:51:32.007583+0000","last_active":"2024-03-26T03:53:03.279360+0000","last_peered":"2024-03-26T03:53:03.279360+0000","last_clean":"2024-03-26T03:53:03.279360+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:53:03.279360+0000","last_undegraded":"2024-03-26T03:53:03.279360+0000","last_fullsized":"2024-03-26T03:53:03.279360+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.007458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.007458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686421+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":188,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.284567+0000","last_change":"2024-03-26T03:52:40.285553+0000","last_active":"2024-03-26T03:53:03.284567+0000","last_peered":"2024-03-26T03:53:03.284567+0000","last_clean":"2024-03-26T03:53:03.284567+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:53:03.284567+0000","last_undegraded":"2024-03-26T03:53:03.284567+0000","last_fullsized":"2024-03-26T03:53:03.284567+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.285400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.285400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.875607+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":204,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278577+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:53:03.278577+0000","last_peered":"2024-03-26T03:53:03.278577+0000","last_clean":"2024-03-26T03:53:03.278577+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:53:03.278577+0000","last_undegraded":"2024-03-26T03:53:03.278577+0000","last_fullsized":"2024-03-26T03:53:03.278577+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.706675+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286634+0000","last_change":"2024-03-26T03:52:25.178369+0000","last_active":"2024-03-26T03:53:03.286634+0000","last_peered":"2024-03-26T03:53:03.286634+0000","last_clean":"2024-03-26T03:53:03.286634+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:53:03.286634+0000","last_undegraded":"2024-03-26T03:53:03.286634+0000","last_fullsized":"2024-03-26T03:53:03.286634+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.178239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.178239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.907344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"137'9224","reported_seq":9383,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.432957+0000","last_change":"2024-03-26T03:52:49.471047+0000","last_active":"2024-03-26T03:53:04.432957+0000","last_peered":"2024-03-26T03:53:04.432957+0000","last_clean":"2024-03-26T03:53:04.432957+0000","last_became_active":"2024-03-26T03:50:44.588356+0000","last_became_peered":"2024-03-26T03:50:44.588356+0000","last_unstale":"2024-03-26T03:53:04.432957+0000","last_undegraded":"2024-03-26T03:53:04.432957+0000","last_fullsized":"2024-03-26T03:53:04.432957+0000","mapping_epoch":95,"log_start":"118'4400","ondisk_log_start":"118'4400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8243","last_scrub_stamp":"2024-03-26T03:52:49.470707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.470707+0000","objects_scrubbed":5693,"log_size":4824,"log_dups_size":0,"ondisk_log_size":4824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.948331+0000","scrub_duration":2903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303431680,"num_objects":4630,"num_object_clones":0,"num_object_copies":13890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9224,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269662+0000","last_change":"2024-03-26T03:51:30.713391+0000","last_active":"2024-03-26T03:53:03.269662+0000","last_peered":"2024-03-26T03:53:03.269662+0000","last_clean":"2024-03-26T03:53:03.269662+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:53:03.269662+0000","last_undegraded":"2024-03-26T03:53:03.269662+0000","last_fullsized":"2024-03-26T03:53:03.269662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:30.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446600+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":434,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287383+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:53:03.287383+0000","last_peered":"2024-03-26T03:53:03.287383+0000","last_clean":"2024-03-26T03:53:03.287383+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:53:03.287383+0000","last_undegraded":"2024-03-26T03:53:03.287383+0000","last_fullsized":"2024-03-26T03:53:03.287383+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:14.983642+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":169,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.276701+0000","last_change":"2024-03-26T03:51:18.413053+0000","last_active":"2024-03-26T03:53:03.276701+0000","last_peered":"2024-03-26T03:53:03.276701+0000","last_clean":"2024-03-26T03:53:03.276701+0000","last_became_active":"2024-03-26T03:50:45.592343+0000","last_became_peered":"2024-03-26T03:50:45.592343+0000","last_unstale":"2024-03-26T03:53:03.276701+0000","last_undegraded":"2024-03-26T03:53:03.276701+0000","last_fullsized":"2024-03-26T03:53:03.276701+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.412940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.412940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411675+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":190,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279460+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:53:03.279460+0000","last_peered":"2024-03-26T03:53:03.279460+0000","last_clean":"2024-03-26T03:53:03.279460+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:53:03.279460+0000","last_undegraded":"2024-03-26T03:53:03.279460+0000","last_fullsized":"2024-03-26T03:53:03.279460+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.072965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287219+0000","last_change":"2024-03-26T03:51:39.840265+0000","last_active":"2024-03-26T03:53:03.287219+0000","last_peered":"2024-03-26T03:53:03.287219+0000","last_clean":"2024-03-26T03:53:03.287219+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:53:03.287219+0000","last_undegraded":"2024-03-26T03:53:03.287219+0000","last_fullsized":"2024-03-26T03:53:03.287219+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.840138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.840138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653095+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"137'9444","reported_seq":9539,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.135355+0000","last_change":"2024-03-26T03:50:44.586811+0000","last_active":"2024-03-26T03:53:04.135355+0000","last_peered":"2024-03-26T03:53:04.135355+0000","last_clean":"2024-03-26T03:53:04.135355+0000","last_became_active":"2024-03-26T03:50:44.586504+0000","last_became_peered":"2024-03-26T03:50:44.586504+0000","last_unstale":"2024-03-26T03:53:04.135355+0000","last_undegraded":"2024-03-26T03:53:04.135355+0000","last_fullsized":"2024-03-26T03:53:04.135355+0000","mapping_epoch":95,"log_start":"116'4400","ondisk_log_start":"116'4400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":5044,"log_dups_size":0,"ondisk_log_size":5044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316407808,"num_objects":4828,"num_object_clones":0,"num_object_copies":14484,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4828,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9444,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":150,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285424+0000","last_change":"2024-03-26T03:50:37.125164+0000","last_active":"2024-03-26T03:53:03.285424+0000","last_peered":"2024-03-26T03:53:03.285424+0000","last_clean":"2024-03-26T03:53:03.285424+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:53:03.285424+0000","last_undegraded":"2024-03-26T03:53:03.285424+0000","last_fullsized":"2024-03-26T03:53:03.285424+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:37.125085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:37.125085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686313+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.295851+0000","last_change":"2024-03-26T03:51:40.382951+0000","last_active":"2024-03-26T03:53:03.295851+0000","last_peered":"2024-03-26T03:53:03.295851+0000","last_clean":"2024-03-26T03:53:03.295851+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:53:03.295851+0000","last_undegraded":"2024-03-26T03:53:03.295851+0000","last_fullsized":"2024-03-26T03:53:03.295851+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:40.377234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:40.377234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729093+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":436,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.287436+0000","last_change":"2024-03-26T03:51:17.909032+0000","last_active":"2024-03-26T03:53:03.287436+0000","last_peered":"2024-03-26T03:53:03.287436+0000","last_clean":"2024-03-26T03:53:03.287436+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:53:03.287436+0000","last_undegraded":"2024-03-26T03:53:03.287436+0000","last_fullsized":"2024-03-26T03:53:03.287436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:51:17.908906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.908906+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653061+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":253,"num_read_kb":248,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":184,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297708+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:53:03.297708+0000","last_peered":"2024-03-26T03:53:03.297708+0000","last_clean":"2024-03-26T03:53:03.297708+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:53:03.297708+0000","last_undegraded":"2024-03-26T03:53:03.297708+0000","last_fullsized":"2024-03-26T03:53:03.297708+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.887919+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270320+0000","last_change":"2024-03-26T03:52:25.471379+0000","last_active":"2024-03-26T03:53:03.270320+0000","last_peered":"2024-03-26T03:53:03.270320+0000","last_clean":"2024-03-26T03:53:03.270320+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:53:03.270320+0000","last_undegraded":"2024-03-26T03:53:03.270320+0000","last_fullsized":"2024-03-26T03:53:03.270320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.471302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.471302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:32.035263+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"137'9452","reported_seq":9623,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.185368+0000","last_change":"2024-03-26T03:52:31.921264+0000","last_active":"2024-03-26T03:53:04.185368+0000","last_peered":"2024-03-26T03:53:04.185368+0000","last_clean":"2024-03-26T03:53:04.185368+0000","last_became_active":"2024-03-26T03:50:44.593854+0000","last_became_peered":"2024-03-26T03:50:44.593854+0000","last_unstale":"2024-03-26T03:53:04.185368+0000","last_undegraded":"2024-03-26T03:53:04.185368+0000","last_fullsized":"2024-03-26T03:53:04.185368+0000","mapping_epoch":95,"log_start":"115'4000","ondisk_log_start":"115'4000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7468","last_scrub_stamp":"2024-03-26T03:52:31.921157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.921157+0000","objects_scrubbed":6796,"log_size":5452,"log_dups_size":0,"ondisk_log_size":5452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.185172+0000","scrub_duration":1567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312868864,"num_objects":4774,"num_object_clones":0,"num_object_copies":14322,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4774,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9452,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":164,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.282859+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:53:03.282859+0000","last_peered":"2024-03-26T03:53:03.282859+0000","last_clean":"2024-03-26T03:53:03.282859+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:53:03.282859+0000","last_undegraded":"2024-03-26T03:53:03.282859+0000","last_fullsized":"2024-03-26T03:53:03.282859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.478472+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285643+0000","last_change":"2024-03-26T03:51:20.065167+0000","last_active":"2024-03-26T03:53:03.285643+0000","last_peered":"2024-03-26T03:53:03.285643+0000","last_clean":"2024-03-26T03:53:03.285643+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:53:03.285643+0000","last_undegraded":"2024-03-26T03:53:03.285643+0000","last_fullsized":"2024-03-26T03:53:03.285643+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:20.065038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:20.065038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686396+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":215,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296931+0000","last_change":"2024-03-26T03:51:32.540653+0000","last_active":"2024-03-26T03:53:03.296931+0000","last_peered":"2024-03-26T03:53:03.296931+0000","last_clean":"2024-03-26T03:53:03.296931+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:53:03.296931+0000","last_undegraded":"2024-03-26T03:53:03.296931+0000","last_fullsized":"2024-03-26T03:53:03.296931+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:51:32.540268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.540268+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":101,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296467+0000","last_change":"2024-03-26T03:51:41.399750+0000","last_active":"2024-03-26T03:53:03.296467+0000","last_peered":"2024-03-26T03:53:03.296467+0000","last_clean":"2024-03-26T03:53:03.296467+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:53:03.296467+0000","last_undegraded":"2024-03-26T03:53:03.296467+0000","last_fullsized":"2024-03-26T03:53:03.296467+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.399622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.399622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"137'9294","reported_seq":9388,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.167798+0000","last_change":"2024-03-26T03:50:45.597482+0000","last_active":"2024-03-26T03:53:04.167798+0000","last_peered":"2024-03-26T03:53:04.167798+0000","last_clean":"2024-03-26T03:53:04.167798+0000","last_became_active":"2024-03-26T03:50:45.596978+0000","last_became_peered":"2024-03-26T03:50:45.596978+0000","last_unstale":"2024-03-26T03:53:04.167798+0000","last_undegraded":"2024-03-26T03:53:04.167798+0000","last_fullsized":"2024-03-26T03:53:04.167798+0000","mapping_epoch":96,"log_start":"118'4700","ondisk_log_start":"118'4700","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312475648,"num_objects":4768,"num_object_clones":0,"num_object_copies":14304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9294,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":151,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.294662+0000","last_change":"2024-03-26T03:51:18.180126+0000","last_active":"2024-03-26T03:53:03.294662+0000","last_peered":"2024-03-26T03:53:03.294662+0000","last_clean":"2024-03-26T03:53:03.294662+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:53:03.294662+0000","last_undegraded":"2024-03-26T03:53:03.294662+0000","last_fullsized":"2024-03-26T03:53:03.294662+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.179908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.179908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.728969+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269824+0000","last_change":"2024-03-26T03:52:29.399095+0000","last_active":"2024-03-26T03:53:03.269824+0000","last_peered":"2024-03-26T03:53:03.269824+0000","last_clean":"2024-03-26T03:53:03.269824+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:53:03.269824+0000","last_undegraded":"2024-03-26T03:53:03.269824+0000","last_fullsized":"2024-03-26T03:53:03.269824+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.398965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.398965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.994443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":193,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278848+0000","last_change":"2024-03-26T03:51:17.091083+0000","last_active":"2024-03-26T03:53:03.278848+0000","last_peered":"2024-03-26T03:53:03.278848+0000","last_clean":"2024-03-26T03:53:03.278848+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:53:03.278848+0000","last_undegraded":"2024-03-26T03:53:03.278848+0000","last_fullsized":"2024-03-26T03:53:03.278848+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:51:17.090921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.090921+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686451+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":226,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.296706+0000","last_change":"2024-03-26T03:51:15.205153+0000","last_active":"2024-03-26T03:53:03.296706+0000","last_peered":"2024-03-26T03:53:03.296706+0000","last_clean":"2024-03-26T03:53:03.296706+0000","last_became_active":"2024-03-26T03:50:45.602990+0000","last_became_peered":"2024-03-26T03:50:45.602990+0000","last_unstale":"2024-03-26T03:53:03.296706+0000","last_undegraded":"2024-03-26T03:53:03.296706+0000","last_fullsized":"2024-03-26T03:53:03.296706+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:51:15.205025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.205025+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729063+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"137'9189","reported_seq":9284,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.385715+0000","last_change":"2024-03-26T03:50:44.586742+0000","last_active":"2024-03-26T03:53:04.385715+0000","last_peered":"2024-03-26T03:53:04.385715+0000","last_clean":"2024-03-26T03:53:04.385715+0000","last_became_active":"2024-03-26T03:50:44.586424+0000","last_became_peered":"2024-03-26T03:50:44.586424+0000","last_unstale":"2024-03-26T03:53:04.385715+0000","last_undegraded":"2024-03-26T03:53:04.385715+0000","last_fullsized":"2024-03-26T03:53:04.385715+0000","mapping_epoch":95,"log_start":"119'5000","ondisk_log_start":"119'5000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4189,"log_dups_size":0,"ondisk_log_size":4189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302579712,"num_objects":4617,"num_object_clones":0,"num_object_copies":13851,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4617,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9189,"num_write_kb":441792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":165,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.286751+0000","last_change":"2024-03-26T03:51:33.934747+0000","last_active":"2024-03-26T03:53:03.286751+0000","last_peered":"2024-03-26T03:53:03.286751+0000","last_clean":"2024-03-26T03:53:03.286751+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:53:03.286751+0000","last_undegraded":"2024-03-26T03:53:03.286751+0000","last_fullsized":"2024-03-26T03:53:03.286751+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.934617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.653127+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":172,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.297856+0000","last_change":"2024-03-26T03:51:30.466650+0000","last_active":"2024-03-26T03:53:03.297856+0000","last_peered":"2024-03-26T03:53:03.297856+0000","last_clean":"2024-03-26T03:53:03.297856+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:53:03.297856+0000","last_undegraded":"2024-03-26T03:53:03.297856+0000","last_fullsized":"2024-03-26T03:53:03.297856+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:51:30.466512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.466512+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:05.729035+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.280858+0000","last_change":"2024-03-26T03:51:29.300134+0000","last_active":"2024-03-26T03:53:03.280858+0000","last_peered":"2024-03-26T03:53:03.280858+0000","last_clean":"2024-03-26T03:53:03.280858+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:53:03.280858+0000","last_undegraded":"2024-03-26T03:53:03.280858+0000","last_fullsized":"2024-03-26T03:53:03.280858+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:51:29.299953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.299953+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498040+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":224,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275015+0000","last_change":"2024-03-26T03:52:31.483741+0000","last_active":"2024-03-26T03:53:03.275015+0000","last_peered":"2024-03-26T03:53:03.275015+0000","last_clean":"2024-03-26T03:53:03.275015+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:53:03.275015+0000","last_undegraded":"2024-03-26T03:53:03.275015+0000","last_fullsized":"2024-03-26T03:53:03.275015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:31.483593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.483593+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.632795+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"137'9276","reported_seq":9371,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.397542+0000","last_change":"2024-03-26T03:50:44.593372+0000","last_active":"2024-03-26T03:53:04.397542+0000","last_peered":"2024-03-26T03:53:04.397542+0000","last_clean":"2024-03-26T03:53:04.397542+0000","last_became_active":"2024-03-26T03:50:44.592595+0000","last_became_peered":"2024-03-26T03:50:44.592595+0000","last_unstale":"2024-03-26T03:53:04.397542+0000","last_undegraded":"2024-03-26T03:53:04.397542+0000","last_fullsized":"2024-03-26T03:53:04.397542+0000","mapping_epoch":95,"log_start":"120'5100","ondisk_log_start":"120'5100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4176,"log_dups_size":0,"ondisk_log_size":4176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311033856,"num_objects":4746,"num_object_clones":0,"num_object_copies":14238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9276,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":151,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279279+0000","last_change":"2024-03-26T03:51:21.386995+0000","last_active":"2024-03-26T03:53:03.279279+0000","last_peered":"2024-03-26T03:53:03.279279+0000","last_clean":"2024-03-26T03:53:03.279279+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:53:03.279279+0000","last_undegraded":"2024-03-26T03:53:03.279279+0000","last_fullsized":"2024-03-26T03:53:03.279279+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.386911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.386911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498071+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":171,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.277688+0000","last_change":"2024-03-26T03:52:45.556559+0000","last_active":"2024-03-26T03:53:03.277688+0000","last_peered":"2024-03-26T03:53:03.277688+0000","last_clean":"2024-03-26T03:53:03.277688+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:53:03.277688+0000","last_undegraded":"2024-03-26T03:53:03.277688+0000","last_fullsized":"2024-03-26T03:53:03.277688+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.193240+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":101,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.284375+0000","last_change":"2024-03-26T03:52:46.432028+0000","last_active":"2024-03-26T03:53:03.284375+0000","last_peered":"2024-03-26T03:53:03.284375+0000","last_clean":"2024-03-26T03:53:03.284375+0000","last_became_active":"2024-03-26T03:51:24.676652+0000","last_became_peered":"2024-03-26T03:51:24.676652+0000","last_unstale":"2024-03-26T03:53:03.284375+0000","last_undegraded":"2024-03-26T03:53:03.284375+0000","last_fullsized":"2024-03-26T03:53:03.284375+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:52:46.429387+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.429387+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.072480+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279733+0000","last_change":"2024-03-26T03:52:37.885150+0000","last_active":"2024-03-26T03:53:03.279733+0000","last_peered":"2024-03-26T03:53:03.279733+0000","last_clean":"2024-03-26T03:53:03.279733+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:53:03.279733+0000","last_undegraded":"2024-03-26T03:53:03.279733+0000","last_fullsized":"2024-03-26T03:53:03.279733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.707219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"137'9246","reported_seq":9339,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.449458+0000","last_change":"2024-03-26T03:50:45.583826+0000","last_active":"2024-03-26T03:53:04.449458+0000","last_peered":"2024-03-26T03:53:04.449458+0000","last_clean":"2024-03-26T03:53:04.449458+0000","last_became_active":"2024-03-26T03:50:45.583350+0000","last_became_peered":"2024-03-26T03:50:45.583350+0000","last_unstale":"2024-03-26T03:53:04.449458+0000","last_undegraded":"2024-03-26T03:53:04.449458+0000","last_fullsized":"2024-03-26T03:53:04.449458+0000","mapping_epoch":96,"log_start":"119'5000","ondisk_log_start":"119'5000","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4246,"log_dups_size":0,"ondisk_log_size":4246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306446336,"num_objects":4676,"num_object_clones":0,"num_object_copies":14028,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4676,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9246,"num_write_kb":445504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.269639+0000","last_change":"2024-03-26T03:51:35.580349+0000","last_active":"2024-03-26T03:53:03.269639+0000","last_peered":"2024-03-26T03:53:03.269639+0000","last_clean":"2024-03-26T03:53:03.269639+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:53:03.269639+0000","last_undegraded":"2024-03-26T03:53:03.269639+0000","last_fullsized":"2024-03-26T03:53:03.269639+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.580269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.580269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:06.446657+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279784+0000","last_change":"2024-03-26T03:52:20.706812+0000","last_active":"2024-03-26T03:53:03.279784+0000","last_peered":"2024-03-26T03:53:03.279784+0000","last_clean":"2024-03-26T03:53:03.279784+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:53:03.279784+0000","last_undegraded":"2024-03-26T03:53:03.279784+0000","last_fullsized":"2024-03-26T03:53:03.279784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:20.706714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:20.706714+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:29.973564+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":163,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.274907+0000","last_change":"2024-03-26T03:51:26.426050+0000","last_active":"2024-03-26T03:53:03.274907+0000","last_peered":"2024-03-26T03:53:03.274907+0000","last_clean":"2024-03-26T03:53:03.274907+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:53:03.274907+0000","last_undegraded":"2024-03-26T03:53:03.274907+0000","last_fullsized":"2024-03-26T03:53:03.274907+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:26.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:26.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":190,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.278475+0000","last_change":"2024-03-26T03:51:38.907371+0000","last_active":"2024-03-26T03:53:03.278475+0000","last_peered":"2024-03-26T03:53:03.278475+0000","last_clean":"2024-03-26T03:53:03.278475+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:53:03.278475+0000","last_undegraded":"2024-03-26T03:53:03.278475+0000","last_fullsized":"2024-03-26T03:53:03.278475+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:38.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.907089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686511+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"137'9164","reported_seq":9258,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.305119+0000","last_change":"2024-03-26T03:50:45.591630+0000","last_active":"2024-03-26T03:53:04.305119+0000","last_peered":"2024-03-26T03:53:04.305119+0000","last_clean":"2024-03-26T03:53:04.305119+0000","last_became_active":"2024-03-26T03:50:45.591306+0000","last_became_peered":"2024-03-26T03:50:45.591306+0000","last_unstale":"2024-03-26T03:53:04.305119+0000","last_undegraded":"2024-03-26T03:53:04.305119+0000","last_fullsized":"2024-03-26T03:53:04.305119+0000","mapping_epoch":96,"log_start":"118'4900","ondisk_log_start":"118'4900","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.686344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307953688,"num_objects":4700,"num_object_clones":0,"num_object_copies":14100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4700,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9164,"num_write_kb":443585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":156,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275521+0000","last_change":"2024-03-26T03:51:19.390232+0000","last_active":"2024-03-26T03:53:03.275521+0000","last_peered":"2024-03-26T03:53:03.275521+0000","last_clean":"2024-03-26T03:53:03.275521+0000","last_became_active":"2024-03-26T03:50:45.595009+0000","last_became_peered":"2024-03-26T03:50:45.595009+0000","last_unstale":"2024-03-26T03:53:03.275521+0000","last_undegraded":"2024-03-26T03:53:03.275521+0000","last_fullsized":"2024-03-26T03:53:03.275521+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.390111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.390111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411700+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":176,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.270242+0000","last_change":"2024-03-26T03:52:42.284044+0000","last_active":"2024-03-26T03:53:03.270242+0000","last_peered":"2024-03-26T03:53:03.270242+0000","last_clean":"2024-03-26T03:53:03.270242+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:53:03.270242+0000","last_undegraded":"2024-03-26T03:53:03.270242+0000","last_fullsized":"2024-03-26T03:53:03.270242+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.283958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.455074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":170,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.280530+0000","last_change":"2024-03-26T03:51:25.157523+0000","last_active":"2024-03-26T03:53:03.280530+0000","last_peered":"2024-03-26T03:53:03.280530+0000","last_clean":"2024-03-26T03:53:03.280530+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:53:03.280530+0000","last_undegraded":"2024-03-26T03:53:03.280530+0000","last_fullsized":"2024-03-26T03:53:03.280530+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.157414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.157414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498172+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":217,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279576+0000","last_change":"2024-03-26T03:51:31.354516+0000","last_active":"2024-03-26T03:53:03.279576+0000","last_peered":"2024-03-26T03:53:03.279576+0000","last_clean":"2024-03-26T03:53:03.279576+0000","last_became_active":"2024-03-26T03:50:45.596131+0000","last_became_peered":"2024-03-26T03:50:45.596131+0000","last_unstale":"2024-03-26T03:53:03.279576+0000","last_undegraded":"2024-03-26T03:53:03.279576+0000","last_fullsized":"2024-03-26T03:53:03.279576+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:31.354413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:31.354413+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497928+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"137'9237","reported_seq":9406,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.313536+0000","last_change":"2024-03-26T03:52:39.118362+0000","last_active":"2024-03-26T03:53:04.313536+0000","last_peered":"2024-03-26T03:53:04.313536+0000","last_clean":"2024-03-26T03:53:04.313536+0000","last_became_active":"2024-03-26T03:50:44.590889+0000","last_became_peered":"2024-03-26T03:50:44.590889+0000","last_unstale":"2024-03-26T03:53:04.313536+0000","last_undegraded":"2024-03-26T03:53:04.313536+0000","last_fullsized":"2024-03-26T03:53:04.313536+0000","mapping_epoch":95,"log_start":"113'3700","ondisk_log_start":"113'3700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7740","last_scrub_stamp":"2024-03-26T03:52:39.118187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.118187+0000","objects_scrubbed":6400,"log_size":5537,"log_dups_size":0,"ondisk_log_size":5537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.537245+0000","scrub_duration":2793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317652992,"num_objects":4847,"num_object_clones":0,"num_object_copies":14541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9237,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":166,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.291877+0000","last_change":"2024-03-26T03:52:44.431735+0000","last_active":"2024-03-26T03:53:03.291877+0000","last_peered":"2024-03-26T03:53:03.291877+0000","last_clean":"2024-03-26T03:53:03.291877+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:53:03.291877+0000","last_undegraded":"2024-03-26T03:53:03.291877+0000","last_fullsized":"2024-03-26T03:53:03.291877+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.431608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.431608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.078115+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":161,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.285895+0000","last_change":"2024-03-26T03:52:30.927185+0000","last_active":"2024-03-26T03:53:03.285895+0000","last_peered":"2024-03-26T03:53:03.285895+0000","last_clean":"2024-03-26T03:53:03.285895+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:53:03.285895+0000","last_undegraded":"2024-03-26T03:53:03.285895+0000","last_fullsized":"2024-03-26T03:53:03.285895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:30.926620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.926620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:33.001795+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":101,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279307+0000","last_change":"2024-03-26T03:51:23.188460+0000","last_active":"2024-03-26T03:53:03.279307+0000","last_peered":"2024-03-26T03:53:03.279307+0000","last_clean":"2024-03-26T03:53:03.279307+0000","last_became_active":"2024-03-26T03:50:45.593583+0000","last_became_peered":"2024-03-26T03:50:45.593583+0000","last_unstale":"2024-03-26T03:53:03.279307+0000","last_undegraded":"2024-03-26T03:53:03.279307+0000","last_fullsized":"2024-03-26T03:53:03.279307+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.188316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.188316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.498100+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":194,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279523+0000","last_change":"2024-03-26T03:51:32.392253+0000","last_active":"2024-03-26T03:53:03.279523+0000","last_peered":"2024-03-26T03:53:03.279523+0000","last_clean":"2024-03-26T03:53:03.279523+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:53:03.279523+0000","last_undegraded":"2024-03-26T03:53:03.279523+0000","last_fullsized":"2024-03-26T03:53:03.279523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:32.392163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.392163+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:07.497973+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"137'9299","reported_seq":9462,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:04.423863+0000","last_change":"2024-03-26T03:52:46.641388+0000","last_active":"2024-03-26T03:53:04.423863+0000","last_peered":"2024-03-26T03:53:04.423863+0000","last_clean":"2024-03-26T03:53:04.423863+0000","last_became_active":"2024-03-26T03:50:44.587505+0000","last_became_peered":"2024-03-26T03:50:44.587505+0000","last_unstale":"2024-03-26T03:53:04.423863+0000","last_undegraded":"2024-03-26T03:53:04.423863+0000","last_fullsized":"2024-03-26T03:53:04.423863+0000","mapping_epoch":95,"log_start":"118'4500","ondisk_log_start":"118'4500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'8232","last_scrub_stamp":"2024-03-26T03:52:46.640967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.640967+0000","objects_scrubbed":5921,"log_size":4799,"log_dups_size":0,"ondisk_log_size":4799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.138786+0000","scrub_duration":3071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312279040,"num_objects":4765,"num_object_clones":0,"num_object_copies":14295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9299,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":158,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.279837+0000","last_change":"2024-03-26T03:51:41.329652+0000","last_active":"2024-03-26T03:53:03.279837+0000","last_peered":"2024-03-26T03:53:03.279837+0000","last_clean":"2024-03-26T03:53:03.279837+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:53:03.279837+0000","last_undegraded":"2024-03-26T03:53:03.279837+0000","last_fullsized":"2024-03-26T03:53:03.279837+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.329573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.329573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:03.999992+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":223,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.284494+0000","last_change":"2024-03-26T03:51:30.103885+0000","last_active":"2024-03-26T03:53:03.284494+0000","last_peered":"2024-03-26T03:53:03.284494+0000","last_clean":"2024-03-26T03:53:03.284494+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:53:03.284494+0000","last_undegraded":"2024-03-26T03:53:03.284494+0000","last_fullsized":"2024-03-26T03:53:03.284494+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:30.098793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.098793+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":162,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.275258+0000","last_change":"2024-03-26T03:51:32.274566+0000","last_active":"2024-03-26T03:53:03.275258+0000","last_peered":"2024-03-26T03:53:03.275258+0000","last_clean":"2024-03-26T03:53:03.275258+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:53:03.275258+0000","last_undegraded":"2024-03-26T03:53:03.275258+0000","last_fullsized":"2024-03-26T03:53:03.275258+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.274349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.274349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:04.411751+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":199,"reported_epoch":137,"state":"active+clean","last_fresh":"2024-03-26T03:53:03.281468+0000","last_change":"2024-03-26T03:52:31.734237+0000","last_active":"2024-03-26T03:53:03.281468+0000","last_peered":"2024-03-26T03:53:03.281468+0000","last_clean":"2024-03-26T03:53:03.281468+0000","last_became_active":"2024-03-26T03:50:45.598403+0000","last_became_peered":"2024-03-26T03:50:45.598403+0000","last_unstale":"2024-03-26T03:53:03.281468+0000","last_undegraded":"2024-03-26T03:53:03.281468+0000","last_fullsized":"2024-03-26T03:53:03.281468+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.994530+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4984012824,"num_objects":76051,"num_object_clones":0,"num_object_copies":228153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76051,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148653,"num_write_kb":7190465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14961684480,"data_stored":14961672264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75553,"ondisk_log_size":75553,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":253,"num_read_kb":248,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070520,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1601224,"kb_used_data":1529668,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92143928,"statfs":{"total":95995035648,"available":94355382272,"internally_reserved":0,"allocated":1566380032,"data_stored":1566015240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73136586},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574935,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2558116,"kb_used_data":2458784,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91187036,"statfs":{"total":95995035648,"available":93375524864,"internally_reserved":0,"allocated":2517794816,"data_stored":2517400529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738460,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2588892,"kb_used_data":2477080,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91156260,"statfs":{"total":95995035648,"available":93344010240,"internally_reserved":0,"allocated":2536529920,"data_stored":2536144794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411410,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2562160,"kb_used_data":2462828,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91182992,"statfs":{"total":95995035648,"available":93371383808,"internally_reserved":0,"allocated":2521935872,"data_stored":2521593158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3209108,"kb_used_data":3069520,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":90536044,"statfs":{"total":95995035648,"available":92708909056,"internally_reserved":0,"allocated":3143188480,"data_stored":3142810706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953543,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1619528,"kb_used_data":1547972,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92125624,"statfs":{"total":95995035648,"available":94336638976,"internally_reserved":0,"allocated":1585123328,"data_stored":1584756308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659201,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1302456,"kb_used_data":1239220,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":92442696,"statfs":{"total":95995035648,"available":94661320704,"internally_reserved":0,"allocated":1268961280,"data_stored":1268610682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2505969664,"data_stored":2505965592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2493976576,"data_stored":2493972504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2486173696,"data_stored":2486173696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3100446720,"data_stored":3100442648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1568473088,"data_stored":1568473088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1253703680,"data_stored":1253703680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1552941056,"data_stored":1552941056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:53:10.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:53:10.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:10.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:10.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:10.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:10.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:53:10.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:10.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:10 smithi006 ceph-mon[37501]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:10.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:10 smithi006 ceph-mon[37501]: pgmap v176: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 774 op/s 2024-03-26T03:53:10.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:10 smithi006 ceph-mon[29446]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:10.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:10 smithi006 ceph-mon[29446]: pgmap v176: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 774 op/s 2024-03-26T03:53:10.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:10.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:10.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.820 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:10.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:10 smithi086 ceph-mon[32965]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:10 smithi086 ceph-mon[32965]: pgmap v176: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 774 op/s 2024-03-26T03:53:10.892 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:53:10.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:10.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:10.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:10.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:11.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:11.093 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:11.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:11.173 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:11.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:11.257 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:11.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:11.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:11.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:11.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:11.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:11.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:11.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:11.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:11.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:11.771 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:11.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:11.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:11.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:11.971 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:12.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:12.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:12.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:12.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.357 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:53:12.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:12.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:12.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.628 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:12.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:12 smithi086 ceph-mon[32965]: pgmap v177: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-26T03:53:12.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/39962454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.687 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:12 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:12.771 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:12.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[37501]: pgmap v177: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-26T03:53:12.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/39962454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[29446]: pgmap v177: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-03-26T03:53:12.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/39962454' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:12 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:53:12.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:12.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:12.983 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:53:12.984 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:53:12.984 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T03:53:12.984 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T03:53:12.984 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:53:13.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:13.044 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:13.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.184 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:13.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:13.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:13.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:13.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:13.447 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:13.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:13.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:13 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:53:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:13 smithi086 ceph-mon[32965]: osdmap e139: 8 total, 8 up, 7 in 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[37501]: osdmap e139: 8 total, 8 up, 7 in 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:53:13.647 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:13 smithi006 ceph-mon[29446]: osdmap e139: 8 total, 8 up, 7 in 2024-03-26T03:53:13.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:53:13.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:13.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:13.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:13.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:13.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:13.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:13.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:13.963 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:53:13.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:14.015 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:14.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:14.161 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:14.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:14.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:14.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:14.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:14.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:14.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:14.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 3.12 scrub starts 2024-03-26T03:53:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 3.12 scrub ok 2024-03-26T03:53:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: pgmap v179: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 607 op/s 2024-03-26T03:53:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T03:53:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T03:53:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T03:53:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T03:53:14.642 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 3.12 scrub starts 2024-03-26T03:53:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 3.12 scrub ok 2024-03-26T03:53:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: pgmap v179: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 607 op/s 2024-03-26T03:53:14.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T03:53:14.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T03:53:14.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T03:53:14.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:14 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T03:53:14.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:14.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.695 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:14.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:14.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:14.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 3.12 scrub starts 2024-03-26T03:53:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 3.12 scrub ok 2024-03-26T03:53:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: pgmap v179: 148 pgs: 148 active+clean; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 607 op/s 2024-03-26T03:53:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T03:53:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T03:53:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T03:53:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:14 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T03:53:14.845 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:14.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:14.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:14.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:14.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:14.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:15.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:15.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.111 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:15.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:15.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:15.250 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:15.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:15.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:15.390 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:15.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:15.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:15.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T03:53:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T03:53:15.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:15.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T03:53:15.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T03:53:15.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:15.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:15.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:15.666 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:15 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T03:53:15.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:15 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T03:53:15.667 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:15.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:15.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:15.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:15.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:15.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:6261 7.5 7:ad7093d4:::benchmark_data_smithi006_77260_object6260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:53:15.973 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 137.48774202999999, 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.083550506999999996, 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 6261 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482041+0000", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.051e-06 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482096+0000", 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4685000000000002e-05 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482108+0000", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1993e-05 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482118+0000", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6209999999999999e-06 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482166+0000", 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7883000000000002e-05 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482287+0000", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012121300000000001 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482399+0000", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112178 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482822+0000", 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00042292099999999998 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.483327+0000", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00050472400000000001 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565474+0000", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082147297999999994 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565538+0000", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4065000000000004e-05 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:15.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565591+0000", 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2874999999999998e-05 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:45157 7.5 7:a10d2ec8:::benchmark_data_smithi006_77260_object45156:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:21.574294+0000", 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 114.39548828, 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.065194721999999997, 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:15.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45157 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574294+0000", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574294+0000", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574295+0000", 2024-03-26T03:53:15.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.209e-06 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574352+0000", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6728999999999998e-05 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574363+0000", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0699999999999999e-05 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574369+0000", 2024-03-26T03:53:15.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.874e-06 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574423+0000", 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4008999999999997e-05 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574580+0000", 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000157054 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:15.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.574697+0000", 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117248 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.575900+0000", 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001202683 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.576931+0000", 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010316329999999999 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.639407+0000", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062476114999999999 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.639440+0000", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2743000000000001e-05 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:21.639489+0000", 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8724999999999999e-05 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46138 7.5 7:a2b32a51:::benchmark_data_smithi006_77260_object46137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.533241174, 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.132391285, 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:15.985 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46138 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436543+0000", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5600000000000001e-06 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436591+0000", 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8069999999999999e-05 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436603+0000", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1931e-05 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436610+0000", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1570000000000003e-06 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436675+0000", 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4828999999999999e-05 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436816+0000", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000140953 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436983+0000", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016772 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439654+0000", 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026707150000000002 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.443582+0000", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039278680000000002 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568851+0000", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.12526948500000001 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568887+0000", 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5108999999999999e-05 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568932+0000", 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5887999999999998e-05 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46141 7.5 7:a45f0658:::benchmark_data_smithi006_77260_object46140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 113.531488732, 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13568999400000001, 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:15.991 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46141 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438295+0000", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0130000000000001e-06 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438342+0000", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7655999999999999e-05 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438361+0000", 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8967999999999999e-05 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438370+0000", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5660000000000007e-06 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438410+0000", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0234999999999997e-05 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438505+0000", 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5279999999999996e-05 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438604+0000", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8387999999999994e-05 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439762+0000", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001158563 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.443943+0000", 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004181176 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573919+0000", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.129975919 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573956+0000", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6600000000000002e-05 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573984+0000", 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7630000000000001e-05 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:56262 7.5 7:a1f98551:::benchmark_data_smithi006_77260_object56261:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:30.652076+0000", 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.317706147, 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063556794, 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56262 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652076+0000", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652076+0000", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652077+0000", 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2440000000000001e-06 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652131+0000", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3761999999999998e-05 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652143+0000", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1634000000000001e-05 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652150+0000", 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4610000000000002e-06 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:15.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652202+0000", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2157999999999998e-05 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652339+0000", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013655300000000001 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652488+0000", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000149104 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.656135+0000", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036473310000000002 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.656161+0000", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5732999999999999e-05 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715565+0000", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059403926000000003 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715602+0000", 2024-03-26T03:53:16.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7313e-05 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715633+0000", 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0574999999999998e-05 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:56265 7.5 7:a2abaf46:::benchmark_data_smithi006_77260_object56264:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:30.652292+0000", 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.317490668, 2024-03-26T03:53:16.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063440753000000003, 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 56265 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652292+0000", 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652292+0000", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652292+0000", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7800000000000001e-07 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652355+0000", 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2508000000000005e-05 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652363+0000", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7589999999999997e-06 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.652370+0000", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0639999999999996e-06 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.654239+0000", 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001869494 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.654374+0000", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013459600000000001 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.654523+0000", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000148997 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.656254+0000", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001731043 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.656270+0000", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6368e-05 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715672+0000", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059402061999999999 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715706+0000", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4180999999999997e-05 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:30.715732+0000", 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6003000000000002e-05 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:73166 7.5 7:a8d64edc:::benchmark_data_smithi006_77260_object73165:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:46.316171+0000", 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 89.653611409000007, 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.064144143000000001, 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.008 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 73166 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316171+0000", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316171+0000", 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316172+0000", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.249e-06 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316231+0000", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8405000000000001e-05 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316245+0000", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4041e-05 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316254+0000", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9390000000000003e-06 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316308+0000", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4812000000000003e-05 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316450+0000", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141559 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.316594+0000", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014423600000000001 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.318632+0000", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020381380000000001 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.320632+0000", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019997830000000002 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.380250+0000", 2024-03-26T03:53:16.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.059617442 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.380286+0000", 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6882999999999999e-05 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:46.380315+0000", 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8656000000000001e-05 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106682 7.5 7:ab108606:::benchmark_data_smithi006_77260_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.677212603999997, 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099219867000000003, 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106682 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292571+0000", 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2979999999999999e-06 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292604+0000", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2778999999999999e-05 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292619+0000", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4980999999999999e-05 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292627+0000", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6209999999999993e-06 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292687+0000", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0056999999999998e-05 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292828+0000", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000140908 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.293028+0000", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019991400000000001 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.296547+0000", 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035184069999999999 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.300475+0000", 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039280000000000001 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391667+0000", 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091192208999999996 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391736+0000", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9216999999999998e-05 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391790+0000", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3476000000000001e-05 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106684 7.5 7:a3280a30:::benchmark_data_smithi006_77260_object106683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:53:16.019 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.676949981999996, 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099095332999999994, 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106684 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292833+0000", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3900000000000005e-07 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292890+0000", 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7488000000000003e-05 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292898+0000", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9000000000000006e-06 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292903+0000", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.583e-06 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.294905+0000", 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020025569999999999 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.295041+0000", 2024-03-26T03:53:16.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:16.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013517100000000001 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.295184+0000", 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014320399999999999 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.296697+0000", 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001513183 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.300509+0000", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003812092 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391836+0000", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091327152999999994 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391888+0000", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1773e-05 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391928+0000", 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9690000000000001e-05 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106698 7.5 7:aa9cc05b:::benchmark_data_smithi006_77260_object106697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 55.658259135999998, 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.094714796000000004, 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.027 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106698 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311528+0000", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4880000000000001e-06 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311576+0000", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8798999999999998e-05 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311588+0000", 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1221e-05 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311594+0000", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5980000000000002e-06 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311641+0000", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6535000000000001e-05 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311732+0000", 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1625000000000005e-05 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311833+0000", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010040899999999999 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.313748+0000", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019150180000000001 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.394986+0000", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.081237986999999998 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406121+0000", 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011135107999999999 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406188+0000", 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7009999999999997e-05 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406238+0000", 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9997999999999998e-05 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.032 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:111024 7.5 7:af5debec:::benchmark_data_smithi006_77260_object111023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 51.141240429, 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082518326000000003, 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111024 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.033 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828543+0000", 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.268e-06 2024-03-26T03:53:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828574+0000", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0793000000000001e-05 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828585+0000", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0773e-05 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828591+0000", 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5699999999999998e-06 2024-03-26T03:53:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828636+0000", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4672000000000002e-05 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828747+0000", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011108900000000001 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828863+0000", 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115643 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.830833+0000", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001970795 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.831829+0000", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00099511799999999991 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.910945+0000", 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079116540999999999 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.911007+0000", 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1715000000000001e-05 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.911060+0000", 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3349e-05 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:131664 7.5 7:a6c5c53b:::benchmark_data_smithi006_77260_object19310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T03:53:16.038 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 28.984380529999999, 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10817117900000001, 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131664 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985403+0000", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1119999999999999e-06 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985407+0000", 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4739999999999999e-06 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985417+0000", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9559999999999992e-06 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985425+0000", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6090000000000003e-06 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985467+0000", 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2314000000000002e-05 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985554+0000", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7089000000000006e-05 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985630+0000", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5465999999999999e-05 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.987810+0000", 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002180305 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.998012+0000", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010202093000000001 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093484+0000", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.095471577000000002 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093523+0000", 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8995999999999997e-05 2024-03-26T03:53:16.043 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093573+0000", 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0188000000000001e-05 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:131670 7.5 7:a85b9c68:::benchmark_data_smithi006_77260_object19316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 28.978528249, 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.102408474, 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.044 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131670 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:53:16.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991256+0000", 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7010000000000001e-06 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991262+0000", 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5810000000000002e-06 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991275+0000", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3040999999999999e-05 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991283+0000", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0590000000000003e-06 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991338+0000", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4768000000000001e-05 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991476+0000", 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013815400000000001 2024-03-26T03:53:16.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991604+0000", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012799 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.993610+0000", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002005682 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.998045+0000", 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00443452 2024-03-26T03:53:16.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093608+0000", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.095563674000000001 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093638+0000", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9791e-05 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093663+0000", 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4513000000000001e-05 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.049 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133138 7.5 7:a462d8d7:::benchmark_data_smithi006_77260_object20784:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:48.766386+0000", 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 27.203396212000001, 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.066198240000000005, 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133138 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.050 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766386+0000", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766386+0000", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766387+0000", 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3349999999999999e-06 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766393+0000", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.096e-06 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766402+0000", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4229999999999994e-06 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766408+0000", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2870000000000001e-06 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766452+0000", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3343999999999997e-05 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766531+0000", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9672999999999994e-05 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.766602+0000", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0797999999999995e-05 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.768568+0000", 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001966321 2024-03-26T03:53:16.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.769458+0000", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00088920500000000001 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.832481+0000", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063023579999999996 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.832539+0000", 2024-03-26T03:53:16.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7704000000000002e-05 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:48.832584+0000", 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5473999999999999e-05 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133496 7.5 7:a5640036:::benchmark_data_smithi006_77260_object21142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 26.761556658, 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076748271000000007, 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.055 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133496 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:53:16.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208227+0000", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.051e-06 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208232+0000", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0769999999999998e-06 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208242+0000", 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8199999999999992e-06 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.057 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208248+0000", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1140000000000002e-06 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208286+0000", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8500000000000001e-05 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208381+0000", 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4513999999999997e-05 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.058 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208455+0000", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4407999999999996e-05 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.210647+0000", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021914780000000002 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.215727+0000", 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050805080000000001 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.059 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284873+0000", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.069146142999999993 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284929+0000", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5832999999999999e-05 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284974+0000", 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4824999999999999e-05 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.060 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133791 7.5 7:a1dee8fa:::benchmark_data_smithi006_77260_object21437:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.539021+0000", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 26.43076155, 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.057734093, 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133791 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.061 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539021+0000", 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539021+0000", 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539022+0000", 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.043e-06 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.062 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539026+0000", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6609999999999999e-06 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539044+0000", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7581e-05 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539052+0000", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.807e-06 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539103+0000", 2024-03-26T03:53:16.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0905000000000001e-05 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539241+0000", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000137828 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.539370+0000", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000129147 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541467+0000", 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020967749999999999 2024-03-26T03:53:16.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541957+0000", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049015099999999998 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.596683+0000", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054726202000000002 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.596720+0000", 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6949999999999997e-05 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.596755+0000", 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5043000000000003e-05 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133937 7.5 7:ad842931:::benchmark_data_smithi006_77260_object21583:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.749192+0000", 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 26.220590665, 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.062097271000000002, 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.066 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133937 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749192+0000", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749192+0000", 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749193+0000", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.497e-06 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749199+0000", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4049999999999999e-06 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749211+0000", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2858e-05 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749220+0000", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5310000000000001e-06 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749278+0000", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8239999999999998e-05 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749407+0000", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012857600000000001 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749526+0000", 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119492 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.754828+0000", 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053022060000000003 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.754882+0000", 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3179e-05 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811219+0000", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056336903000000001 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811254+0000", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5831000000000003e-05 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811289+0000", 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4552999999999999e-05 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133939 7.5 7:a4153903:::benchmark_data_smithi006_77260_object21585:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.749881+0000", 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 26.219900977999998, 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.061528925999999998, 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133939 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749881+0000", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749881+0000", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749882+0000", 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2900000000000002e-07 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749886+0000", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2549999999999999e-06 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749899+0000", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2675e-05 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.749906+0000", 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1389999999999998e-06 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752028+0000", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021215069999999999 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752169+0000", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014102900000000001 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.752312+0000", 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014306099999999999 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.754946+0000", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026344519999999998 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.755473+0000", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00052633200000000002 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811356+0000", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.055883587999999998 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811385+0000", 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8999000000000001e-05 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.811410+0000", 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5060000000000001e-05 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:134171 7.5 7:ac24066f:::benchmark_data_smithi006_77260_object21817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 25.932080099, 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084603088000000007, 2024-03-26T03:53:16.077 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134171 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037704+0000", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4720000000000001e-06 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037709+0000", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7150000000000003e-06 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037720+0000", 2024-03-26T03:53:16.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0473e-05 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037726+0000", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4180000000000002e-06 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037792+0000", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5982000000000006e-05 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037958+0000", 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016622100000000001 2024-03-26T03:53:16.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.038050+0000", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1804000000000004e-05 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.040390+0000", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023399929999999998 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.043713+0000", 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003322532 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122230+0000", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.078516846000000001 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122268+0000", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8077000000000002e-05 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122305+0000", 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7555000000000002e-05 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.082 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:156478 7.5 7:af328b33:::benchmark_data_smithi006_77260_object44124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:12.415454+0000", 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 3.5543283949999998, 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.068168414999999996, 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 156478 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:53:16.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415454+0000", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415454+0000", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415456+0000", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8300000000000001e-06 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:53:16.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415465+0000", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9719999999999998e-06 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415484+0000", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9585999999999998e-05 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415494+0000", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4390000000000006e-06 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415554+0000", 2024-03-26T03:53:16.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9973999999999999e-05 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.415558+0000", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6199999999999998e-06 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.417455+0000", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001896958 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.417608+0000", 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015253700000000001 2024-03-26T03:53:16.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.417738+0000", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013045999999999999 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.420348+0000", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026097149999999999 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.420490+0000", 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141779 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.483552+0000", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.063061934 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.483588+0000", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6109999999999998e-05 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.483622+0000", 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4501000000000001e-05 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:53:16.088 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:53:16.089 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:53:16.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:16.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:16.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:16.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:16.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:53:16.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:16.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:16.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:16.541 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:16 smithi006 ceph-mon[37501]: pgmap v180: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T03:53:16.541 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:16 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1654239615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:16.543 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:16 smithi006 ceph-mon[29446]: pgmap v180: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T03:53:16.543 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:16 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1654239615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:16.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:16.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:16.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:16.764 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:53:16.765 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:53:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:53:16.766 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:53:16.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:16.831 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:16 smithi086 ceph-mon[32965]: pgmap v180: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T03:53:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:16 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1654239615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:16.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:16.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:16.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:16.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:17.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:17.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:17.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:17.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:17.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:17.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:17.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:17.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:17.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:17.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:17.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:17.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:17.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:17.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:17.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:53:17.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:17.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:17.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:18.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:18.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.054 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:18.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:18.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:18.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:18.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:18.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:18.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:18.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:18.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:18.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.614 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:18.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:18.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T03:53:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T03:53:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: pgmap v181: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-26T03:53:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 6.13 scrub starts 2024-03-26T03:53:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 6.13 scrub ok 2024-03-26T03:53:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T03:53:18.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:18 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T03:53:18.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T03:53:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T03:53:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: pgmap v181: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-26T03:53:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 6.13 scrub starts 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 6.13 scrub ok 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: pgmap v181: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 985 op/s 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 6.13 scrub starts 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 6.13 scrub ok 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T03:53:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:18 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T03:53:18.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:18.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:19.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:19.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.032 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:19.107 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:53:19.112 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:53:19.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:19.176 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:19.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:19.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:19.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:19.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:19.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:19.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:19.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:19.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:19.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:19.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:19.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T03:53:19.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T03:53:19.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T03:53:19.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T03:53:19.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:19 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T03:53:19.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T03:53:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T03:53:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T03:53:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T03:53:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T03:53:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:19 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T03:53:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:19.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:20.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:20.136 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:20.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:20.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:20.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:20.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:20.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:20.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:20.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:20.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: 7.5 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: 7.5 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:53:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:20 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:53:20.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:20.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:20.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:20.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:20.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: 7.5 scrub starts 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: pgmap v182: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.0 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 978 op/s 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T03:53:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:53:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:20 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:53:20.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:20.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:20.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:20.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:21.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:21.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:21.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:21.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:21.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:21.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:21.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:21.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:21.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:21.454 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:21.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:21.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:21.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:21.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:21.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:21 smithi086 ceph-mon[32965]: 7.5 scrub ok 2024-03-26T03:53:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:21 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T03:53:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:21 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T03:53:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:21 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T03:53:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:21 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T03:53:21.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:21.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:21.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:21.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:21.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[37501]: 7.5 scrub ok 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[29446]: 7.5 scrub ok 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T03:53:21.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:21 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T03:53:21.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:22.013 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:22.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:22.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:22.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.160 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:22.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:22.228 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:22.228 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:22.239 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-03-26T03:53:17.249764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4338948544,"num_objects":66429,"num_object_clones":0,"num_object_copies":199287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66429,"num_whiteouts":0,"num_read":1483,"num_read_kb":1297,"num_write":159312,"num_write_kb":7192926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2362855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76333,"ondisk_log_size":76333,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":13522856,"kb_used_data":12854808,"kb_used_omap":10,"kb_used_meta":667957,"kb_avail":642693208,"statfs":{"total":671965249536,"available":658117844992,"internally_reserved":0,"allocated":13163323392,"data_stored":13160665341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":683988074},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,4,7,4,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-645660672,"num_objects":-9852,"num_object_clones":0,"num_object_copies":-29556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9852,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9852,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004993"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.402965+0000","last_change":"2024-03-26T03:50:34.035051+0000","last_active":"2024-03-26T03:53:12.402965+0000","last_peered":"2024-03-26T03:53:12.402965+0000","last_clean":"2024-03-26T03:53:12.402965+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:53:12.402965+0000","last_undegraded":"2024-03-26T03:53:12.402965+0000","last_fullsized":"2024-03-26T03:53:12.402965+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:34.034948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:34.034948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.379933+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480303+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:53:12.480303+0000","last_peered":"2024-03-26T03:53:12.480303+0000","last_clean":"2024-03-26T03:53:12.480303+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:53:12.480303+0000","last_undegraded":"2024-03-26T03:53:12.480303+0000","last_fullsized":"2024-03-26T03:53:12.480303+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.054186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480396+0000","last_change":"2024-03-26T03:51:39.331101+0000","last_active":"2024-03-26T03:53:12.480396+0000","last_peered":"2024-03-26T03:53:12.480396+0000","last_clean":"2024-03-26T03:53:12.480396+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:53:12.480396+0000","last_undegraded":"2024-03-26T03:53:12.480396+0000","last_fullsized":"2024-03-26T03:53:12.480396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.330345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":216,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.402878+0000","last_change":"2024-03-26T03:52:33.804342+0000","last_active":"2024-03-26T03:53:12.402878+0000","last_peered":"2024-03-26T03:53:12.402878+0000","last_clean":"2024-03-26T03:53:12.402878+0000","last_became_active":"2024-03-26T03:50:45.598168+0000","last_became_peered":"2024-03-26T03:50:45.598168+0000","last_unstale":"2024-03-26T03:53:12.402878+0000","last_undegraded":"2024-03-26T03:53:12.402878+0000","last_fullsized":"2024-03-26T03:53:12.402878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.024936+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410773+0000","last_change":"2024-03-26T03:52:43.442740+0000","last_active":"2024-03-26T03:53:12.410773+0000","last_peered":"2024-03-26T03:53:12.410773+0000","last_clean":"2024-03-26T03:53:12.410773+0000","last_became_active":"2024-03-26T03:50:45.597941+0000","last_became_peered":"2024-03-26T03:50:45.597941+0000","last_unstale":"2024-03-26T03:53:12.410773+0000","last_undegraded":"2024-03-26T03:53:12.410773+0000","last_fullsized":"2024-03-26T03:53:12.410773+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.003941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:13.396612+0000","last_change":"2024-03-26T03:53:13.396637+0000","last_active":"2024-03-26T03:53:13.396612+0000","last_peered":"2024-03-26T03:53:13.396612+0000","last_clean":"2024-03-26T03:53:13.396612+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:53:13.396612+0000","last_undegraded":"2024-03-26T03:53:13.396612+0000","last_fullsized":"2024-03-26T03:53:13.396612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.396551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.396551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:31.835565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414046+0000","last_change":"2024-03-26T03:51:21.399264+0000","last_active":"2024-03-26T03:53:12.414046+0000","last_peered":"2024-03-26T03:53:12.414046+0000","last_clean":"2024-03-26T03:53:12.414046+0000","last_became_active":"2024-03-26T03:50:45.581989+0000","last_became_peered":"2024-03-26T03:50:45.581989+0000","last_unstale":"2024-03-26T03:53:12.414046+0000","last_undegraded":"2024-03-26T03:53:12.414046+0000","last_fullsized":"2024-03-26T03:53:12.414046+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.398535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.398535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392306+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":235,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414666+0000","last_change":"2024-03-26T03:50:40.221494+0000","last_active":"2024-03-26T03:53:12.414666+0000","last_peered":"2024-03-26T03:53:12.414666+0000","last_clean":"2024-03-26T03:53:12.414666+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:53:12.414666+0000","last_undegraded":"2024-03-26T03:53:12.414666+0000","last_fullsized":"2024-03-26T03:53:12.414666+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:50:40.221401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:40.221401+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390642+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412632+0000","last_change":"2024-03-26T03:51:17.173504+0000","last_active":"2024-03-26T03:53:12.412632+0000","last_peered":"2024-03-26T03:53:12.412632+0000","last_clean":"2024-03-26T03:53:12.412632+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:53:12.412632+0000","last_undegraded":"2024-03-26T03:53:12.412632+0000","last_fullsized":"2024-03-26T03:53:12.412632+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:17.173394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.173394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488617+0000","last_change":"2024-03-26T03:51:23.617976+0000","last_active":"2024-03-26T03:53:12.488617+0000","last_peered":"2024-03-26T03:53:12.488617+0000","last_clean":"2024-03-26T03:53:12.488617+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:53:12.488617+0000","last_undegraded":"2024-03-26T03:53:12.488617+0000","last_fullsized":"2024-03-26T03:53:12.488617+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.617899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.617899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488967+0000","last_change":"2024-03-26T03:51:18.480480+0000","last_active":"2024-03-26T03:53:12.488967+0000","last_peered":"2024-03-26T03:53:12.488967+0000","last_clean":"2024-03-26T03:53:12.488967+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:53:12.488967+0000","last_undegraded":"2024-03-26T03:53:12.488967+0000","last_fullsized":"2024-03-26T03:53:12.488967+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.480360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.480360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":211,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403185+0000","last_change":"2024-03-26T03:51:27.245595+0000","last_active":"2024-03-26T03:53:12.403185+0000","last_peered":"2024-03-26T03:53:12.403185+0000","last_clean":"2024-03-26T03:53:12.403185+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:53:12.403185+0000","last_undegraded":"2024-03-26T03:53:12.403185+0000","last_fullsized":"2024-03-26T03:53:12.403185+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:27.245475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.245475+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380029+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488478+0000","last_change":"2024-03-26T03:52:26.427599+0000","last_active":"2024-03-26T03:53:12.488478+0000","last_peered":"2024-03-26T03:53:12.488478+0000","last_clean":"2024-03-26T03:53:12.488478+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:53:12.488478+0000","last_undegraded":"2024-03-26T03:53:12.488478+0000","last_fullsized":"2024-03-26T03:53:12.488478+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.427501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.427501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.945072+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416805+0000","last_change":"2024-03-26T03:52:52.666181+0000","last_active":"2024-03-26T03:53:12.416805+0000","last_peered":"2024-03-26T03:53:12.416805+0000","last_clean":"2024-03-26T03:53:12.416805+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:53:12.416805+0000","last_undegraded":"2024-03-26T03:53:12.416805+0000","last_fullsized":"2024-03-26T03:53:12.416805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.666107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.666107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.715049+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488743+0000","last_change":"2024-03-26T03:52:28.376930+0000","last_active":"2024-03-26T03:53:12.488743+0000","last_peered":"2024-03-26T03:53:12.488743+0000","last_clean":"2024-03-26T03:53:12.488743+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:53:12.488743+0000","last_undegraded":"2024-03-26T03:53:12.488743+0000","last_fullsized":"2024-03-26T03:53:12.488743+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:28.376798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:28.376798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:53.233172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":195,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403357+0000","last_change":"2024-03-26T03:52:38.915695+0000","last_active":"2024-03-26T03:53:12.403357+0000","last_peered":"2024-03-26T03:53:12.403357+0000","last_clean":"2024-03-26T03:53:12.403357+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:53:12.403357+0000","last_undegraded":"2024-03-26T03:53:12.403357+0000","last_fullsized":"2024-03-26T03:53:12.403357+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.912180+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.404005+0000","last_change":"2024-03-26T03:52:51.658215+0000","last_active":"2024-03-26T03:53:12.404005+0000","last_peered":"2024-03-26T03:53:12.404005+0000","last_clean":"2024-03-26T03:53:12.404005+0000","last_became_active":"2024-03-26T03:50:45.588396+0000","last_became_peered":"2024-03-26T03:50:45.588396+0000","last_unstale":"2024-03-26T03:53:12.404005+0000","last_undegraded":"2024-03-26T03:53:12.404005+0000","last_fullsized":"2024-03-26T03:53:12.404005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:17.128768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":157,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413977+0000","last_change":"2024-03-26T03:51:15.157751+0000","last_active":"2024-03-26T03:53:12.413977+0000","last_peered":"2024-03-26T03:53:12.413977+0000","last_clean":"2024-03-26T03:53:12.413977+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:53:12.413977+0000","last_undegraded":"2024-03-26T03:53:12.413977+0000","last_fullsized":"2024-03-26T03:53:12.413977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:15.157477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.157477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:13.631426+0000","last_change":"2024-03-26T03:53:13.631462+0000","last_active":"2024-03-26T03:53:13.631426+0000","last_peered":"2024-03-26T03:53:13.631426+0000","last_clean":"2024-03-26T03:53:13.631426+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:53:13.631426+0000","last_undegraded":"2024-03-26T03:53:13.631426+0000","last_fullsized":"2024-03-26T03:53:13.631426+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.631351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.631351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:41.275139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.409644+0000","last_change":"2024-03-26T03:52:45.404305+0000","last_active":"2024-03-26T03:53:12.409644+0000","last_peered":"2024-03-26T03:53:12.409644+0000","last_clean":"2024-03-26T03:53:12.409644+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:53:12.409644+0000","last_undegraded":"2024-03-26T03:53:12.409644+0000","last_fullsized":"2024-03-26T03:53:12.409644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:45.403401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.403401+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.513250+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489183+0000","last_change":"2024-03-26T03:51:38.222500+0000","last_active":"2024-03-26T03:53:12.489183+0000","last_peered":"2024-03-26T03:53:12.489183+0000","last_clean":"2024-03-26T03:53:12.489183+0000","last_became_active":"2024-03-26T03:50:45.591442+0000","last_became_peered":"2024-03-26T03:50:45.591442+0000","last_unstale":"2024-03-26T03:53:12.489183+0000","last_undegraded":"2024-03-26T03:53:12.489183+0000","last_fullsized":"2024-03-26T03:53:12.489183+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.222375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392444+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403642+0000","last_change":"2024-03-26T03:52:49.690105+0000","last_active":"2024-03-26T03:53:12.403642+0000","last_peered":"2024-03-26T03:53:12.403642+0000","last_clean":"2024-03-26T03:53:12.403642+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:53:12.403642+0000","last_undegraded":"2024-03-26T03:53:12.403642+0000","last_fullsized":"2024-03-26T03:53:12.403642+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:49.690025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.690025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.171459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403206+0000","last_change":"2024-03-26T03:51:13.376075+0000","last_active":"2024-03-26T03:53:12.403206+0000","last_peered":"2024-03-26T03:53:12.403206+0000","last_clean":"2024-03-26T03:53:12.403206+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:53:12.403206+0000","last_undegraded":"2024-03-26T03:53:12.403206+0000","last_fullsized":"2024-03-26T03:53:12.403206+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:13.375964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:13.375964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.379961+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":205,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410760+0000","last_change":"2024-03-26T03:52:35.473032+0000","last_active":"2024-03-26T03:53:12.410760+0000","last_peered":"2024-03-26T03:53:12.410760+0000","last_clean":"2024-03-26T03:53:12.410760+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:53:12.410760+0000","last_undegraded":"2024-03-26T03:53:12.410760+0000","last_fullsized":"2024-03-26T03:53:12.410760+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.269829+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417399+0000","last_change":"2024-03-26T03:52:23.708646+0000","last_active":"2024-03-26T03:53:12.417399+0000","last_peered":"2024-03-26T03:53:12.417399+0000","last_clean":"2024-03-26T03:53:12.417399+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:53:12.417399+0000","last_undegraded":"2024-03-26T03:53:12.417399+0000","last_fullsized":"2024-03-26T03:53:12.417399+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:52:23.708510+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.708510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:25.385430+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403911+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:53:12.403911+0000","last_peered":"2024-03-26T03:53:12.403911+0000","last_clean":"2024-03-26T03:53:12.403911+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:53:12.403911+0000","last_undegraded":"2024-03-26T03:53:12.403911+0000","last_fullsized":"2024-03-26T03:53:12.403911+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.226521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:13.785476+0000","last_change":"2024-03-26T03:53:13.785620+0000","last_active":"2024-03-26T03:53:13.785476+0000","last_peered":"2024-03-26T03:53:13.785476+0000","last_clean":"2024-03-26T03:53:13.785476+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:53:13.785476+0000","last_undegraded":"2024-03-26T03:53:13.785476+0000","last_fullsized":"2024-03-26T03:53:13.785476+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:13.785265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.785265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:16.531268+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":198,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412672+0000","last_change":"2024-03-26T03:52:42.591753+0000","last_active":"2024-03-26T03:53:12.412672+0000","last_peered":"2024-03-26T03:53:12.412672+0000","last_clean":"2024-03-26T03:53:12.412672+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:53:12.412672+0000","last_undegraded":"2024-03-26T03:53:12.412672+0000","last_fullsized":"2024-03-26T03:53:12.412672+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:52:42.591503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.591503+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.339508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":161,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413155+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:53:12.413155+0000","last_peered":"2024-03-26T03:53:12.413155+0000","last_clean":"2024-03-26T03:53:12.413155+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:53:12.413155+0000","last_undegraded":"2024-03-26T03:53:12.413155+0000","last_fullsized":"2024-03-26T03:53:12.413155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:07.860041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.479629+0000","last_change":"2024-03-26T03:51:33.548319+0000","last_active":"2024-03-26T03:53:12.479629+0000","last_peered":"2024-03-26T03:53:12.479629+0000","last_clean":"2024-03-26T03:53:12.479629+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:53:12.479629+0000","last_undegraded":"2024-03-26T03:53:12.479629+0000","last_fullsized":"2024-03-26T03:53:12.479629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.548192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414309+0000","last_change":"2024-03-26T03:51:35.260411+0000","last_active":"2024-03-26T03:53:12.414309+0000","last_peered":"2024-03-26T03:53:12.414309+0000","last_clean":"2024-03-26T03:53:12.414309+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:53:12.414309+0000","last_undegraded":"2024-03-26T03:53:12.414309+0000","last_fullsized":"2024-03-26T03:53:12.414309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.258167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.258167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392359+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":179,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489364+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:53:12.489364+0000","last_peered":"2024-03-26T03:53:12.489364+0000","last_clean":"2024-03-26T03:53:12.489364+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:53:12.489364+0000","last_undegraded":"2024-03-26T03:53:12.489364+0000","last_fullsized":"2024-03-26T03:53:12.489364+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.838957+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416711+0000","last_change":"2024-03-26T03:52:34.605587+0000","last_active":"2024-03-26T03:53:12.416711+0000","last_peered":"2024-03-26T03:53:12.416711+0000","last_clean":"2024-03-26T03:53:12.416711+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:53:12.416711+0000","last_undegraded":"2024-03-26T03:53:12.416711+0000","last_fullsized":"2024-03-26T03:53:12.416711+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:34.605509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:34.605509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.539945+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403197+0000","last_change":"2024-03-26T03:51:39.355828+0000","last_active":"2024-03-26T03:53:12.403197+0000","last_peered":"2024-03-26T03:53:12.403197+0000","last_clean":"2024-03-26T03:53:12.403197+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:53:12.403197+0000","last_undegraded":"2024-03-26T03:53:12.403197+0000","last_fullsized":"2024-03-26T03:53:12.403197+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.355728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.355728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380180+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480609+0000","last_change":"2024-03-26T03:52:23.578870+0000","last_active":"2024-03-26T03:53:12.480609+0000","last_peered":"2024-03-26T03:53:12.480609+0000","last_clean":"2024-03-26T03:53:12.480609+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:53:12.480609+0000","last_undegraded":"2024-03-26T03:53:12.480609+0000","last_fullsized":"2024-03-26T03:53:12.480609+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.989992+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414368+0000","last_change":"2024-03-26T03:50:45.606641+0000","last_active":"2024-03-26T03:53:12.414368+0000","last_peered":"2024-03-26T03:53:12.414368+0000","last_clean":"2024-03-26T03:53:12.414368+0000","last_became_active":"2024-03-26T03:50:45.587808+0000","last_became_peered":"2024-03-26T03:50:45.587808+0000","last_unstale":"2024-03-26T03:53:12.414368+0000","last_undegraded":"2024-03-26T03:53:12.414368+0000","last_fullsized":"2024-03-26T03:53:12.414368+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-03-26T03:50:35.236237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.236237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390702+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.411059+0000","last_change":"2024-03-26T03:50:45.590379+0000","last_active":"2024-03-26T03:53:12.411059+0000","last_peered":"2024-03-26T03:53:12.411059+0000","last_clean":"2024-03-26T03:53:12.411059+0000","last_became_active":"2024-03-26T03:50:45.589899+0000","last_became_peered":"2024-03-26T03:50:45.589899+0000","last_unstale":"2024-03-26T03:53:12.411059+0000","last_undegraded":"2024-03-26T03:53:12.411059+0000","last_fullsized":"2024-03-26T03:53:12.411059+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:50:21.094208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390549+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417028+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:53:12.417028+0000","last_peered":"2024-03-26T03:53:12.417028+0000","last_clean":"2024-03-26T03:53:12.417028+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:53:12.417028+0000","last_undegraded":"2024-03-26T03:53:12.417028+0000","last_fullsized":"2024-03-26T03:53:12.417028+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.913379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":182,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489187+0000","last_change":"2024-03-26T03:52:39.291143+0000","last_active":"2024-03-26T03:53:12.489187+0000","last_peered":"2024-03-26T03:53:12.489187+0000","last_clean":"2024-03-26T03:53:12.489187+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:53:12.489187+0000","last_undegraded":"2024-03-26T03:53:12.489187+0000","last_fullsized":"2024-03-26T03:53:12.489187+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:39.291063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.291063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:55.327602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":219,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413798+0000","last_change":"2024-03-26T03:50:45.612604+0000","last_active":"2024-03-26T03:53:12.413798+0000","last_peered":"2024-03-26T03:53:12.413798+0000","last_clean":"2024-03-26T03:53:12.413798+0000","last_became_active":"2024-03-26T03:50:45.595599+0000","last_became_peered":"2024-03-26T03:50:45.595599+0000","last_unstale":"2024-03-26T03:53:12.413798+0000","last_undegraded":"2024-03-26T03:53:12.413798+0000","last_fullsized":"2024-03-26T03:53:12.413798+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:50:20.135174+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390582+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412163+0000","last_change":"2024-03-26T03:52:26.596430+0000","last_active":"2024-03-26T03:53:12.412163+0000","last_peered":"2024-03-26T03:53:12.412163+0000","last_clean":"2024-03-26T03:53:12.412163+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:53:12.412163+0000","last_undegraded":"2024-03-26T03:53:12.412163+0000","last_fullsized":"2024-03-26T03:53:12.412163+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.596311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.596311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.308150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410685+0000","last_change":"2024-03-26T03:52:40.410756+0000","last_active":"2024-03-26T03:53:12.410685+0000","last_peered":"2024-03-26T03:53:12.410685+0000","last_clean":"2024-03-26T03:53:12.410685+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:53:12.410685+0000","last_undegraded":"2024-03-26T03:53:12.410685+0000","last_fullsized":"2024-03-26T03:53:12.410685+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.947379+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417215+0000","last_change":"2024-03-26T03:51:15.916808+0000","last_active":"2024-03-26T03:53:12.417215+0000","last_peered":"2024-03-26T03:53:12.417215+0000","last_clean":"2024-03-26T03:53:12.417215+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:53:12.417215+0000","last_undegraded":"2024-03-26T03:53:12.417215+0000","last_fullsized":"2024-03-26T03:53:12.417215+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:15.916730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.916730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691936+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":201,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413762+0000","last_change":"2024-03-26T03:51:29.988370+0000","last_active":"2024-03-26T03:53:12.413762+0000","last_peered":"2024-03-26T03:53:12.413762+0000","last_clean":"2024-03-26T03:53:12.413762+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:53:12.413762+0000","last_undegraded":"2024-03-26T03:53:12.413762+0000","last_fullsized":"2024-03-26T03:53:12.413762+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:51:29.988111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.988111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.411132+0000","last_change":"2024-03-26T03:52:41.432452+0000","last_active":"2024-03-26T03:53:12.411132+0000","last_peered":"2024-03-26T03:53:12.411132+0000","last_clean":"2024-03-26T03:53:12.411132+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:53:12.411132+0000","last_undegraded":"2024-03-26T03:53:12.411132+0000","last_fullsized":"2024-03-26T03:53:12.411132+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:41.432332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:41.432332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.168460+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413608+0000","last_change":"2024-03-26T03:52:40.835637+0000","last_active":"2024-03-26T03:53:12.413608+0000","last_peered":"2024-03-26T03:53:12.413608+0000","last_clean":"2024-03-26T03:53:12.413608+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:53:12.413608+0000","last_undegraded":"2024-03-26T03:53:12.413608+0000","last_fullsized":"2024-03-26T03:53:12.413608+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.905532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403410+0000","last_change":"2024-03-26T03:51:30.319510+0000","last_active":"2024-03-26T03:53:12.403410+0000","last_peered":"2024-03-26T03:53:12.403410+0000","last_clean":"2024-03-26T03:53:12.403410+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:53:12.403410+0000","last_undegraded":"2024-03-26T03:53:12.403410+0000","last_fullsized":"2024-03-26T03:53:12.403410+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:30.319423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.319423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.379990+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":199,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403392+0000","last_change":"2024-03-26T03:51:17.380156+0000","last_active":"2024-03-26T03:53:12.403392+0000","last_peered":"2024-03-26T03:53:12.403392+0000","last_clean":"2024-03-26T03:53:12.403392+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:53:12.403392+0000","last_undegraded":"2024-03-26T03:53:12.403392+0000","last_fullsized":"2024-03-26T03:53:12.403392+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:51:17.380041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.380041+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412377+0000","last_change":"2024-03-26T03:51:19.175698+0000","last_active":"2024-03-26T03:53:12.412377+0000","last_peered":"2024-03-26T03:53:12.412377+0000","last_clean":"2024-03-26T03:53:12.412377+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:53:12.412377+0000","last_undegraded":"2024-03-26T03:53:12.412377+0000","last_fullsized":"2024-03-26T03:53:12.412377+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.175455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.175455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":237,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489103+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:53:12.489103+0000","last_peered":"2024-03-26T03:53:12.489103+0000","last_clean":"2024-03-26T03:53:12.489103+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:53:12.489103+0000","last_undegraded":"2024-03-26T03:53:12.489103+0000","last_fullsized":"2024-03-26T03:53:12.489103+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.768552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416895+0000","last_change":"2024-03-26T03:51:14.953771+0000","last_active":"2024-03-26T03:53:12.416895+0000","last_peered":"2024-03-26T03:53:12.416895+0000","last_clean":"2024-03-26T03:53:12.416895+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:53:12.416895+0000","last_undegraded":"2024-03-26T03:53:12.416895+0000","last_fullsized":"2024-03-26T03:53:12.416895+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:14.953632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:14.953632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691903+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":201,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413951+0000","last_change":"2024-03-26T03:51:42.190694+0000","last_active":"2024-03-26T03:53:12.413951+0000","last_peered":"2024-03-26T03:53:12.413951+0000","last_clean":"2024-03-26T03:53:12.413951+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:53:12.413951+0000","last_undegraded":"2024-03-26T03:53:12.413951+0000","last_fullsized":"2024-03-26T03:53:12.413951+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:42.190557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:42.190557+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392389+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.404086+0000","last_change":"2024-03-26T03:51:34.417542+0000","last_active":"2024-03-26T03:53:12.404086+0000","last_peered":"2024-03-26T03:53:12.404086+0000","last_clean":"2024-03-26T03:53:12.404086+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:53:12.404086+0000","last_undegraded":"2024-03-26T03:53:12.404086+0000","last_fullsized":"2024-03-26T03:53:12.404086+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:34.417463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.417463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414336+0000","last_change":"2024-03-26T03:51:34.258348+0000","last_active":"2024-03-26T03:53:12.414336+0000","last_peered":"2024-03-26T03:53:12.414336+0000","last_clean":"2024-03-26T03:53:12.414336+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:53:12.414336+0000","last_undegraded":"2024-03-26T03:53:12.414336+0000","last_fullsized":"2024-03-26T03:53:12.414336+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392419+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.601246+0000","last_change":"2024-03-26T03:53:12.601302+0000","last_active":"2024-03-26T03:53:12.601246+0000","last_peered":"2024-03-26T03:53:12.601246+0000","last_clean":"2024-03-26T03:53:12.601246+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:53:12.601246+0000","last_undegraded":"2024-03-26T03:53:12.601246+0000","last_fullsized":"2024-03-26T03:53:12.601246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:53:12.600988+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:12.600988+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:29.368784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":203,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489174+0000","last_change":"2024-03-26T03:51:36.601210+0000","last_active":"2024-03-26T03:53:12.489174+0000","last_peered":"2024-03-26T03:53:12.489174+0000","last_clean":"2024-03-26T03:53:12.489174+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:53:12.489174+0000","last_undegraded":"2024-03-26T03:53:12.489174+0000","last_fullsized":"2024-03-26T03:53:12.489174+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:51:36.601123+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:50:20.001688+0000","last_clean_scrub_stamp":"2024-03-26T03:51:36.601123+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":164,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.409961+0000","last_change":"2024-03-26T03:51:34.179163+0000","last_active":"2024-03-26T03:53:12.409961+0000","last_peered":"2024-03-26T03:53:12.409961+0000","last_clean":"2024-03-26T03:53:12.409961+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:53:12.409961+0000","last_undegraded":"2024-03-26T03:53:12.409961+0000","last_fullsized":"2024-03-26T03:53:12.409961+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:51:34.179031+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.179031+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":158,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.415169+0000","last_change":"2024-03-26T03:50:24.093555+0000","last_active":"2024-03-26T03:53:12.415169+0000","last_peered":"2024-03-26T03:53:12.415169+0000","last_clean":"2024-03-26T03:53:12.415169+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:53:12.415169+0000","last_undegraded":"2024-03-26T03:53:12.415169+0000","last_fullsized":"2024-03-26T03:53:12.415169+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:24.093421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:50:24.093421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416827+0000","last_change":"2024-03-26T03:51:19.966491+0000","last_active":"2024-03-26T03:53:12.416827+0000","last_peered":"2024-03-26T03:53:12.416827+0000","last_clean":"2024-03-26T03:53:12.416827+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:53:12.416827+0000","last_undegraded":"2024-03-26T03:53:12.416827+0000","last_fullsized":"2024-03-26T03:53:12.416827+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.966387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.966387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691965+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":210,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412860+0000","last_change":"2024-03-26T03:52:37.650277+0000","last_active":"2024-03-26T03:53:12.412860+0000","last_peered":"2024-03-26T03:53:12.412860+0000","last_clean":"2024-03-26T03:53:12.412860+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:53:12.412860+0000","last_undegraded":"2024-03-26T03:53:12.412860+0000","last_fullsized":"2024-03-26T03:53:12.412860+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.102642+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417109+0000","last_change":"2024-03-26T03:50:45.595675+0000","last_active":"2024-03-26T03:53:12.417109+0000","last_peered":"2024-03-26T03:53:12.417109+0000","last_clean":"2024-03-26T03:53:12.417109+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:53:12.417109+0000","last_undegraded":"2024-03-26T03:53:12.417109+0000","last_fullsized":"2024-03-26T03:53:12.417109+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:28.691326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:28.691326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691839+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489232+0000","last_change":"2024-03-26T03:51:38.612538+0000","last_active":"2024-03-26T03:53:12.489232+0000","last_peered":"2024-03-26T03:53:12.489232+0000","last_clean":"2024-03-26T03:53:12.489232+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:53:12.489232+0000","last_undegraded":"2024-03-26T03:53:12.489232+0000","last_fullsized":"2024-03-26T03:53:12.489232+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.612390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.612390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412487+0000","last_change":"2024-03-26T03:51:38.327055+0000","last_active":"2024-03-26T03:53:12.412487+0000","last_peered":"2024-03-26T03:53:12.412487+0000","last_clean":"2024-03-26T03:53:12.412487+0000","last_became_active":"2024-03-26T03:50:45.593133+0000","last_became_peered":"2024-03-26T03:50:45.593133+0000","last_unstale":"2024-03-26T03:53:12.412487+0000","last_undegraded":"2024-03-26T03:53:12.412487+0000","last_fullsized":"2024-03-26T03:53:12.412487+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.326923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.326923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":208,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489286+0000","last_change":"2024-03-26T03:52:33.466952+0000","last_active":"2024-03-26T03:53:12.489286+0000","last_peered":"2024-03-26T03:53:12.489286+0000","last_clean":"2024-03-26T03:53:12.489286+0000","last_became_active":"2024-03-26T03:50:24.531528+0000","last_became_peered":"2024-03-26T03:50:24.531528+0000","last_unstale":"2024-03-26T03:53:12.489286+0000","last_undegraded":"2024-03-26T03:53:12.489286+0000","last_fullsized":"2024-03-26T03:53:12.489286+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:48.107634+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"139'9663","reported_seq":9828,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.326480+0000","last_change":"2024-03-26T03:52:44.145650+0000","last_active":"2024-03-26T03:53:14.326480+0000","last_peered":"2024-03-26T03:53:14.326480+0000","last_clean":"2024-03-26T03:53:14.326480+0000","last_became_active":"2024-03-26T03:50:45.595232+0000","last_became_peered":"2024-03-26T03:50:45.595232+0000","last_unstale":"2024-03-26T03:53:14.326480+0000","last_undegraded":"2024-03-26T03:53:14.326480+0000","last_fullsized":"2024-03-26T03:53:14.326480+0000","mapping_epoch":96,"log_start":"120'5400","ondisk_log_start":"120'5400","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'7919","last_scrub_stamp":"2024-03-26T03:52:44.145396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.145396+0000","objects_scrubbed":5918,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.921446+0000","scrub_duration":2302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269811712,"num_objects":4117,"num_object_clones":0,"num_object_copies":12351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9663,"num_write_kb":440960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":177,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488928+0000","last_change":"2024-03-26T03:52:50.408368+0000","last_active":"2024-03-26T03:53:12.488928+0000","last_peered":"2024-03-26T03:53:12.488928+0000","last_clean":"2024-03-26T03:53:12.488928+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:53:12.488928+0000","last_undegraded":"2024-03-26T03:53:12.488928+0000","last_fullsized":"2024-03-26T03:53:12.488928+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.408254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.408254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.251022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":186,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417492+0000","last_change":"2024-03-26T03:52:32.622645+0000","last_active":"2024-03-26T03:53:12.417492+0000","last_peered":"2024-03-26T03:53:12.417492+0000","last_clean":"2024-03-26T03:53:12.417492+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:53:12.417492+0000","last_undegraded":"2024-03-26T03:53:12.417492+0000","last_fullsized":"2024-03-26T03:53:12.417492+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:52:32.622512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.622512+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.145922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":246,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488553+0000","last_change":"2024-03-26T03:52:22.438654+0000","last_active":"2024-03-26T03:53:12.488553+0000","last_peered":"2024-03-26T03:53:12.488553+0000","last_clean":"2024-03-26T03:53:12.488553+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:53:12.488553+0000","last_undegraded":"2024-03-26T03:53:12.488553+0000","last_fullsized":"2024-03-26T03:53:12.488553+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:52:22.438565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:22.438565+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.719143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":240,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.121684+0000","last_change":"2024-03-26T03:51:20.930418+0000","last_active":"2024-03-26T03:53:14.121684+0000","last_peered":"2024-03-26T03:53:14.121684+0000","last_clean":"2024-03-26T03:53:14.121684+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:53:14.121684+0000","last_undegraded":"2024-03-26T03:53:14.121684+0000","last_fullsized":"2024-03-26T03:53:14.121684+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:51:20.930317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:20.930317+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"139'9790","reported_seq":9887,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.481054+0000","last_change":"2024-03-26T03:50:44.592931+0000","last_active":"2024-03-26T03:53:14.481054+0000","last_peered":"2024-03-26T03:53:14.481054+0000","last_clean":"2024-03-26T03:53:14.481054+0000","last_became_active":"2024-03-26T03:50:44.592128+0000","last_became_peered":"2024-03-26T03:50:44.592128+0000","last_unstale":"2024-03-26T03:53:14.481054+0000","last_undegraded":"2024-03-26T03:53:14.481054+0000","last_fullsized":"2024-03-26T03:53:14.481054+0000","mapping_epoch":95,"log_start":"121'5600","ondisk_log_start":"121'5600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272629760,"num_objects":4160,"num_object_clones":0,"num_object_copies":12480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410861+0000","last_change":"2024-03-26T03:51:23.414926+0000","last_active":"2024-03-26T03:53:12.410861+0000","last_peered":"2024-03-26T03:53:12.410861+0000","last_clean":"2024-03-26T03:53:12.410861+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:53:12.410861+0000","last_undegraded":"2024-03-26T03:53:12.410861+0000","last_fullsized":"2024-03-26T03:53:12.410861+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417285+0000","last_change":"2024-03-26T03:52:31.638104+0000","last_active":"2024-03-26T03:53:12.417285+0000","last_peered":"2024-03-26T03:53:12.417285+0000","last_clean":"2024-03-26T03:53:12.417285+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:53:12.417285+0000","last_undegraded":"2024-03-26T03:53:12.417285+0000","last_fullsized":"2024-03-26T03:53:12.417285+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.637987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.637987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.891585+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":218,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403419+0000","last_change":"2024-03-26T03:52:30.697090+0000","last_active":"2024-03-26T03:53:12.403419+0000","last_peered":"2024-03-26T03:53:12.403419+0000","last_clean":"2024-03-26T03:53:12.403419+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:53:12.403419+0000","last_undegraded":"2024-03-26T03:53:12.403419+0000","last_fullsized":"2024-03-26T03:53:12.403419+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.990440+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410069+0000","last_change":"2024-03-26T03:51:36.110972+0000","last_active":"2024-03-26T03:53:12.410069+0000","last_peered":"2024-03-26T03:53:12.410069+0000","last_clean":"2024-03-26T03:53:12.410069+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:53:12.410069+0000","last_undegraded":"2024-03-26T03:53:12.410069+0000","last_fullsized":"2024-03-26T03:53:12.410069+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:36.110583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:36.110583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"139'10228","reported_seq":10402,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.444126+0000","last_change":"2024-03-26T03:52:36.221752+0000","last_active":"2024-03-26T03:53:14.444126+0000","last_peered":"2024-03-26T03:53:14.444126+0000","last_clean":"2024-03-26T03:53:14.444126+0000","last_became_active":"2024-03-26T03:50:44.585858+0000","last_became_peered":"2024-03-26T03:50:44.585858+0000","last_unstale":"2024-03-26T03:53:14.444126+0000","last_undegraded":"2024-03-26T03:53:14.444126+0000","last_fullsized":"2024-03-26T03:53:14.444126+0000","mapping_epoch":95,"log_start":"120'5400","ondisk_log_start":"120'5400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7838","last_scrub_stamp":"2024-03-26T03:52:36.221537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:36.221537+0000","objects_scrubbed":6697,"log_size":4828,"log_dups_size":0,"ondisk_log_size":4828,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.562362+0000","scrub_duration":2663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276168704,"num_objects":4214,"num_object_clones":0,"num_object_copies":12642,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4214,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10228,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416602+0000","last_change":"2024-03-26T03:51:27.913279+0000","last_active":"2024-03-26T03:53:12.416602+0000","last_peered":"2024-03-26T03:53:12.416602+0000","last_clean":"2024-03-26T03:53:12.416602+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:53:12.416602+0000","last_undegraded":"2024-03-26T03:53:12.416602+0000","last_fullsized":"2024-03-26T03:53:12.416602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:27.913164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.913164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691995+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":166,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.415521+0000","last_change":"2024-03-26T03:52:29.251669+0000","last_active":"2024-03-26T03:53:12.415521+0000","last_peered":"2024-03-26T03:53:12.415521+0000","last_clean":"2024-03-26T03:53:12.415521+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:53:12.415521+0000","last_undegraded":"2024-03-26T03:53:12.415521+0000","last_fullsized":"2024-03-26T03:53:12.415521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.251258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.251258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.082401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417403+0000","last_change":"2024-03-26T03:52:35.600199+0000","last_active":"2024-03-26T03:53:12.417403+0000","last_peered":"2024-03-26T03:53:12.417403+0000","last_clean":"2024-03-26T03:53:12.417403+0000","last_became_active":"2024-03-26T03:50:45.594682+0000","last_became_peered":"2024-03-26T03:50:45.594682+0000","last_unstale":"2024-03-26T03:53:12.417403+0000","last_undegraded":"2024-03-26T03:53:12.417403+0000","last_fullsized":"2024-03-26T03:53:12.417403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:35.600091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.600091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:57.681359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413571+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:53:12.413571+0000","last_peered":"2024-03-26T03:53:12.413571+0000","last_clean":"2024-03-26T03:53:12.413571+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:53:12.413571+0000","last_undegraded":"2024-03-26T03:53:12.413571+0000","last_fullsized":"2024-03-26T03:53:12.413571+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.717740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"139'9934","reported_seq":10108,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.345800+0000","last_change":"2024-03-26T03:52:33.625525+0000","last_active":"2024-03-26T03:53:14.345800+0000","last_peered":"2024-03-26T03:53:14.345800+0000","last_clean":"2024-03-26T03:53:14.345800+0000","last_became_active":"2024-03-26T03:50:44.589615+0000","last_became_peered":"2024-03-26T03:50:44.589615+0000","last_unstale":"2024-03-26T03:53:14.345800+0000","last_undegraded":"2024-03-26T03:53:14.345800+0000","last_fullsized":"2024-03-26T03:53:14.345800+0000","mapping_epoch":95,"log_start":"117'4400","ondisk_log_start":"117'4400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7504","last_scrub_stamp":"2024-03-26T03:52:33.625304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.625304+0000","objects_scrubbed":6650,"log_size":5534,"log_dups_size":0,"ondisk_log_size":5534,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.340680+0000","scrub_duration":2355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271450112,"num_objects":4142,"num_object_clones":0,"num_object_copies":12426,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4142,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9934,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412281+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:53:12.412281+0000","last_peered":"2024-03-26T03:53:12.412281+0000","last_clean":"2024-03-26T03:53:12.412281+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:53:12.412281+0000","last_undegraded":"2024-03-26T03:53:12.412281+0000","last_fullsized":"2024-03-26T03:53:12.412281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.605441+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.415830+0000","last_change":"2024-03-26T03:52:42.382709+0000","last_active":"2024-03-26T03:53:12.415830+0000","last_peered":"2024-03-26T03:53:12.415830+0000","last_clean":"2024-03-26T03:53:12.415830+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:53:12.415830+0000","last_undegraded":"2024-03-26T03:53:12.415830+0000","last_fullsized":"2024-03-26T03:53:12.415830+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.382447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.382447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:08.004037+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488819+0000","last_change":"2024-03-26T03:52:31.320265+0000","last_active":"2024-03-26T03:53:12.488819+0000","last_peered":"2024-03-26T03:53:12.488819+0000","last_clean":"2024-03-26T03:53:12.488819+0000","last_became_active":"2024-03-26T03:50:45.592202+0000","last_became_peered":"2024-03-26T03:50:45.592202+0000","last_unstale":"2024-03-26T03:53:12.488819+0000","last_undegraded":"2024-03-26T03:53:12.488819+0000","last_fullsized":"2024-03-26T03:53:12.488819+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.320181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.320181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:37.000450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":216,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480226+0000","last_change":"2024-03-26T03:51:34.486120+0000","last_active":"2024-03-26T03:53:12.480226+0000","last_peered":"2024-03-26T03:53:12.480226+0000","last_clean":"2024-03-26T03:53:12.480226+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:53:12.480226+0000","last_undegraded":"2024-03-26T03:53:12.480226+0000","last_fullsized":"2024-03-26T03:53:12.480226+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:34.485979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.485979+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"139'10007","reported_seq":10172,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.452805+0000","last_change":"2024-03-26T03:52:48.148490+0000","last_active":"2024-03-26T03:53:14.452805+0000","last_peered":"2024-03-26T03:53:14.452805+0000","last_clean":"2024-03-26T03:53:14.452805+0000","last_became_active":"2024-03-26T03:50:45.590035+0000","last_became_peered":"2024-03-26T03:50:45.590035+0000","last_unstale":"2024-03-26T03:53:14.452805+0000","last_undegraded":"2024-03-26T03:53:14.452805+0000","last_fullsized":"2024-03-26T03:53:14.452805+0000","mapping_epoch":96,"log_start":"120'5200","ondisk_log_start":"120'5200","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8409","last_scrub_stamp":"2024-03-26T03:52:48.148171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.148171+0000","objects_scrubbed":5895,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.767336+0000","scrub_duration":3557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276627456,"num_objects":4221,"num_object_clones":0,"num_object_copies":12663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10007,"num_write_kb":455296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414446+0000","last_change":"2024-03-26T03:51:25.463769+0000","last_active":"2024-03-26T03:53:12.414446+0000","last_peered":"2024-03-26T03:53:12.414446+0000","last_clean":"2024-03-26T03:53:12.414446+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:53:12.414446+0000","last_undegraded":"2024-03-26T03:53:12.414446+0000","last_fullsized":"2024-03-26T03:53:12.414446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.463664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.463664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392280+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488673+0000","last_change":"2024-03-26T03:51:41.572876+0000","last_active":"2024-03-26T03:53:12.488673+0000","last_peered":"2024-03-26T03:53:12.488673+0000","last_clean":"2024-03-26T03:53:12.488673+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:53:12.488673+0000","last_undegraded":"2024-03-26T03:53:12.488673+0000","last_fullsized":"2024-03-26T03:53:12.488673+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:51:41.572734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.572734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.440961+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":181,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403096+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:53:12.403096+0000","last_peered":"2024-03-26T03:53:12.403096+0000","last_clean":"2024-03-26T03:53:12.403096+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:53:12.403096+0000","last_undegraded":"2024-03-26T03:53:12.403096+0000","last_fullsized":"2024-03-26T03:53:12.403096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.516195+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":285,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413679+0000","last_change":"2024-03-26T03:51:34.973521+0000","last_active":"2024-03-26T03:53:12.413679+0000","last_peered":"2024-03-26T03:53:12.413679+0000","last_clean":"2024-03-26T03:53:12.413679+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:53:12.413679+0000","last_undegraded":"2024-03-26T03:53:12.413679+0000","last_fullsized":"2024-03-26T03:53:12.413679+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:51:34.973366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.973366+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412601+0000","last_change":"2024-03-26T03:52:25.587745+0000","last_active":"2024-03-26T03:53:12.412601+0000","last_peered":"2024-03-26T03:53:12.412601+0000","last_clean":"2024-03-26T03:53:12.412601+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:53:12.412601+0000","last_undegraded":"2024-03-26T03:53:12.412601+0000","last_fullsized":"2024-03-26T03:53:12.412601+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.587502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.587502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:25.609331+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"139'9939","reported_seq":10113,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.474273+0000","last_change":"2024-03-26T03:52:33.050130+0000","last_active":"2024-03-26T03:53:14.474273+0000","last_peered":"2024-03-26T03:53:14.474273+0000","last_clean":"2024-03-26T03:53:14.474273+0000","last_became_active":"2024-03-26T03:50:44.584865+0000","last_became_peered":"2024-03-26T03:50:44.584865+0000","last_unstale":"2024-03-26T03:53:14.474273+0000","last_undegraded":"2024-03-26T03:53:14.474273+0000","last_fullsized":"2024-03-26T03:53:14.474273+0000","mapping_epoch":95,"log_start":"119'5100","ondisk_log_start":"119'5100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7431","last_scrub_stamp":"2024-03-26T03:52:33.049847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.049847+0000","objects_scrubbed":6680,"log_size":4839,"log_dups_size":0,"ondisk_log_size":4839,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.994227+0000","scrub_duration":2449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269549568,"num_objects":4113,"num_object_clones":0,"num_object_copies":12339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9939,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417583+0000","last_change":"2024-03-26T03:52:44.692881+0000","last_active":"2024-03-26T03:53:12.417583+0000","last_peered":"2024-03-26T03:53:12.417583+0000","last_clean":"2024-03-26T03:53:12.417583+0000","last_became_active":"2024-03-26T03:50:45.583936+0000","last_became_peered":"2024-03-26T03:50:45.583936+0000","last_unstale":"2024-03-26T03:53:12.417583+0000","last_undegraded":"2024-03-26T03:53:12.417583+0000","last_fullsized":"2024-03-26T03:53:12.417583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.026793+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412288+0000","last_change":"2024-03-26T03:51:32.007583+0000","last_active":"2024-03-26T03:53:12.412288+0000","last_peered":"2024-03-26T03:53:12.412288+0000","last_clean":"2024-03-26T03:53:12.412288+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:53:12.412288+0000","last_undegraded":"2024-03-26T03:53:12.412288+0000","last_fullsized":"2024-03-26T03:53:12.412288+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.007458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.007458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":192,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.408923+0000","last_change":"2024-03-26T03:52:40.285553+0000","last_active":"2024-03-26T03:53:12.408923+0000","last_peered":"2024-03-26T03:53:12.408923+0000","last_clean":"2024-03-26T03:53:12.408923+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:53:12.408923+0000","last_undegraded":"2024-03-26T03:53:12.408923+0000","last_fullsized":"2024-03-26T03:53:12.408923+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.285400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.285400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.875607+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":209,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.483616+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:53:12.483616+0000","last_peered":"2024-03-26T03:53:12.483616+0000","last_clean":"2024-03-26T03:53:12.483616+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:53:12.483616+0000","last_undegraded":"2024-03-26T03:53:12.483616+0000","last_fullsized":"2024-03-26T03:53:12.483616+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.706675+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.409157+0000","last_change":"2024-03-26T03:52:25.178369+0000","last_active":"2024-03-26T03:53:12.409157+0000","last_peered":"2024-03-26T03:53:12.409157+0000","last_clean":"2024-03-26T03:53:12.409157+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:53:12.409157+0000","last_undegraded":"2024-03-26T03:53:12.409157+0000","last_fullsized":"2024-03-26T03:53:12.409157+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.178239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.178239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.907344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"139'9816","reported_seq":9979,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.478550+0000","last_change":"2024-03-26T03:52:49.471047+0000","last_active":"2024-03-26T03:53:14.478550+0000","last_peered":"2024-03-26T03:53:14.478550+0000","last_clean":"2024-03-26T03:53:14.478550+0000","last_became_active":"2024-03-26T03:50:44.588356+0000","last_became_peered":"2024-03-26T03:50:44.588356+0000","last_unstale":"2024-03-26T03:53:14.478550+0000","last_undegraded":"2024-03-26T03:53:14.478550+0000","last_fullsized":"2024-03-26T03:53:14.478550+0000","mapping_epoch":95,"log_start":"119'5000","ondisk_log_start":"119'5000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8243","last_scrub_stamp":"2024-03-26T03:52:49.470707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.470707+0000","objects_scrubbed":5693,"log_size":4816,"log_dups_size":0,"ondisk_log_size":4816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.948331+0000","scrub_duration":2903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264634368,"num_objects":4038,"num_object_clones":0,"num_object_copies":12114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4038,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9816,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":163,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489194+0000","last_change":"2024-03-26T03:51:30.713391+0000","last_active":"2024-03-26T03:53:12.489194+0000","last_peered":"2024-03-26T03:53:12.489194+0000","last_clean":"2024-03-26T03:53:12.489194+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:53:12.489194+0000","last_undegraded":"2024-03-26T03:53:12.489194+0000","last_fullsized":"2024-03-26T03:53:12.489194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:30.713294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.713294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":438,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.416704+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:53:12.416704+0000","last_peered":"2024-03-26T03:53:12.416704+0000","last_clean":"2024-03-26T03:53:12.416704+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:53:12.416704+0000","last_undegraded":"2024-03-26T03:53:12.416704+0000","last_fullsized":"2024-03-26T03:53:12.416704+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:14.983642+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":173,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414765+0000","last_change":"2024-03-26T03:51:18.413053+0000","last_active":"2024-03-26T03:53:12.414765+0000","last_peered":"2024-03-26T03:53:12.414765+0000","last_clean":"2024-03-26T03:53:12.414765+0000","last_became_active":"2024-03-26T03:50:45.592343+0000","last_became_peered":"2024-03-26T03:50:45.592343+0000","last_unstale":"2024-03-26T03:53:12.414765+0000","last_undegraded":"2024-03-26T03:53:12.414765+0000","last_fullsized":"2024-03-26T03:53:12.414765+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.412940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.412940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":196,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403790+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:53:12.403790+0000","last_peered":"2024-03-26T03:53:12.403790+0000","last_clean":"2024-03-26T03:53:12.403790+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:53:12.403790+0000","last_undegraded":"2024-03-26T03:53:12.403790+0000","last_fullsized":"2024-03-26T03:53:12.403790+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.072965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417335+0000","last_change":"2024-03-26T03:51:39.840265+0000","last_active":"2024-03-26T03:53:12.417335+0000","last_peered":"2024-03-26T03:53:12.417335+0000","last_clean":"2024-03-26T03:53:12.417335+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:53:12.417335+0000","last_undegraded":"2024-03-26T03:53:12.417335+0000","last_fullsized":"2024-03-26T03:53:12.417335+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.840138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.840138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691951+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"139'10090","reported_seq":10188,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.184048+0000","last_change":"2024-03-26T03:50:44.586811+0000","last_active":"2024-03-26T03:53:14.184048+0000","last_peered":"2024-03-26T03:53:14.184048+0000","last_clean":"2024-03-26T03:53:14.184048+0000","last_became_active":"2024-03-26T03:50:44.586504+0000","last_became_peered":"2024-03-26T03:50:44.586504+0000","last_unstale":"2024-03-26T03:53:14.184048+0000","last_undegraded":"2024-03-26T03:53:14.184048+0000","last_fullsized":"2024-03-26T03:53:14.184048+0000","mapping_epoch":95,"log_start":"118'5000","ondisk_log_start":"118'5000","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":5090,"log_dups_size":0,"ondisk_log_size":5090,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274071552,"num_objects":4182,"num_object_clones":0,"num_object_copies":12546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10090,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":154,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413146+0000","last_change":"2024-03-26T03:50:37.125164+0000","last_active":"2024-03-26T03:53:12.413146+0000","last_peered":"2024-03-26T03:53:12.413146+0000","last_clean":"2024-03-26T03:53:12.413146+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:53:12.413146+0000","last_undegraded":"2024-03-26T03:53:12.413146+0000","last_fullsized":"2024-03-26T03:53:12.413146+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:37.125085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:37.125085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.411937+0000","last_change":"2024-03-26T03:51:40.382951+0000","last_active":"2024-03-26T03:53:12.411937+0000","last_peered":"2024-03-26T03:53:12.411937+0000","last_clean":"2024-03-26T03:53:12.411937+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:53:12.411937+0000","last_undegraded":"2024-03-26T03:53:12.411937+0000","last_fullsized":"2024-03-26T03:53:12.411937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:40.377234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:40.377234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":450,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:13.135126+0000","last_change":"2024-03-26T03:51:17.909032+0000","last_active":"2024-03-26T03:53:13.135126+0000","last_peered":"2024-03-26T03:53:13.135126+0000","last_clean":"2024-03-26T03:53:13.135126+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:53:13.135126+0000","last_undegraded":"2024-03-26T03:53:13.135126+0000","last_fullsized":"2024-03-26T03:53:13.135126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:51:17.908906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.908906+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":263,"num_read_kb":258,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":189,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412418+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:53:12.412418+0000","last_peered":"2024-03-26T03:53:12.412418+0000","last_clean":"2024-03-26T03:53:12.412418+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:53:12.412418+0000","last_undegraded":"2024-03-26T03:53:12.412418+0000","last_fullsized":"2024-03-26T03:53:12.412418+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.887919+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489450+0000","last_change":"2024-03-26T03:52:25.471379+0000","last_active":"2024-03-26T03:53:12.489450+0000","last_peered":"2024-03-26T03:53:12.489450+0000","last_clean":"2024-03-26T03:53:12.489450+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:53:12.489450+0000","last_undegraded":"2024-03-26T03:53:12.489450+0000","last_fullsized":"2024-03-26T03:53:12.489450+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.471302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.471302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:32.035263+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"139'10060","reported_seq":10236,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.178554+0000","last_change":"2024-03-26T03:52:31.921264+0000","last_active":"2024-03-26T03:53:14.178554+0000","last_peered":"2024-03-26T03:53:14.178554+0000","last_clean":"2024-03-26T03:53:14.178554+0000","last_became_active":"2024-03-26T03:50:44.593854+0000","last_became_peered":"2024-03-26T03:50:44.593854+0000","last_unstale":"2024-03-26T03:53:14.178554+0000","last_undegraded":"2024-03-26T03:53:14.178554+0000","last_fullsized":"2024-03-26T03:53:14.178554+0000","mapping_epoch":95,"log_start":"118'4700","ondisk_log_start":"118'4700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7468","last_scrub_stamp":"2024-03-26T03:52:31.921157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.921157+0000","objects_scrubbed":6796,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.185172+0000","scrub_duration":1567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273022976,"num_objects":4166,"num_object_clones":0,"num_object_copies":12498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10060,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":168,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480387+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:53:12.480387+0000","last_peered":"2024-03-26T03:53:12.480387+0000","last_clean":"2024-03-26T03:53:12.480387+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:53:12.480387+0000","last_undegraded":"2024-03-26T03:53:12.480387+0000","last_fullsized":"2024-03-26T03:53:12.480387+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.478472+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":174,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413543+0000","last_change":"2024-03-26T03:51:20.065167+0000","last_active":"2024-03-26T03:53:12.413543+0000","last_peered":"2024-03-26T03:53:12.413543+0000","last_clean":"2024-03-26T03:53:12.413543+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:53:12.413543+0000","last_undegraded":"2024-03-26T03:53:12.413543+0000","last_fullsized":"2024-03-26T03:53:12.413543+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:20.065038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:20.065038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":219,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412498+0000","last_change":"2024-03-26T03:51:32.540653+0000","last_active":"2024-03-26T03:53:12.412498+0000","last_peered":"2024-03-26T03:53:12.412498+0000","last_clean":"2024-03-26T03:53:12.412498+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:53:12.412498+0000","last_undegraded":"2024-03-26T03:53:12.412498+0000","last_fullsized":"2024-03-26T03:53:12.412498+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:51:32.540268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.540268+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.480057+0000","last_change":"2024-03-26T03:51:41.399750+0000","last_active":"2024-03-26T03:53:12.480057+0000","last_peered":"2024-03-26T03:53:12.480057+0000","last_clean":"2024-03-26T03:53:12.480057+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:53:12.480057+0000","last_undegraded":"2024-03-26T03:53:12.480057+0000","last_fullsized":"2024-03-26T03:53:12.480057+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.399622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.399622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"139'9908","reported_seq":10006,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.183165+0000","last_change":"2024-03-26T03:50:45.597482+0000","last_active":"2024-03-26T03:53:14.183165+0000","last_peered":"2024-03-26T03:53:14.183165+0000","last_clean":"2024-03-26T03:53:14.183165+0000","last_became_active":"2024-03-26T03:50:45.596978+0000","last_became_peered":"2024-03-26T03:50:45.596978+0000","last_unstale":"2024-03-26T03:53:14.183165+0000","last_undegraded":"2024-03-26T03:53:14.183165+0000","last_fullsized":"2024-03-26T03:53:14.183165+0000","mapping_epoch":96,"log_start":"120'5300","ondisk_log_start":"120'5300","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4608,"log_dups_size":0,"ondisk_log_size":4608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272236544,"num_objects":4154,"num_object_clones":0,"num_object_copies":12462,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4154,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9908,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412265+0000","last_change":"2024-03-26T03:51:18.180126+0000","last_active":"2024-03-26T03:53:12.412265+0000","last_peered":"2024-03-26T03:53:12.412265+0000","last_clean":"2024-03-26T03:53:12.412265+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:53:12.412265+0000","last_undegraded":"2024-03-26T03:53:12.412265+0000","last_fullsized":"2024-03-26T03:53:12.412265+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.179908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.179908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489080+0000","last_change":"2024-03-26T03:52:29.399095+0000","last_active":"2024-03-26T03:53:12.489080+0000","last_peered":"2024-03-26T03:53:12.489080+0000","last_clean":"2024-03-26T03:53:12.489080+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:53:12.489080+0000","last_undegraded":"2024-03-26T03:53:12.489080+0000","last_fullsized":"2024-03-26T03:53:12.489080+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.398965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.398965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.994443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":197,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413680+0000","last_change":"2024-03-26T03:51:17.091083+0000","last_active":"2024-03-26T03:53:12.413680+0000","last_peered":"2024-03-26T03:53:12.413680+0000","last_clean":"2024-03-26T03:53:12.413680+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:53:12.413680+0000","last_undegraded":"2024-03-26T03:53:12.413680+0000","last_fullsized":"2024-03-26T03:53:12.413680+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:51:17.090921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.090921+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":232,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412277+0000","last_change":"2024-03-26T03:51:15.205153+0000","last_active":"2024-03-26T03:53:12.412277+0000","last_peered":"2024-03-26T03:53:12.412277+0000","last_clean":"2024-03-26T03:53:12.412277+0000","last_became_active":"2024-03-26T03:50:45.602990+0000","last_became_peered":"2024-03-26T03:50:45.602990+0000","last_unstale":"2024-03-26T03:53:12.412277+0000","last_undegraded":"2024-03-26T03:53:12.412277+0000","last_fullsized":"2024-03-26T03:53:12.412277+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:51:15.205025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.205025+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"139'9762","reported_seq":9861,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.385078+0000","last_change":"2024-03-26T03:50:44.586742+0000","last_active":"2024-03-26T03:53:14.385078+0000","last_peered":"2024-03-26T03:53:14.385078+0000","last_clean":"2024-03-26T03:53:14.385078+0000","last_became_active":"2024-03-26T03:50:44.586424+0000","last_became_peered":"2024-03-26T03:50:44.586424+0000","last_unstale":"2024-03-26T03:53:14.385078+0000","last_undegraded":"2024-03-26T03:53:14.385078+0000","last_fullsized":"2024-03-26T03:53:14.385078+0000","mapping_epoch":95,"log_start":"121'5500","ondisk_log_start":"121'5500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4262,"log_dups_size":0,"ondisk_log_size":4262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9762,"num_write_kb":441792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":169,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.417062+0000","last_change":"2024-03-26T03:51:33.934747+0000","last_active":"2024-03-26T03:53:12.417062+0000","last_peered":"2024-03-26T03:53:12.417062+0000","last_clean":"2024-03-26T03:53:12.417062+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:53:12.417062+0000","last_undegraded":"2024-03-26T03:53:12.417062+0000","last_fullsized":"2024-03-26T03:53:12.417062+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.934617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:16.691980+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":176,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.412518+0000","last_change":"2024-03-26T03:51:30.466650+0000","last_active":"2024-03-26T03:53:12.412518+0000","last_peered":"2024-03-26T03:53:12.412518+0000","last_clean":"2024-03-26T03:53:12.412518+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:53:12.412518+0000","last_undegraded":"2024-03-26T03:53:12.412518+0000","last_fullsized":"2024-03-26T03:53:12.412518+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:51:30.466512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.466512+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":228,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403302+0000","last_change":"2024-03-26T03:51:29.300134+0000","last_active":"2024-03-26T03:53:12.403302+0000","last_peered":"2024-03-26T03:53:12.403302+0000","last_clean":"2024-03-26T03:53:12.403302+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:53:12.403302+0000","last_undegraded":"2024-03-26T03:53:12.403302+0000","last_fullsized":"2024-03-26T03:53:12.403302+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:51:29.299953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.299953+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380102+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":230,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414521+0000","last_change":"2024-03-26T03:52:31.483741+0000","last_active":"2024-03-26T03:53:12.414521+0000","last_peered":"2024-03-26T03:53:12.414521+0000","last_clean":"2024-03-26T03:53:12.414521+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:53:12.414521+0000","last_undegraded":"2024-03-26T03:53:12.414521+0000","last_fullsized":"2024-03-26T03:53:12.414521+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:31.483593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.483593+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.632795+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"139'9910","reported_seq":10009,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.476134+0000","last_change":"2024-03-26T03:50:44.593372+0000","last_active":"2024-03-26T03:53:14.476134+0000","last_peered":"2024-03-26T03:53:14.476134+0000","last_clean":"2024-03-26T03:53:14.476134+0000","last_became_active":"2024-03-26T03:50:44.592595+0000","last_became_peered":"2024-03-26T03:50:44.592595+0000","last_unstale":"2024-03-26T03:53:14.476134+0000","last_undegraded":"2024-03-26T03:53:14.476134+0000","last_fullsized":"2024-03-26T03:53:14.476134+0000","mapping_epoch":95,"log_start":"121'5700","ondisk_log_start":"121'5700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4210,"log_dups_size":0,"ondisk_log_size":4210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.390732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269484032,"num_objects":4112,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9910,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":155,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403071+0000","last_change":"2024-03-26T03:51:21.386995+0000","last_active":"2024-03-26T03:53:12.403071+0000","last_peered":"2024-03-26T03:53:12.403071+0000","last_clean":"2024-03-26T03:53:12.403071+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:53:12.403071+0000","last_undegraded":"2024-03-26T03:53:12.403071+0000","last_fullsized":"2024-03-26T03:53:12.403071+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.386911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.386911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380118+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488797+0000","last_change":"2024-03-26T03:52:45.556559+0000","last_active":"2024-03-26T03:53:12.488797+0000","last_peered":"2024-03-26T03:53:12.488797+0000","last_clean":"2024-03-26T03:53:12.488797+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:53:12.488797+0000","last_undegraded":"2024-03-26T03:53:12.488797+0000","last_fullsized":"2024-03-26T03:53:12.488797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.193240+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410015+0000","last_change":"2024-03-26T03:52:46.432028+0000","last_active":"2024-03-26T03:53:12.410015+0000","last_peered":"2024-03-26T03:53:12.410015+0000","last_clean":"2024-03-26T03:53:12.410015+0000","last_became_active":"2024-03-26T03:51:24.676652+0000","last_became_peered":"2024-03-26T03:51:24.676652+0000","last_unstale":"2024-03-26T03:53:12.410015+0000","last_undegraded":"2024-03-26T03:53:12.410015+0000","last_fullsized":"2024-03-26T03:53:12.410015+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:52:46.429387+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.429387+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.072480+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.402916+0000","last_change":"2024-03-26T03:52:37.885150+0000","last_active":"2024-03-26T03:53:12.402916+0000","last_peered":"2024-03-26T03:53:12.402916+0000","last_clean":"2024-03-26T03:53:12.402916+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:53:12.402916+0000","last_undegraded":"2024-03-26T03:53:12.402916+0000","last_fullsized":"2024-03-26T03:53:12.402916+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.707219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"139'9870","reported_seq":9967,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.433157+0000","last_change":"2024-03-26T03:50:45.583826+0000","last_active":"2024-03-26T03:53:14.433157+0000","last_peered":"2024-03-26T03:53:14.433157+0000","last_clean":"2024-03-26T03:53:14.433157+0000","last_became_active":"2024-03-26T03:50:45.583350+0000","last_became_peered":"2024-03-26T03:50:45.583350+0000","last_unstale":"2024-03-26T03:53:14.433157+0000","last_undegraded":"2024-03-26T03:53:14.433157+0000","last_fullsized":"2024-03-26T03:53:14.433157+0000","mapping_epoch":96,"log_start":"122'5700","ondisk_log_start":"122'5700","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4170,"log_dups_size":0,"ondisk_log_size":4170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265551872,"num_objects":4052,"num_object_clones":0,"num_object_copies":12156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9870,"num_write_kb":445504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.488952+0000","last_change":"2024-03-26T03:51:35.580349+0000","last_active":"2024-03-26T03:53:12.488952+0000","last_peered":"2024-03-26T03:53:12.488952+0000","last_clean":"2024-03-26T03:53:12.488952+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:53:12.488952+0000","last_undegraded":"2024-03-26T03:53:12.488952+0000","last_fullsized":"2024-03-26T03:53:12.488952+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.580269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.580269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":229,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.404151+0000","last_change":"2024-03-26T03:52:20.706812+0000","last_active":"2024-03-26T03:53:12.404151+0000","last_peered":"2024-03-26T03:53:12.404151+0000","last_clean":"2024-03-26T03:53:12.404151+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:53:12.404151+0000","last_undegraded":"2024-03-26T03:53:12.404151+0000","last_fullsized":"2024-03-26T03:53:12.404151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:20.706714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:20.706714+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:29.973564+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.414472+0000","last_change":"2024-03-26T03:51:26.426050+0000","last_active":"2024-03-26T03:53:12.414472+0000","last_peered":"2024-03-26T03:53:12.414472+0000","last_clean":"2024-03-26T03:53:12.414472+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:53:12.414472+0000","last_undegraded":"2024-03-26T03:53:12.414472+0000","last_fullsized":"2024-03-26T03:53:12.414472+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:26.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:26.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392333+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":194,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.483504+0000","last_change":"2024-03-26T03:51:38.907371+0000","last_active":"2024-03-26T03:53:12.483504+0000","last_peered":"2024-03-26T03:53:12.483504+0000","last_clean":"2024-03-26T03:53:12.483504+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:53:12.483504+0000","last_undegraded":"2024-03-26T03:53:12.483504+0000","last_fullsized":"2024-03-26T03:53:12.483504+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:38.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.907089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"139'9763","reported_seq":9861,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.287213+0000","last_change":"2024-03-26T03:50:45.591630+0000","last_active":"2024-03-26T03:53:14.287213+0000","last_peered":"2024-03-26T03:53:14.287213+0000","last_clean":"2024-03-26T03:53:14.287213+0000","last_became_active":"2024-03-26T03:50:45.591306+0000","last_became_peered":"2024-03-26T03:50:45.591306+0000","last_unstale":"2024-03-26T03:53:14.287213+0000","last_undegraded":"2024-03-26T03:53:14.287213+0000","last_fullsized":"2024-03-26T03:53:14.287213+0000","mapping_epoch":96,"log_start":"121'5500","ondisk_log_start":"121'5500","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4263,"log_dups_size":0,"ondisk_log_size":4263,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268697624,"num_objects":4101,"num_object_clones":0,"num_object_copies":12303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4101,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9763,"num_write_kb":443585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":160,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413845+0000","last_change":"2024-03-26T03:51:19.390232+0000","last_active":"2024-03-26T03:53:12.413845+0000","last_peered":"2024-03-26T03:53:12.413845+0000","last_clean":"2024-03-26T03:53:12.413845+0000","last_became_active":"2024-03-26T03:50:45.595009+0000","last_became_peered":"2024-03-26T03:50:45.595009+0000","last_unstale":"2024-03-26T03:53:12.413845+0000","last_undegraded":"2024-03-26T03:53:12.413845+0000","last_fullsized":"2024-03-26T03:53:12.413845+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.390111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.390111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392125+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":180,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.489383+0000","last_change":"2024-03-26T03:52:42.284044+0000","last_active":"2024-03-26T03:53:12.489383+0000","last_peered":"2024-03-26T03:53:12.489383+0000","last_clean":"2024-03-26T03:53:12.489383+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:53:12.489383+0000","last_undegraded":"2024-03-26T03:53:12.489383+0000","last_fullsized":"2024-03-26T03:53:12.489383+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.283958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.455074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":175,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403013+0000","last_change":"2024-03-26T03:51:25.157523+0000","last_active":"2024-03-26T03:53:12.403013+0000","last_peered":"2024-03-26T03:53:12.403013+0000","last_clean":"2024-03-26T03:53:12.403013+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:53:12.403013+0000","last_undegraded":"2024-03-26T03:53:12.403013+0000","last_fullsized":"2024-03-26T03:53:12.403013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.157414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.157414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380196+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":221,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403572+0000","last_change":"2024-03-26T03:51:31.354516+0000","last_active":"2024-03-26T03:53:12.403572+0000","last_peered":"2024-03-26T03:53:12.403572+0000","last_clean":"2024-03-26T03:53:12.403572+0000","last_became_active":"2024-03-26T03:50:45.596131+0000","last_became_peered":"2024-03-26T03:50:45.596131+0000","last_unstale":"2024-03-26T03:53:12.403572+0000","last_undegraded":"2024-03-26T03:53:12.403572+0000","last_fullsized":"2024-03-26T03:53:12.403572+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:31.354413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:31.354413+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380048+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"139'9858","reported_seq":10031,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.313776+0000","last_change":"2024-03-26T03:52:39.118362+0000","last_active":"2024-03-26T03:53:14.313776+0000","last_peered":"2024-03-26T03:53:14.313776+0000","last_clean":"2024-03-26T03:53:14.313776+0000","last_became_active":"2024-03-26T03:50:44.590889+0000","last_became_peered":"2024-03-26T03:50:44.590889+0000","last_unstale":"2024-03-26T03:53:14.313776+0000","last_undegraded":"2024-03-26T03:53:14.313776+0000","last_fullsized":"2024-03-26T03:53:14.313776+0000","mapping_epoch":95,"log_start":"117'4400","ondisk_log_start":"117'4400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7740","last_scrub_stamp":"2024-03-26T03:52:39.118187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.118187+0000","objects_scrubbed":6400,"log_size":5458,"log_dups_size":0,"ondisk_log_size":5458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.537245+0000","scrub_duration":2793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276955136,"num_objects":4226,"num_object_clones":0,"num_object_copies":12678,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4226,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9858,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":170,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.409335+0000","last_change":"2024-03-26T03:52:44.431735+0000","last_active":"2024-03-26T03:53:12.409335+0000","last_peered":"2024-03-26T03:53:12.409335+0000","last_clean":"2024-03-26T03:53:12.409335+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:53:12.409335+0000","last_undegraded":"2024-03-26T03:53:12.409335+0000","last_fullsized":"2024-03-26T03:53:12.409335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.431608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.431608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.078115+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":165,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.413790+0000","last_change":"2024-03-26T03:52:30.927185+0000","last_active":"2024-03-26T03:53:12.413790+0000","last_peered":"2024-03-26T03:53:12.413790+0000","last_clean":"2024-03-26T03:53:12.413790+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:53:12.413790+0000","last_undegraded":"2024-03-26T03:53:12.413790+0000","last_fullsized":"2024-03-26T03:53:12.413790+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:30.926620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.926620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:33.001795+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":105,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403110+0000","last_change":"2024-03-26T03:51:23.188460+0000","last_active":"2024-03-26T03:53:12.403110+0000","last_peered":"2024-03-26T03:53:12.403110+0000","last_clean":"2024-03-26T03:53:12.403110+0000","last_became_active":"2024-03-26T03:50:45.593583+0000","last_became_peered":"2024-03-26T03:50:45.593583+0000","last_unstale":"2024-03-26T03:53:12.403110+0000","last_undegraded":"2024-03-26T03:53:12.403110+0000","last_fullsized":"2024-03-26T03:53:12.403110+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.188316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.188316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":198,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403953+0000","last_change":"2024-03-26T03:51:32.392253+0000","last_active":"2024-03-26T03:53:12.403953+0000","last_peered":"2024-03-26T03:53:12.403953+0000","last_clean":"2024-03-26T03:53:12.403953+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:53:12.403953+0000","last_undegraded":"2024-03-26T03:53:12.403953+0000","last_fullsized":"2024-03-26T03:53:12.403953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:32.392163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.392163+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:13.380066+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"139'9907","reported_seq":10072,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:14.467775+0000","last_change":"2024-03-26T03:52:46.641388+0000","last_active":"2024-03-26T03:53:14.467775+0000","last_peered":"2024-03-26T03:53:14.467775+0000","last_clean":"2024-03-26T03:53:14.467775+0000","last_became_active":"2024-03-26T03:50:44.587505+0000","last_became_peered":"2024-03-26T03:50:44.587505+0000","last_unstale":"2024-03-26T03:53:14.467775+0000","last_undegraded":"2024-03-26T03:53:14.467775+0000","last_fullsized":"2024-03-26T03:53:14.467775+0000","mapping_epoch":95,"log_start":"120'5100","ondisk_log_start":"120'5100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'8232","last_scrub_stamp":"2024-03-26T03:52:46.640967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.640967+0000","objects_scrubbed":5921,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.138786+0000","scrub_duration":3071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272433152,"num_objects":4157,"num_object_clones":0,"num_object_copies":12471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9907,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":162,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.404212+0000","last_change":"2024-03-26T03:51:41.329652+0000","last_active":"2024-03-26T03:53:12.404212+0000","last_peered":"2024-03-26T03:53:12.404212+0000","last_clean":"2024-03-26T03:53:12.404212+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:53:12.404212+0000","last_undegraded":"2024-03-26T03:53:12.404212+0000","last_fullsized":"2024-03-26T03:53:12.404212+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.329573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.329573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":229,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.409265+0000","last_change":"2024-03-26T03:51:30.103885+0000","last_active":"2024-03-26T03:53:12.409265+0000","last_peered":"2024-03-26T03:53:12.409265+0000","last_clean":"2024-03-26T03:53:12.409265+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:53:12.409265+0000","last_undegraded":"2024-03-26T03:53:12.409265+0000","last_fullsized":"2024-03-26T03:53:12.409265+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:30.098793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.098793+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":167,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.410636+0000","last_change":"2024-03-26T03:51:32.274566+0000","last_active":"2024-03-26T03:53:12.410636+0000","last_peered":"2024-03-26T03:53:12.410636+0000","last_clean":"2024-03-26T03:53:12.410636+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:53:12.410636+0000","last_undegraded":"2024-03-26T03:53:12.410636+0000","last_fullsized":"2024-03-26T03:53:12.410636+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.274349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.274349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:15.392245+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":204,"reported_epoch":139,"state":"active+clean","last_fresh":"2024-03-26T03:53:12.403593+0000","last_change":"2024-03-26T03:52:31.734237+0000","last_active":"2024-03-26T03:53:12.403593+0000","last_peered":"2024-03-26T03:53:12.403593+0000","last_clean":"2024-03-26T03:53:12.403593+0000","last_became_active":"2024-03-26T03:50:45.598403+0000","last_became_peered":"2024-03-26T03:50:45.598403+0000","last_unstale":"2024-03-26T03:53:12.403593+0000","last_undegraded":"2024-03-26T03:53:12.403593+0000","last_fullsized":"2024-03-26T03:53:12.403593+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.994530+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4338352152,"num_objects":66199,"num_object_clones":0,"num_object_copies":198597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66199,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":158505,"num_write_kb":7190465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13023457280,"data_stored":13023445064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75505,"ondisk_log_size":75505,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":263,"num_read_kb":258,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070522,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1400228,"kb_used_data":1332956,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92344924,"statfs":{"total":95995035648,"available":94561202176,"internally_reserved":0,"allocated":1364946944,"data_stored":1364571332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2244676,"kb_used_data":2141304,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":91500476,"statfs":{"total":95995035648,"available":93696487424,"internally_reserved":0,"allocated":2192695296,"data_stored":2192290189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":105839045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2269308,"kb_used_data":2157616,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":91475844,"statfs":{"total":95995035648,"available":93671264256,"internally_reserved":0,"allocated":2209398784,"data_stored":2209002838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2251280,"kb_used_data":2152068,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":91493872,"statfs":{"total":95995035648,"available":93689724928,"internally_reserved":0,"allocated":2203717632,"data_stored":2203364098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,4,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2782132,"kb_used_data":2638504,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":90963020,"statfs":{"total":95995035648,"available":93146132480,"internally_reserved":0,"allocated":2701828096,"data_stored":2701439502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":147061191},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953545,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1417000,"kb_used_data":1349724,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":92328152,"statfs":{"total":95995035648,"available":94544027648,"internally_reserved":0,"allocated":1382117376,"data_stored":1381739536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659203,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1158232,"kb_used_data":1082636,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92586920,"statfs":{"total":95995035648,"available":94809006080,"internally_reserved":0,"allocated":1108619264,"data_stored":1108257846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2175668224,"data_stored":2175664152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2172063744,"data_stored":2172059672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2165768192,"data_stored":2165768192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2697728000,"data_stored":2697723928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1363017728,"data_stored":1363017728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1095892992,"data_stored":1095892992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1353318400,"data_stored":1353318400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:53:22.244 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T03:53:22.244 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:53:22.245 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T03:53:22.245 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T03:53:22.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:53:22.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:22.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:22.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:22.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:22.455 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T03:53:22.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:22.600 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:22.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:22.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:22.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:22.815 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T03:53:22.815 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T03:53:22.815 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[37501]: pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-26T03:53:22.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T03:53:22.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:22 smithi006 ceph-mon[29446]: pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-26T03:53:22.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:22 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T03:53:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:22 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T03:53:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:22 smithi086 ceph-mon[32965]: pgmap v183: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 993 op/s 2024-03-26T03:53:22.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:22.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:22.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:22.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:22.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:23.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:23.127 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:23.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:23.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:23.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:23.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:23.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:23.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:23.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:23.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:23.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:53:23.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:23.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:23.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:23.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:23.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T03:53:23.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T03:53:23.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[29446]: 7.1 scrub starts 2024-03-26T03:53:23.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T03:53:23.770 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T03:53:23.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T03:53:23.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[37501]: 7.1 scrub starts 2024-03-26T03:53:23.771 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:23 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T03:53:23.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:23.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:23 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T03:53:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:23 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T03:53:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:23 smithi086 ceph-mon[32965]: 7.1 scrub starts 2024-03-26T03:53:23.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:23 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T03:53:23.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:23.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:23.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:24.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:24.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.084 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:24.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:24.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:24.228 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:24.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:24.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:24.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:24.451 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:24.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:24.503 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:24.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:24.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:24.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:24.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:24.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:24.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:24.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:24.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: pgmap v184: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: 7.1 scrub ok 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T03:53:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:24.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:24.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:24.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:24.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: pgmap v184: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: 7.1 scrub ok 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: pgmap v184: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 3.7 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 943 B/s rd, 0 B/s wr, 915 op/s 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: 7.1 scrub ok 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T03:53:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T03:53:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:24.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:25.022 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.067 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:25.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:25.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.216 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:25.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:25.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:25.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:25.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:25.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:25.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:25.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:25.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T03:53:25.569 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T03:53:25.569 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:25.569 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T03:53:25.570 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:25 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T03:53:25.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:25.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:25.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:25.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:25.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:25.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:25.821 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:25 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T03:53:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:25 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T03:53:25.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:25.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:25.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:26.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:26.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:26.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:26.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:26.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:26.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:26.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:26.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:26.490 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:53:26.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:26.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:26.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:26.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:26.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: pgmap v185: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T03:53:26.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: pgmap v185: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T03:53:26.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T03:53:26.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T03:53:26.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:26 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: pgmap v185: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T03:53:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:26 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:26.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:26.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:26.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:27.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:27.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:27.139 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:27.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:27.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:27.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:27.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:27.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:27.527 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:27.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:27.677 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:27.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[37501]: osdmap e140: 8 total, 8 up, 7 in 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T03:53:27.732 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T03:53:27.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:53:27.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:53:27.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:27.733 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:27 smithi006 ceph-mon[29446]: osdmap e140: 8 total, 8 up, 7 in 2024-03-26T03:53:27.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:27.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:27.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T03:53:27.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T03:53:27.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:53:27.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:53:27.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3225512050' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:27.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:27 smithi086 ceph-mon[32965]: osdmap e140: 8 total, 8 up, 7 in 2024-03-26T03:53:27.847 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:53:27.847 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:53:27.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:27.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:27.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:28.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:28.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:28.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:28.202 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:28.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:28.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:28.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:28.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:28.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:28.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:28.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:28.633 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[37501]: pgmap v187: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-03-26T03:53:28.633 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[37501]: 7.2 scrub starts 2024-03-26T03:53:28.633 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[37501]: 4.15 deep-scrub starts 2024-03-26T03:53:28.634 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[37501]: 4.15 deep-scrub ok 2024-03-26T03:53:28.634 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[29446]: pgmap v187: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-03-26T03:53:28.634 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[29446]: 7.2 scrub starts 2024-03-26T03:53:28.634 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[29446]: 4.15 deep-scrub starts 2024-03-26T03:53:28.634 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:28 smithi006 ceph-mon[29446]: 4.15 deep-scrub ok 2024-03-26T03:53:28.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:28.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:28.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:28.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:28 smithi086 ceph-mon[32965]: pgmap v187: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 994 op/s 2024-03-26T03:53:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:28 smithi086 ceph-mon[32965]: 7.2 scrub starts 2024-03-26T03:53:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:28 smithi086 ceph-mon[32965]: 4.15 deep-scrub starts 2024-03-26T03:53:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:28 smithi086 ceph-mon[32965]: 4.15 deep-scrub ok 2024-03-26T03:53:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:28.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:28.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:28.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:28.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:29.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:29.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:29.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:29.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:29.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:29.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:29.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:29.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:53:29.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T03:53:29.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T03:53:29.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[37501]: 6.d deep-scrub starts 2024-03-26T03:53:29.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[37501]: 6.d deep-scrub ok 2024-03-26T03:53:29.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[37501]: 7.2 scrub ok 2024-03-26T03:53:29.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T03:53:29.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T03:53:29.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[29446]: 6.d deep-scrub starts 2024-03-26T03:53:29.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[29446]: 6.d deep-scrub ok 2024-03-26T03:53:29.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:29 smithi006 ceph-mon[29446]: 7.2 scrub ok 2024-03-26T03:53:29.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:29.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:29.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:29.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:29.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:29.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:29.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:29 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T03:53:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:29 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T03:53:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:29 smithi086 ceph-mon[32965]: 6.d deep-scrub starts 2024-03-26T03:53:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:29 smithi086 ceph-mon[32965]: 6.d deep-scrub ok 2024-03-26T03:53:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:29 smithi086 ceph-mon[32965]: 7.2 scrub ok 2024-03-26T03:53:29.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:29.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:29.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:30.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:30.097 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:53:30.146 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:53:30.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:30.197 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:53:30.222 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:30.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:30.236 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:53:30.278 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:53:30.323 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:53:30.323 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:30.365 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:53:30.367 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:30.405 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:53:30.412 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:53:30.412 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:53:30.412 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:53:30.413 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:53:30.413 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:53:30.413 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:53:30.413 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:53:30.413 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:53:30.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:30.496 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:30.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:53:30.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:30.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:30.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:30.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:30.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T03:53:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T03:53:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: pgmap v188: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.16 deep-scrub starts 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.16 deep-scrub ok 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: 7.d scrub starts 2024-03-26T03:53:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:30 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2829008177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: pgmap v188: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.16 deep-scrub starts 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.16 deep-scrub ok 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: 7.d scrub starts 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2829008177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:30.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: pgmap v188: 148 pgs: 148 active+clean; 3.4 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 990 op/s 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.16 deep-scrub starts 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.16 deep-scrub ok 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: 7.d scrub starts 2024-03-26T03:53:30.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:30 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2829008177' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:30.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:30.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:30.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:31.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:31.093 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:31.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:31.235 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:31.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:31.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:31.387 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:31.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:31.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:31.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:31.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:31.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:31.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:31.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:31.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:31.771 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T03:53:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T03:53:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T03:53:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T03:53:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T03:53:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T03:53:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:31 smithi086 ceph-mon[32965]: 7.d scrub ok 2024-03-26T03:53:31.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T03:53:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[29446]: 7.d scrub ok 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T03:53:31.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:31 smithi006 ceph-mon[37501]: 7.d scrub ok 2024-03-26T03:53:31.921 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:31.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:31.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:32.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:32.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:32.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:32.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:32.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:32.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:32.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:32.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:32.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:32.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:32.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:32.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:32.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:32 smithi086 ceph-mon[32965]: pgmap v189: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T03:53:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:32 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:32 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:32 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T03:53:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:32 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T03:53:32.838 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:32.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:32.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[37501]: pgmap v189: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T03:53:32.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[29446]: pgmap v189: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 968 op/s 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T03:53:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:32 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T03:53:32.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:32.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:32.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:32.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:32.987 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:33.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:33.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:53:33.077 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:53:33.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:33.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:33.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:53:33.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:33.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:33.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:33.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:33.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:33.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:33.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:33.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:33.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:33.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 4.10 deep-scrub starts 2024-03-26T03:53:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 4.10 deep-scrub ok 2024-03-26T03:53:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T03:53:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T03:53:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T03:53:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T03:53:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: osdmap e141: 8 total, 8 up, 7 in 2024-03-26T03:53:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:33 smithi086 ceph-mon[32965]: 7.3 scrub starts 2024-03-26T03:53:33.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:33.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:33.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 4.10 deep-scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 4.10 deep-scrub ok 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: osdmap e141: 8 total, 8 up, 7 in 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[37501]: 7.3 scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 4.10 deep-scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 4.10 deep-scrub ok 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T03:53:33.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T03:53:33.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T03:53:33.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: osdmap e141: 8 total, 8 up, 7 in 2024-03-26T03:53:33.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:33 smithi006 ceph-mon[29446]: 7.3 scrub starts 2024-03-26T03:53:33.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:34.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:34.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:34.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:34.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:34.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:34.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:34.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:34.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:34.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:34.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:34.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:34.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.697 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:34.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:34.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:34.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:34.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:34 smithi086 ceph-mon[32965]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:34 smithi086 ceph-mon[32965]: pgmap v191: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-03-26T03:53:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:34 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T03:53:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:34 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T03:53:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[37501]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[37501]: pgmap v191: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[29446]: from='client.24769 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[29446]: pgmap v191: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 585 op/s 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T03:53:34.888 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:34 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T03:53:34.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:34.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:34.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:35.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:35.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:35.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:35.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:35.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:35.372 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:35.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:35.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:35.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:35.641 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:35.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:35.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:35.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:35.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 7.3 scrub ok 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T03:53:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 7.3 scrub ok 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T03:53:35.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:35 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T03:53:35.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:35.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:36.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:36.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:36.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T03:53:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 7.3 scrub ok 2024-03-26T03:53:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T03:53:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T03:53:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T03:53:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:35 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T03:53:36.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:36.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:36.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:36.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:36.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:36.371 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":190,"stamp":"2024-03-26T03:53:32.562260+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3380812224,"num_objects":51809,"num_object_clones":0,"num_object_copies":155427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51809,"num_whiteouts":0,"num_read":1498,"num_read_kb":1312,"num_write":173932,"num_write_kb":7192926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2362855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76453,"ondisk_log_size":76453,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":10787368,"kb_used_data":10016108,"kb_used_omap":10,"kb_used_meta":770677,"kb_avail":645428696,"statfs":{"total":671965249536,"available":660918984704,"internally_reserved":0,"allocated":10256494592,"data_stored":10253798496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":789173354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-634519552,"num_objects":-9682,"num_object_clones":0,"num_object_copies":-29046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9682,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9682,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.310817"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481348+0000","last_change":"2024-03-26T03:50:34.035051+0000","last_active":"2024-03-26T03:53:26.481348+0000","last_peered":"2024-03-26T03:53:26.481348+0000","last_clean":"2024-03-26T03:53:26.481348+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:53:26.481348+0000","last_undegraded":"2024-03-26T03:53:26.481348+0000","last_fullsized":"2024-03-26T03:53:26.481348+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:34.034948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:50:34.034948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":167,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482202+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:53:26.482202+0000","last_peered":"2024-03-26T03:53:26.482202+0000","last_clean":"2024-03-26T03:53:26.482202+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:53:26.482202+0000","last_undegraded":"2024-03-26T03:53:26.482202+0000","last_fullsized":"2024-03-26T03:53:26.482202+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.054186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.488652+0000","last_change":"2024-03-26T03:51:39.331101+0000","last_active":"2024-03-26T03:53:26.488652+0000","last_peered":"2024-03-26T03:53:26.488652+0000","last_clean":"2024-03-26T03:53:26.488652+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:53:26.488652+0000","last_undegraded":"2024-03-26T03:53:26.488652+0000","last_fullsized":"2024-03-26T03:53:26.488652+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.330345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":218,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482591+0000","last_change":"2024-03-26T03:52:33.804342+0000","last_active":"2024-03-26T03:53:26.482591+0000","last_peered":"2024-03-26T03:53:26.482591+0000","last_clean":"2024-03-26T03:53:26.482591+0000","last_became_active":"2024-03-26T03:50:45.598168+0000","last_became_peered":"2024-03-26T03:50:45.598168+0000","last_unstale":"2024-03-26T03:53:26.482591+0000","last_undegraded":"2024-03-26T03:53:26.482591+0000","last_fullsized":"2024-03-26T03:53:26.482591+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.024936+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":170,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.494802+0000","last_change":"2024-03-26T03:52:43.442740+0000","last_active":"2024-03-26T03:53:26.494802+0000","last_peered":"2024-03-26T03:53:26.494802+0000","last_clean":"2024-03-26T03:53:26.494802+0000","last_became_active":"2024-03-26T03:50:45.597941+0000","last_became_peered":"2024-03-26T03:50:45.597941+0000","last_unstale":"2024-03-26T03:53:26.494802+0000","last_undegraded":"2024-03-26T03:53:26.494802+0000","last_fullsized":"2024-03-26T03:53:26.494802+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.003941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":174,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481344+0000","last_change":"2024-03-26T03:53:13.396637+0000","last_active":"2024-03-26T03:53:26.481344+0000","last_peered":"2024-03-26T03:53:26.481344+0000","last_clean":"2024-03-26T03:53:26.481344+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:53:26.481344+0000","last_undegraded":"2024-03-26T03:53:26.481344+0000","last_fullsized":"2024-03-26T03:53:26.481344+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.396551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.396551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:31.835565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":175,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.492190+0000","last_change":"2024-03-26T03:51:21.399264+0000","last_active":"2024-03-26T03:53:26.492190+0000","last_peered":"2024-03-26T03:53:26.492190+0000","last_clean":"2024-03-26T03:53:26.492190+0000","last_became_active":"2024-03-26T03:50:45.581989+0000","last_became_peered":"2024-03-26T03:50:45.581989+0000","last_unstale":"2024-03-26T03:53:26.492190+0000","last_undegraded":"2024-03-26T03:53:26.492190+0000","last_fullsized":"2024-03-26T03:53:26.492190+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.398535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.398535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":245,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.494569+0000","last_change":"2024-03-26T03:53:26.317218+0000","last_active":"2024-03-26T03:53:26.494569+0000","last_peered":"2024-03-26T03:53:26.494569+0000","last_clean":"2024-03-26T03:53:26.494569+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:53:26.494569+0000","last_undegraded":"2024-03-26T03:53:26.494569+0000","last_fullsized":"2024-03-26T03:53:26.494569+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:53.761677+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":164,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486625+0000","last_change":"2024-03-26T03:53:22.532263+0000","last_active":"2024-03-26T03:53:26.486625+0000","last_peered":"2024-03-26T03:53:26.486625+0000","last_clean":"2024-03-26T03:53:26.486625+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:53:26.486625+0000","last_undegraded":"2024-03-26T03:53:26.486625+0000","last_fullsized":"2024-03-26T03:53:26.486625+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:22.532151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:22.532151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:28.487704+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498691+0000","last_change":"2024-03-26T03:53:24.585622+0000","last_active":"2024-03-26T03:53:26.498691+0000","last_peered":"2024-03-26T03:53:26.498691+0000","last_clean":"2024-03-26T03:53:26.498691+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:53:26.498691+0000","last_undegraded":"2024-03-26T03:53:26.498691+0000","last_fullsized":"2024-03-26T03:53:26.498691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.585508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.585508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:28.460358+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":190,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498978+0000","last_change":"2024-03-26T03:53:18.575050+0000","last_active":"2024-03-26T03:53:26.498978+0000","last_peered":"2024-03-26T03:53:26.498978+0000","last_clean":"2024-03-26T03:53:26.498978+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:53:26.498978+0000","last_undegraded":"2024-03-26T03:53:26.498978+0000","last_fullsized":"2024-03-26T03:53:26.498978+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:34.696377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":213,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481492+0000","last_change":"2024-03-26T03:51:27.245595+0000","last_active":"2024-03-26T03:53:26.481492+0000","last_peered":"2024-03-26T03:53:26.481492+0000","last_clean":"2024-03-26T03:53:26.481492+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:53:26.481492+0000","last_undegraded":"2024-03-26T03:53:26.481492+0000","last_fullsized":"2024-03-26T03:53:26.481492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:27.245475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.245475+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":164,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498482+0000","last_change":"2024-03-26T03:52:26.427599+0000","last_active":"2024-03-26T03:53:26.498482+0000","last_peered":"2024-03-26T03:53:26.498482+0000","last_clean":"2024-03-26T03:53:26.498482+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:53:26.498482+0000","last_undegraded":"2024-03-26T03:53:26.498482+0000","last_fullsized":"2024-03-26T03:53:26.498482+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.427501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.427501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.945072+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490297+0000","last_change":"2024-03-26T03:52:52.666181+0000","last_active":"2024-03-26T03:53:26.490297+0000","last_peered":"2024-03-26T03:53:26.490297+0000","last_clean":"2024-03-26T03:53:26.490297+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:53:26.490297+0000","last_undegraded":"2024-03-26T03:53:26.490297+0000","last_fullsized":"2024-03-26T03:53:26.490297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.666107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.666107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.715049+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498793+0000","last_change":"2024-03-26T03:52:28.376930+0000","last_active":"2024-03-26T03:53:26.498793+0000","last_peered":"2024-03-26T03:53:26.498793+0000","last_clean":"2024-03-26T03:53:26.498793+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:53:26.498793+0000","last_undegraded":"2024-03-26T03:53:26.498793+0000","last_fullsized":"2024-03-26T03:53:26.498793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:28.376798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:28.376798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:53.233172+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":197,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481581+0000","last_change":"2024-03-26T03:52:38.915695+0000","last_active":"2024-03-26T03:53:26.481581+0000","last_peered":"2024-03-26T03:53:26.481581+0000","last_clean":"2024-03-26T03:53:26.481581+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:53:26.481581+0000","last_undegraded":"2024-03-26T03:53:26.481581+0000","last_fullsized":"2024-03-26T03:53:26.481581+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.912180+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":171,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481986+0000","last_change":"2024-03-26T03:52:51.658215+0000","last_active":"2024-03-26T03:53:26.481986+0000","last_peered":"2024-03-26T03:53:26.481986+0000","last_clean":"2024-03-26T03:53:26.481986+0000","last_became_active":"2024-03-26T03:50:45.588396+0000","last_became_peered":"2024-03-26T03:50:45.588396+0000","last_unstale":"2024-03-26T03:53:26.481986+0000","last_undegraded":"2024-03-26T03:53:26.481986+0000","last_fullsized":"2024-03-26T03:53:26.481986+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:17.128768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":159,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481971+0000","last_change":"2024-03-26T03:51:15.157751+0000","last_active":"2024-03-26T03:53:26.481971+0000","last_peered":"2024-03-26T03:53:26.481971+0000","last_clean":"2024-03-26T03:53:26.481971+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:53:26.481971+0000","last_undegraded":"2024-03-26T03:53:26.481971+0000","last_fullsized":"2024-03-26T03:53:26.481971+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:15.157477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.157477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":184,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486882+0000","last_change":"2024-03-26T03:53:13.631462+0000","last_active":"2024-03-26T03:53:26.486882+0000","last_peered":"2024-03-26T03:53:26.486882+0000","last_clean":"2024-03-26T03:53:26.486882+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:53:26.486882+0000","last_undegraded":"2024-03-26T03:53:26.486882+0000","last_fullsized":"2024-03-26T03:53:26.486882+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.631351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.631351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:41.275139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":205,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.495396+0000","last_change":"2024-03-26T03:52:45.404305+0000","last_active":"2024-03-26T03:53:26.495396+0000","last_peered":"2024-03-26T03:53:26.495396+0000","last_clean":"2024-03-26T03:53:26.495396+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:53:26.495396+0000","last_undegraded":"2024-03-26T03:53:26.495396+0000","last_fullsized":"2024-03-26T03:53:26.495396+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:45.403401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.403401+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.513250+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.497172+0000","last_change":"2024-03-26T03:51:38.222500+0000","last_active":"2024-03-26T03:53:26.497172+0000","last_peered":"2024-03-26T03:53:26.497172+0000","last_clean":"2024-03-26T03:53:26.497172+0000","last_became_active":"2024-03-26T03:50:45.591442+0000","last_became_peered":"2024-03-26T03:50:45.591442+0000","last_unstale":"2024-03-26T03:53:26.497172+0000","last_undegraded":"2024-03-26T03:53:26.497172+0000","last_fullsized":"2024-03-26T03:53:26.497172+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.222375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481672+0000","last_change":"2024-03-26T03:52:49.690105+0000","last_active":"2024-03-26T03:53:26.481672+0000","last_peered":"2024-03-26T03:53:26.481672+0000","last_clean":"2024-03-26T03:53:26.481672+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:53:26.481672+0000","last_undegraded":"2024-03-26T03:53:26.481672+0000","last_fullsized":"2024-03-26T03:53:26.481672+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:49.690025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.690025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.171459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481561+0000","last_change":"2024-03-26T03:51:13.376075+0000","last_active":"2024-03-26T03:53:26.481561+0000","last_peered":"2024-03-26T03:53:26.481561+0000","last_clean":"2024-03-26T03:53:26.481561+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:53:26.481561+0000","last_undegraded":"2024-03-26T03:53:26.481561+0000","last_fullsized":"2024-03-26T03:53:26.481561+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:13.375964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:13.375964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":207,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.492334+0000","last_change":"2024-03-26T03:52:35.473032+0000","last_active":"2024-03-26T03:53:26.492334+0000","last_peered":"2024-03-26T03:53:26.492334+0000","last_clean":"2024-03-26T03:53:26.492334+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:53:26.492334+0000","last_undegraded":"2024-03-26T03:53:26.492334+0000","last_fullsized":"2024-03-26T03:53:26.492334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.269829+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":166,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490753+0000","last_change":"2024-03-26T03:52:23.708646+0000","last_active":"2024-03-26T03:53:26.490753+0000","last_peered":"2024-03-26T03:53:26.490753+0000","last_clean":"2024-03-26T03:53:26.490753+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:53:26.490753+0000","last_undegraded":"2024-03-26T03:53:26.490753+0000","last_fullsized":"2024-03-26T03:53:26.490753+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:52:23.708510+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.708510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":175,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482217+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:53:26.482217+0000","last_peered":"2024-03-26T03:53:26.482217+0000","last_clean":"2024-03-26T03:53:26.482217+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:53:26.482217+0000","last_undegraded":"2024-03-26T03:53:26.482217+0000","last_fullsized":"2024-03-26T03:53:26.482217+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.226521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481649+0000","last_change":"2024-03-26T03:53:13.785620+0000","last_active":"2024-03-26T03:53:26.481649+0000","last_peered":"2024-03-26T03:53:26.481649+0000","last_clean":"2024-03-26T03:53:26.481649+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:53:26.481649+0000","last_undegraded":"2024-03-26T03:53:26.481649+0000","last_fullsized":"2024-03-26T03:53:26.481649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:13.785265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.785265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:16.531268+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":200,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493911+0000","last_change":"2024-03-26T03:52:42.591753+0000","last_active":"2024-03-26T03:53:26.493911+0000","last_peered":"2024-03-26T03:53:26.493911+0000","last_clean":"2024-03-26T03:53:26.493911+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:53:26.493911+0000","last_undegraded":"2024-03-26T03:53:26.493911+0000","last_fullsized":"2024-03-26T03:53:26.493911+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:52:42.591503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.591503+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.339508+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481693+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:53:26.481693+0000","last_peered":"2024-03-26T03:53:26.481693+0000","last_clean":"2024-03-26T03:53:26.481693+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:53:26.481693+0000","last_undegraded":"2024-03-26T03:53:26.481693+0000","last_fullsized":"2024-03-26T03:53:26.481693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:07.860041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.497759+0000","last_change":"2024-03-26T03:51:33.548319+0000","last_active":"2024-03-26T03:53:26.497759+0000","last_peered":"2024-03-26T03:53:26.497759+0000","last_clean":"2024-03-26T03:53:26.497759+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:53:26.497759+0000","last_undegraded":"2024-03-26T03:53:26.497759+0000","last_fullsized":"2024-03-26T03:53:26.497759+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.548192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.518079+0000","last_change":"2024-03-26T03:51:35.260411+0000","last_active":"2024-03-26T03:53:26.518079+0000","last_peered":"2024-03-26T03:53:26.518079+0000","last_clean":"2024-03-26T03:53:26.518079+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:53:26.518079+0000","last_undegraded":"2024-03-26T03:53:26.518079+0000","last_fullsized":"2024-03-26T03:53:26.518079+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.258167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.258167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":181,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499225+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:53:26.499225+0000","last_peered":"2024-03-26T03:53:26.499225+0000","last_clean":"2024-03-26T03:53:26.499225+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:53:26.499225+0000","last_undegraded":"2024-03-26T03:53:26.499225+0000","last_fullsized":"2024-03-26T03:53:26.499225+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.838957+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490230+0000","last_change":"2024-03-26T03:52:34.605587+0000","last_active":"2024-03-26T03:53:26.490230+0000","last_peered":"2024-03-26T03:53:26.490230+0000","last_clean":"2024-03-26T03:53:26.490230+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:53:26.490230+0000","last_undegraded":"2024-03-26T03:53:26.490230+0000","last_fullsized":"2024-03-26T03:53:26.490230+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:34.605509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:34.605509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.539945+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":107,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481514+0000","last_change":"2024-03-26T03:51:39.355828+0000","last_active":"2024-03-26T03:53:26.481514+0000","last_peered":"2024-03-26T03:53:26.481514+0000","last_clean":"2024-03-26T03:53:26.481514+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:53:26.481514+0000","last_undegraded":"2024-03-26T03:53:26.481514+0000","last_fullsized":"2024-03-26T03:53:26.481514+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.355728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.355728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.488848+0000","last_change":"2024-03-26T03:52:23.578870+0000","last_active":"2024-03-26T03:53:26.488848+0000","last_peered":"2024-03-26T03:53:26.488848+0000","last_clean":"2024-03-26T03:53:26.488848+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:53:26.488848+0000","last_undegraded":"2024-03-26T03:53:26.488848+0000","last_fullsized":"2024-03-26T03:53:26.488848+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.989992+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.509486+0000","last_change":"2024-03-26T03:50:45.606641+0000","last_active":"2024-03-26T03:53:26.509486+0000","last_peered":"2024-03-26T03:53:26.509486+0000","last_clean":"2024-03-26T03:53:26.509486+0000","last_became_active":"2024-03-26T03:50:45.587808+0000","last_became_peered":"2024-03-26T03:50:45.587808+0000","last_unstale":"2024-03-26T03:53:26.509486+0000","last_undegraded":"2024-03-26T03:53:26.509486+0000","last_fullsized":"2024-03-26T03:53:26.509486+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-03-26T03:50:35.236237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:50:35.236237+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":171,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496128+0000","last_change":"2024-03-26T03:53:17.352408+0000","last_active":"2024-03-26T03:53:26.496128+0000","last_peered":"2024-03-26T03:53:26.496128+0000","last_clean":"2024-03-26T03:53:26.496128+0000","last_became_active":"2024-03-26T03:50:45.589899+0000","last_became_peered":"2024-03-26T03:50:45.589899+0000","last_unstale":"2024-03-26T03:53:26.496128+0000","last_undegraded":"2024-03-26T03:53:26.496128+0000","last_fullsized":"2024-03-26T03:53:26.496128+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.352186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.352186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:30.380486+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":175,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490422+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:53:26.490422+0000","last_peered":"2024-03-26T03:53:26.490422+0000","last_clean":"2024-03-26T03:53:26.490422+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:53:26.490422+0000","last_undegraded":"2024-03-26T03:53:26.490422+0000","last_fullsized":"2024-03-26T03:53:26.490422+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.913379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":184,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499329+0000","last_change":"2024-03-26T03:52:39.291143+0000","last_active":"2024-03-26T03:53:26.499329+0000","last_peered":"2024-03-26T03:53:26.499329+0000","last_clean":"2024-03-26T03:53:26.499329+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:53:26.499329+0000","last_undegraded":"2024-03-26T03:53:26.499329+0000","last_fullsized":"2024-03-26T03:53:26.499329+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:39.291063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.291063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:55.327602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":230,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496640+0000","last_change":"2024-03-26T03:53:19.344881+0000","last_active":"2024-03-26T03:53:26.496640+0000","last_peered":"2024-03-26T03:53:26.496640+0000","last_clean":"2024-03-26T03:53:26.496640+0000","last_became_active":"2024-03-26T03:50:45.595599+0000","last_became_peered":"2024-03-26T03:50:45.595599+0000","last_unstale":"2024-03-26T03:53:26.496640+0000","last_undegraded":"2024-03-26T03:53:26.496640+0000","last_fullsized":"2024-03-26T03:53:26.496640+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:25.336032+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486247+0000","last_change":"2024-03-26T03:52:26.596430+0000","last_active":"2024-03-26T03:53:26.486247+0000","last_peered":"2024-03-26T03:53:26.486247+0000","last_clean":"2024-03-26T03:53:26.486247+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:53:26.486247+0000","last_undegraded":"2024-03-26T03:53:26.486247+0000","last_fullsized":"2024-03-26T03:53:26.486247+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.596311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.596311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:47.308150+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.494755+0000","last_change":"2024-03-26T03:52:40.410756+0000","last_active":"2024-03-26T03:53:26.494755+0000","last_peered":"2024-03-26T03:53:26.494755+0000","last_clean":"2024-03-26T03:53:26.494755+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:53:26.494755+0000","last_undegraded":"2024-03-26T03:53:26.494755+0000","last_fullsized":"2024-03-26T03:53:26.494755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.947379+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":184,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490617+0000","last_change":"2024-03-26T03:53:24.672838+0000","last_active":"2024-03-26T03:53:26.490617+0000","last_peered":"2024-03-26T03:53:26.490617+0000","last_clean":"2024-03-26T03:53:26.490617+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:53:26.490617+0000","last_undegraded":"2024-03-26T03:53:26.490617+0000","last_fullsized":"2024-03-26T03:53:26.490617+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.120373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":204,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482006+0000","last_change":"2024-03-26T03:51:29.988370+0000","last_active":"2024-03-26T03:53:26.482006+0000","last_peered":"2024-03-26T03:53:26.482006+0000","last_clean":"2024-03-26T03:53:26.482006+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:53:26.482006+0000","last_undegraded":"2024-03-26T03:53:26.482006+0000","last_fullsized":"2024-03-26T03:53:26.482006+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:51:29.988111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.988111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.494373+0000","last_change":"2024-03-26T03:52:41.432452+0000","last_active":"2024-03-26T03:53:26.494373+0000","last_peered":"2024-03-26T03:53:26.494373+0000","last_clean":"2024-03-26T03:53:26.494373+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:53:26.494373+0000","last_undegraded":"2024-03-26T03:53:26.494373+0000","last_fullsized":"2024-03-26T03:53:26.494373+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:41.432332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:41.432332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.168460+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481733+0000","last_change":"2024-03-26T03:52:40.835637+0000","last_active":"2024-03-26T03:53:26.481733+0000","last_peered":"2024-03-26T03:53:26.481733+0000","last_clean":"2024-03-26T03:53:26.481733+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:53:26.481733+0000","last_undegraded":"2024-03-26T03:53:26.481733+0000","last_fullsized":"2024-03-26T03:53:26.481733+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.905532+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481597+0000","last_change":"2024-03-26T03:51:30.319510+0000","last_active":"2024-03-26T03:53:26.481597+0000","last_peered":"2024-03-26T03:53:26.481597+0000","last_clean":"2024-03-26T03:53:26.481597+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:53:26.481597+0000","last_undegraded":"2024-03-26T03:53:26.481597+0000","last_fullsized":"2024-03-26T03:53:26.481597+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:30.319423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:30.319423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":202,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481829+0000","last_change":"2024-03-26T03:51:17.380156+0000","last_active":"2024-03-26T03:53:26.481829+0000","last_peered":"2024-03-26T03:53:26.481829+0000","last_clean":"2024-03-26T03:53:26.481829+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:53:26.481829+0000","last_undegraded":"2024-03-26T03:53:26.481829+0000","last_fullsized":"2024-03-26T03:53:26.481829+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:51:17.380041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.380041+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":156,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493516+0000","last_change":"2024-03-26T03:51:19.175698+0000","last_active":"2024-03-26T03:53:26.493516+0000","last_peered":"2024-03-26T03:53:26.493516+0000","last_clean":"2024-03-26T03:53:26.493516+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:53:26.493516+0000","last_undegraded":"2024-03-26T03:53:26.493516+0000","last_fullsized":"2024-03-26T03:53:26.493516+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.175455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.175455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":242,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499096+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:53:26.499096+0000","last_peered":"2024-03-26T03:53:26.499096+0000","last_clean":"2024-03-26T03:53:26.499096+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:53:26.499096+0000","last_undegraded":"2024-03-26T03:53:26.499096+0000","last_fullsized":"2024-03-26T03:53:26.499096+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.768552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":185,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490309+0000","last_change":"2024-03-26T03:53:20.739662+0000","last_active":"2024-03-26T03:53:26.490309+0000","last_peered":"2024-03-26T03:53:26.490309+0000","last_clean":"2024-03-26T03:53:26.490309+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:53:26.490309+0000","last_undegraded":"2024-03-26T03:53:26.490309+0000","last_fullsized":"2024-03-26T03:53:26.490309+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.739547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.739547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:25.034838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":203,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.492864+0000","last_change":"2024-03-26T03:51:42.190694+0000","last_active":"2024-03-26T03:53:26.492864+0000","last_peered":"2024-03-26T03:53:26.492864+0000","last_clean":"2024-03-26T03:53:26.492864+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:53:26.492864+0000","last_undegraded":"2024-03-26T03:53:26.492864+0000","last_fullsized":"2024-03-26T03:53:26.492864+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:42.190557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:42.190557+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":164,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482203+0000","last_change":"2024-03-26T03:51:34.417542+0000","last_active":"2024-03-26T03:53:26.482203+0000","last_peered":"2024-03-26T03:53:26.482203+0000","last_clean":"2024-03-26T03:53:26.482203+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:53:26.482203+0000","last_undegraded":"2024-03-26T03:53:26.482203+0000","last_fullsized":"2024-03-26T03:53:26.482203+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:34.417463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.417463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":226,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493222+0000","last_change":"2024-03-26T03:51:34.258348+0000","last_active":"2024-03-26T03:53:26.493222+0000","last_peered":"2024-03-26T03:53:26.493222+0000","last_clean":"2024-03-26T03:53:26.493222+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:53:26.493222+0000","last_undegraded":"2024-03-26T03:53:26.493222+0000","last_fullsized":"2024-03-26T03:53:26.493222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":191,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.487164+0000","last_change":"2024-03-26T03:53:12.601302+0000","last_active":"2024-03-26T03:53:26.487164+0000","last_peered":"2024-03-26T03:53:26.487164+0000","last_clean":"2024-03-26T03:53:26.487164+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:53:26.487164+0000","last_undegraded":"2024-03-26T03:53:26.487164+0000","last_fullsized":"2024-03-26T03:53:26.487164+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:53:12.600988+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:12.600988+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:29.368784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":213,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:27.547595+0000","last_change":"2024-03-26T03:53:27.547640+0000","last_active":"2024-03-26T03:53:27.547595+0000","last_peered":"2024-03-26T03:53:27.547595+0000","last_clean":"2024-03-26T03:53:27.547595+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:53:27.547595+0000","last_undegraded":"2024-03-26T03:53:27.547595+0000","last_fullsized":"2024-03-26T03:53:27.547595+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.015778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":174,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.495762+0000","last_change":"2024-03-26T03:53:18.332085+0000","last_active":"2024-03-26T03:53:26.495762+0000","last_peered":"2024-03-26T03:53:26.495762+0000","last_clean":"2024-03-26T03:53:26.495762+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:53:26.495762+0000","last_undegraded":"2024-03-26T03:53:26.495762+0000","last_fullsized":"2024-03-26T03:53:26.495762+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:53:18.331943+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.331943+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:26.743313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.497017+0000","last_change":"2024-03-26T03:53:23.353045+0000","last_active":"2024-03-26T03:53:26.497017+0000","last_peered":"2024-03-26T03:53:26.497017+0000","last_clean":"2024-03-26T03:53:26.497017+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:53:26.497017+0000","last_undegraded":"2024-03-26T03:53:26.497017+0000","last_fullsized":"2024-03-26T03:53:26.497017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:23.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:26.709176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490214+0000","last_change":"2024-03-26T03:51:19.966491+0000","last_active":"2024-03-26T03:53:26.490214+0000","last_peered":"2024-03-26T03:53:26.490214+0000","last_clean":"2024-03-26T03:53:26.490214+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:53:26.490214+0000","last_undegraded":"2024-03-26T03:53:26.490214+0000","last_fullsized":"2024-03-26T03:53:26.490214+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.966387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.966387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":212,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.497416+0000","last_change":"2024-03-26T03:52:37.650277+0000","last_active":"2024-03-26T03:53:26.497416+0000","last_peered":"2024-03-26T03:53:26.497416+0000","last_clean":"2024-03-26T03:53:26.497416+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:53:26.497416+0000","last_undegraded":"2024-03-26T03:53:26.497416+0000","last_fullsized":"2024-03-26T03:53:26.497416+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.102642+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":170,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490538+0000","last_change":"2024-03-26T03:53:17.657919+0000","last_active":"2024-03-26T03:53:26.490538+0000","last_peered":"2024-03-26T03:53:26.490538+0000","last_clean":"2024-03-26T03:53:26.490538+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:53:26.490538+0000","last_undegraded":"2024-03-26T03:53:26.490538+0000","last_fullsized":"2024-03-26T03:53:26.490538+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.376109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499360+0000","last_change":"2024-03-26T03:51:38.612538+0000","last_active":"2024-03-26T03:53:26.499360+0000","last_peered":"2024-03-26T03:53:26.499360+0000","last_clean":"2024-03-26T03:53:26.499360+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:53:26.499360+0000","last_undegraded":"2024-03-26T03:53:26.499360+0000","last_fullsized":"2024-03-26T03:53:26.499360+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.612390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.612390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":107,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486495+0000","last_change":"2024-03-26T03:51:38.327055+0000","last_active":"2024-03-26T03:53:26.486495+0000","last_peered":"2024-03-26T03:53:26.486495+0000","last_clean":"2024-03-26T03:53:26.486495+0000","last_became_active":"2024-03-26T03:50:45.593133+0000","last_became_peered":"2024-03-26T03:50:45.593133+0000","last_unstale":"2024-03-26T03:53:26.486495+0000","last_undegraded":"2024-03-26T03:53:26.486495+0000","last_fullsized":"2024-03-26T03:53:26.486495+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.326923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.326923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":210,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.497259+0000","last_change":"2024-03-26T03:52:33.466952+0000","last_active":"2024-03-26T03:53:26.497259+0000","last_peered":"2024-03-26T03:53:26.497259+0000","last_clean":"2024-03-26T03:53:26.497259+0000","last_became_active":"2024-03-26T03:50:24.531528+0000","last_became_peered":"2024-03-26T03:50:24.531528+0000","last_unstale":"2024-03-26T03:53:26.497259+0000","last_undegraded":"2024-03-26T03:53:26.497259+0000","last_fullsized":"2024-03-26T03:53:26.497259+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:48.107634+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"140'10548","reported_seq":10716,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.336717+0000","last_change":"2024-03-26T03:52:44.145650+0000","last_active":"2024-03-26T03:53:29.336717+0000","last_peered":"2024-03-26T03:53:29.336717+0000","last_clean":"2024-03-26T03:53:29.336717+0000","last_became_active":"2024-03-26T03:50:45.595232+0000","last_became_peered":"2024-03-26T03:50:45.595232+0000","last_unstale":"2024-03-26T03:53:29.336717+0000","last_undegraded":"2024-03-26T03:53:29.336717+0000","last_fullsized":"2024-03-26T03:53:29.336717+0000","mapping_epoch":96,"log_start":"126'6300","ondisk_log_start":"126'6300","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'7919","last_scrub_stamp":"2024-03-26T03:52:44.145396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.145396+0000","objects_scrubbed":5918,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.921446+0000","scrub_duration":2302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211812352,"num_objects":3232,"num_object_clones":0,"num_object_copies":9696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10548,"num_write_kb":440960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":179,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498863+0000","last_change":"2024-03-26T03:52:50.408368+0000","last_active":"2024-03-26T03:53:26.498863+0000","last_peered":"2024-03-26T03:53:26.498863+0000","last_clean":"2024-03-26T03:53:26.498863+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:53:26.498863+0000","last_undegraded":"2024-03-26T03:53:26.498863+0000","last_fullsized":"2024-03-26T03:53:26.498863+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.408254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.408254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.251022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":189,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490578+0000","last_change":"2024-03-26T03:52:32.622645+0000","last_active":"2024-03-26T03:53:26.490578+0000","last_peered":"2024-03-26T03:53:26.490578+0000","last_clean":"2024-03-26T03:53:26.490578+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:53:26.490578+0000","last_undegraded":"2024-03-26T03:53:26.490578+0000","last_fullsized":"2024-03-26T03:53:26.490578+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:52:32.622512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.622512+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.145922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":248,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498537+0000","last_change":"2024-03-26T03:52:22.438654+0000","last_active":"2024-03-26T03:53:26.498537+0000","last_peered":"2024-03-26T03:53:26.498537+0000","last_clean":"2024-03-26T03:53:26.498537+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:53:26.498537+0000","last_undegraded":"2024-03-26T03:53:26.498537+0000","last_fullsized":"2024-03-26T03:53:26.498537+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:52:22.438565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:22.438565+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.719143+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":253,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490458+0000","last_change":"2024-03-26T03:53:19.706639+0000","last_active":"2024-03-26T03:53:26.490458+0000","last_peered":"2024-03-26T03:53:26.490458+0000","last_clean":"2024-03-26T03:53:26.490458+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:53:26.490458+0000","last_undegraded":"2024-03-26T03:53:26.490458+0000","last_fullsized":"2024-03-26T03:53:26.490458+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:19.706557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.706557+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:24.796992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"140'10765","reported_seq":10866,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.478727+0000","last_change":"2024-03-26T03:50:44.592931+0000","last_active":"2024-03-26T03:53:29.478727+0000","last_peered":"2024-03-26T03:53:29.478727+0000","last_clean":"2024-03-26T03:53:29.478727+0000","last_became_active":"2024-03-26T03:50:44.592128+0000","last_became_peered":"2024-03-26T03:50:44.592128+0000","last_unstale":"2024-03-26T03:53:29.478727+0000","last_undegraded":"2024-03-26T03:53:29.478727+0000","last_fullsized":"2024-03-26T03:53:29.478727+0000","mapping_epoch":95,"log_start":"126'6500","ondisk_log_start":"126'6500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4265,"log_dups_size":0,"ondisk_log_size":4265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208732160,"num_objects":3185,"num_object_clones":0,"num_object_copies":9555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3185,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10765,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.494886+0000","last_change":"2024-03-26T03:51:23.414926+0000","last_active":"2024-03-26T03:53:26.494886+0000","last_peered":"2024-03-26T03:53:26.494886+0000","last_clean":"2024-03-26T03:53:26.494886+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:53:26.494886+0000","last_undegraded":"2024-03-26T03:53:26.494886+0000","last_fullsized":"2024-03-26T03:53:26.494886+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490797+0000","last_change":"2024-03-26T03:52:31.638104+0000","last_active":"2024-03-26T03:53:26.490797+0000","last_peered":"2024-03-26T03:53:26.490797+0000","last_clean":"2024-03-26T03:53:26.490797+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:53:26.490797+0000","last_undegraded":"2024-03-26T03:53:26.490797+0000","last_fullsized":"2024-03-26T03:53:26.490797+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.637987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.637987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:44.891585+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":220,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481956+0000","last_change":"2024-03-26T03:52:30.697090+0000","last_active":"2024-03-26T03:53:26.481956+0000","last_peered":"2024-03-26T03:53:26.481956+0000","last_clean":"2024-03-26T03:53:26.481956+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:53:26.481956+0000","last_undegraded":"2024-03-26T03:53:26.481956+0000","last_fullsized":"2024-03-26T03:53:26.481956+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.990440+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490767+0000","last_change":"2024-03-26T03:53:19.352440+0000","last_active":"2024-03-26T03:53:26.490767+0000","last_peered":"2024-03-26T03:53:26.490767+0000","last_clean":"2024-03-26T03:53:26.490767+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:53:26.490767+0000","last_undegraded":"2024-03-26T03:53:26.490767+0000","last_fullsized":"2024-03-26T03:53:26.490767+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:19.352319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.352319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:45.533994+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"140'11172","reported_seq":11348,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.486245+0000","last_change":"2024-03-26T03:52:36.221752+0000","last_active":"2024-03-26T03:53:29.486245+0000","last_peered":"2024-03-26T03:53:29.486245+0000","last_clean":"2024-03-26T03:53:29.486245+0000","last_became_active":"2024-03-26T03:50:44.585858+0000","last_became_peered":"2024-03-26T03:50:44.585858+0000","last_unstale":"2024-03-26T03:53:29.486245+0000","last_undegraded":"2024-03-26T03:53:29.486245+0000","last_fullsized":"2024-03-26T03:53:29.486245+0000","mapping_epoch":95,"log_start":"125'6400","ondisk_log_start":"125'6400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7838","last_scrub_stamp":"2024-03-26T03:52:36.221537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:36.221537+0000","objects_scrubbed":6697,"log_size":4772,"log_dups_size":0,"ondisk_log_size":4772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.562362+0000","scrub_duration":2663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214302720,"num_objects":3270,"num_object_clones":0,"num_object_copies":9810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11172,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490031+0000","last_change":"2024-03-26T03:51:27.913279+0000","last_active":"2024-03-26T03:53:26.490031+0000","last_peered":"2024-03-26T03:53:26.490031+0000","last_clean":"2024-03-26T03:53:26.490031+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:53:26.490031+0000","last_undegraded":"2024-03-26T03:53:26.490031+0000","last_fullsized":"2024-03-26T03:53:26.490031+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:27.913164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:27.913164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496650+0000","last_change":"2024-03-26T03:52:29.251669+0000","last_active":"2024-03-26T03:53:26.496650+0000","last_peered":"2024-03-26T03:53:26.496650+0000","last_clean":"2024-03-26T03:53:26.496650+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:53:26.496650+0000","last_undegraded":"2024-03-26T03:53:26.496650+0000","last_fullsized":"2024-03-26T03:53:26.496650+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.251258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.251258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.082401+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490743+0000","last_change":"2024-03-26T03:52:35.600199+0000","last_active":"2024-03-26T03:53:26.490743+0000","last_peered":"2024-03-26T03:53:26.490743+0000","last_clean":"2024-03-26T03:53:26.490743+0000","last_became_active":"2024-03-26T03:50:45.594682+0000","last_became_peered":"2024-03-26T03:50:45.594682+0000","last_unstale":"2024-03-26T03:53:26.490743+0000","last_undegraded":"2024-03-26T03:53:26.490743+0000","last_fullsized":"2024-03-26T03:53:26.490743+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:35.600091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.600091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:57.681359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":191,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481535+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:53:26.481535+0000","last_peered":"2024-03-26T03:53:26.481535+0000","last_clean":"2024-03-26T03:53:26.481535+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:53:26.481535+0000","last_undegraded":"2024-03-26T03:53:26.481535+0000","last_fullsized":"2024-03-26T03:53:26.481535+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.717740+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"140'10791","reported_seq":10967,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.317202+0000","last_change":"2024-03-26T03:52:33.625525+0000","last_active":"2024-03-26T03:53:29.317202+0000","last_peered":"2024-03-26T03:53:29.317202+0000","last_clean":"2024-03-26T03:53:29.317202+0000","last_became_active":"2024-03-26T03:50:44.589615+0000","last_became_peered":"2024-03-26T03:50:44.589615+0000","last_unstale":"2024-03-26T03:53:29.317202+0000","last_undegraded":"2024-03-26T03:53:29.317202+0000","last_fullsized":"2024-03-26T03:53:29.317202+0000","mapping_epoch":95,"log_start":"120'5300","ondisk_log_start":"120'5300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7504","last_scrub_stamp":"2024-03-26T03:52:33.625304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.625304+0000","objects_scrubbed":6650,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.340680+0000","scrub_duration":2355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215285760,"num_objects":3285,"num_object_clones":0,"num_object_copies":9855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.487972+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:53:26.487972+0000","last_peered":"2024-03-26T03:53:26.487972+0000","last_clean":"2024-03-26T03:53:26.487972+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:53:26.487972+0000","last_undegraded":"2024-03-26T03:53:26.487972+0000","last_fullsized":"2024-03-26T03:53:26.487972+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.605441+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.495388+0000","last_change":"2024-03-26T03:52:42.382709+0000","last_active":"2024-03-26T03:53:26.495388+0000","last_peered":"2024-03-26T03:53:26.495388+0000","last_clean":"2024-03-26T03:53:26.495388+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:53:26.495388+0000","last_undegraded":"2024-03-26T03:53:26.495388+0000","last_fullsized":"2024-03-26T03:53:26.495388+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.382447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.382447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:08.004037+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":191,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498870+0000","last_change":"2024-03-26T03:52:31.320265+0000","last_active":"2024-03-26T03:53:26.498870+0000","last_peered":"2024-03-26T03:53:26.498870+0000","last_clean":"2024-03-26T03:53:26.498870+0000","last_became_active":"2024-03-26T03:50:45.592202+0000","last_became_peered":"2024-03-26T03:50:45.592202+0000","last_unstale":"2024-03-26T03:53:26.498870+0000","last_undegraded":"2024-03-26T03:53:26.498870+0000","last_fullsized":"2024-03-26T03:53:26.498870+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.320181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.320181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:37.000450+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":219,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.487133+0000","last_change":"2024-03-26T03:51:34.486120+0000","last_active":"2024-03-26T03:53:26.487133+0000","last_peered":"2024-03-26T03:53:26.487133+0000","last_clean":"2024-03-26T03:53:26.487133+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:53:26.487133+0000","last_undegraded":"2024-03-26T03:53:26.487133+0000","last_fullsized":"2024-03-26T03:53:26.487133+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:34.485979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.485979+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"140'10935","reported_seq":11102,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.509902+0000","last_change":"2024-03-26T03:52:48.148490+0000","last_active":"2024-03-26T03:53:29.509902+0000","last_peered":"2024-03-26T03:53:29.509902+0000","last_clean":"2024-03-26T03:53:29.509902+0000","last_became_active":"2024-03-26T03:50:45.590035+0000","last_became_peered":"2024-03-26T03:50:45.590035+0000","last_unstale":"2024-03-26T03:53:29.509902+0000","last_undegraded":"2024-03-26T03:53:29.509902+0000","last_fullsized":"2024-03-26T03:53:29.509902+0000","mapping_epoch":96,"log_start":"124'6100","ondisk_log_start":"124'6100","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8409","last_scrub_stamp":"2024-03-26T03:52:48.148171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.148171+0000","objects_scrubbed":5895,"log_size":4835,"log_dups_size":0,"ondisk_log_size":4835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.767336+0000","scrub_duration":3557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10935,"num_write_kb":455296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496361+0000","last_change":"2024-03-26T03:51:25.463769+0000","last_active":"2024-03-26T03:53:26.496361+0000","last_peered":"2024-03-26T03:53:26.496361+0000","last_clean":"2024-03-26T03:53:26.496361+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:53:26.496361+0000","last_undegraded":"2024-03-26T03:53:26.496361+0000","last_fullsized":"2024-03-26T03:53:26.496361+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.463664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.463664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":198,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498692+0000","last_change":"2024-03-26T03:51:41.572876+0000","last_active":"2024-03-26T03:53:26.498692+0000","last_peered":"2024-03-26T03:53:26.498692+0000","last_clean":"2024-03-26T03:53:26.498692+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:53:26.498692+0000","last_undegraded":"2024-03-26T03:53:26.498692+0000","last_fullsized":"2024-03-26T03:53:26.498692+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:51:41.572734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.572734+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481469+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:53:26.481469+0000","last_peered":"2024-03-26T03:53:26.481469+0000","last_clean":"2024-03-26T03:53:26.481469+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:53:26.481469+0000","last_undegraded":"2024-03-26T03:53:26.481469+0000","last_fullsized":"2024-03-26T03:53:26.481469+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.516195+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":290,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493853+0000","last_change":"2024-03-26T03:51:34.973521+0000","last_active":"2024-03-26T03:53:26.493853+0000","last_peered":"2024-03-26T03:53:26.493853+0000","last_clean":"2024-03-26T03:53:26.493853+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:53:26.493853+0000","last_undegraded":"2024-03-26T03:53:26.493853+0000","last_fullsized":"2024-03-26T03:53:26.493853+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:51:34.973366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.973366+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.498788+0000","last_change":"2024-03-26T03:52:25.587745+0000","last_active":"2024-03-26T03:53:26.498788+0000","last_peered":"2024-03-26T03:53:26.498788+0000","last_clean":"2024-03-26T03:53:26.498788+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:53:26.498788+0000","last_undegraded":"2024-03-26T03:53:26.498788+0000","last_fullsized":"2024-03-26T03:53:26.498788+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.587502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.587502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"140'10866","reported_seq":11042,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.503312+0000","last_change":"2024-03-26T03:52:33.050130+0000","last_active":"2024-03-26T03:53:29.503312+0000","last_peered":"2024-03-26T03:53:29.503312+0000","last_clean":"2024-03-26T03:53:29.503312+0000","last_became_active":"2024-03-26T03:50:44.584865+0000","last_became_peered":"2024-03-26T03:50:44.584865+0000","last_unstale":"2024-03-26T03:53:29.503312+0000","last_undegraded":"2024-03-26T03:53:29.503312+0000","last_fullsized":"2024-03-26T03:53:29.503312+0000","mapping_epoch":95,"log_start":"124'6100","ondisk_log_start":"124'6100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7431","last_scrub_stamp":"2024-03-26T03:52:33.049847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.049847+0000","objects_scrubbed":6680,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.994227+0000","scrub_duration":2449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":208797696,"num_objects":3186,"num_object_clones":0,"num_object_copies":9558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10866,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":171,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490822+0000","last_change":"2024-03-26T03:52:44.692881+0000","last_active":"2024-03-26T03:53:26.490822+0000","last_peered":"2024-03-26T03:53:26.490822+0000","last_clean":"2024-03-26T03:53:26.490822+0000","last_became_active":"2024-03-26T03:50:45.583936+0000","last_became_peered":"2024-03-26T03:50:45.583936+0000","last_unstale":"2024-03-26T03:53:26.490822+0000","last_undegraded":"2024-03-26T03:53:26.490822+0000","last_fullsized":"2024-03-26T03:53:26.490822+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.026793+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":182,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.502643+0000","last_change":"2024-03-26T03:51:32.007583+0000","last_active":"2024-03-26T03:53:26.502643+0000","last_peered":"2024-03-26T03:53:26.502643+0000","last_clean":"2024-03-26T03:53:26.502643+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:53:26.502643+0000","last_undegraded":"2024-03-26T03:53:26.502643+0000","last_fullsized":"2024-03-26T03:53:26.502643+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.007458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.007458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":194,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490597+0000","last_change":"2024-03-26T03:52:40.285553+0000","last_active":"2024-03-26T03:53:26.490597+0000","last_peered":"2024-03-26T03:53:26.490597+0000","last_clean":"2024-03-26T03:53:26.490597+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:53:26.490597+0000","last_undegraded":"2024-03-26T03:53:26.490597+0000","last_fullsized":"2024-03-26T03:53:26.490597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.285400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.285400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.875607+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":211,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.485371+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:53:26.485371+0000","last_peered":"2024-03-26T03:53:26.485371+0000","last_clean":"2024-03-26T03:53:26.485371+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:53:26.485371+0000","last_undegraded":"2024-03-26T03:53:26.485371+0000","last_fullsized":"2024-03-26T03:53:26.485371+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.706675+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496319+0000","last_change":"2024-03-26T03:52:25.178369+0000","last_active":"2024-03-26T03:53:26.496319+0000","last_peered":"2024-03-26T03:53:26.496319+0000","last_clean":"2024-03-26T03:53:26.496319+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:53:26.496319+0000","last_undegraded":"2024-03-26T03:53:26.496319+0000","last_fullsized":"2024-03-26T03:53:26.496319+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.178239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.178239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:31.907344+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"140'10727","reported_seq":10892,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.510037+0000","last_change":"2024-03-26T03:52:49.471047+0000","last_active":"2024-03-26T03:53:29.510037+0000","last_peered":"2024-03-26T03:53:29.510037+0000","last_clean":"2024-03-26T03:53:29.510037+0000","last_became_active":"2024-03-26T03:50:44.588356+0000","last_became_peered":"2024-03-26T03:50:44.588356+0000","last_unstale":"2024-03-26T03:53:29.510037+0000","last_undegraded":"2024-03-26T03:53:29.510037+0000","last_fullsized":"2024-03-26T03:53:29.510037+0000","mapping_epoch":95,"log_start":"123'5900","ondisk_log_start":"123'5900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8243","last_scrub_stamp":"2024-03-26T03:52:49.470707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.470707+0000","objects_scrubbed":5693,"log_size":4827,"log_dups_size":0,"ondisk_log_size":4827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.948331+0000","scrub_duration":2903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204931072,"num_objects":3127,"num_object_clones":0,"num_object_copies":9381,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3127,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10727,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":173,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499076+0000","last_change":"2024-03-26T03:53:25.552202+0000","last_active":"2024-03-26T03:53:26.499076+0000","last_peered":"2024-03-26T03:53:26.499076+0000","last_clean":"2024-03-26T03:53:26.499076+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:53:26.499076+0000","last_undegraded":"2024-03-26T03:53:26.499076+0000","last_fullsized":"2024-03-26T03:53:26.499076+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:50.318624+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":440,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490162+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:53:26.490162+0000","last_peered":"2024-03-26T03:53:26.490162+0000","last_clean":"2024-03-26T03:53:26.490162+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:53:26.490162+0000","last_undegraded":"2024-03-26T03:53:26.490162+0000","last_fullsized":"2024-03-26T03:53:26.490162+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:14.983642+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":175,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.506215+0000","last_change":"2024-03-26T03:51:18.413053+0000","last_active":"2024-03-26T03:53:26.506215+0000","last_peered":"2024-03-26T03:53:26.506215+0000","last_clean":"2024-03-26T03:53:26.506215+0000","last_became_active":"2024-03-26T03:50:45.592343+0000","last_became_peered":"2024-03-26T03:50:45.592343+0000","last_unstale":"2024-03-26T03:53:26.506215+0000","last_undegraded":"2024-03-26T03:53:26.506215+0000","last_fullsized":"2024-03-26T03:53:26.506215+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.412940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.412940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":201,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482281+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:53:26.482281+0000","last_peered":"2024-03-26T03:53:26.482281+0000","last_clean":"2024-03-26T03:53:26.482281+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:53:26.482281+0000","last_undegraded":"2024-03-26T03:53:26.482281+0000","last_fullsized":"2024-03-26T03:53:26.482281+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.072965+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:28.678409+0000","last_change":"2024-03-26T03:53:28.678428+0000","last_active":"2024-03-26T03:53:28.678409+0000","last_peered":"2024-03-26T03:53:28.678409+0000","last_clean":"2024-03-26T03:53:28.678409+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:53:28.678409+0000","last_undegraded":"2024-03-26T03:53:28.678409+0000","last_fullsized":"2024-03-26T03:53:28.678409+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:35.290355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"140'10996","reported_seq":11145,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.169183+0000","last_change":"2024-03-26T03:53:19.691120+0000","last_active":"2024-03-26T03:53:29.169183+0000","last_peered":"2024-03-26T03:53:29.169183+0000","last_clean":"2024-03-26T03:53:29.169183+0000","last_became_active":"2024-03-26T03:50:44.586504+0000","last_became_peered":"2024-03-26T03:50:44.586504+0000","last_unstale":"2024-03-26T03:53:29.169183+0000","last_undegraded":"2024-03-26T03:53:29.169183+0000","last_fullsized":"2024-03-26T03:53:29.169183+0000","mapping_epoch":95,"log_start":"122'5900","ondisk_log_start":"122'5900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10430","last_scrub_stamp":"2024-03-26T03:53:19.691027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.691027+0000","objects_scrubbed":3868,"log_size":5096,"log_dups_size":0,"ondisk_log_size":5096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:22.357406+0000","scrub_duration":1022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214695936,"num_objects":3276,"num_object_clones":0,"num_object_copies":9828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10996,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":164,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493391+0000","last_change":"2024-03-26T03:53:20.692581+0000","last_active":"2024-03-26T03:53:26.493391+0000","last_peered":"2024-03-26T03:53:26.493391+0000","last_clean":"2024-03-26T03:53:26.493391+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:53:26.493391+0000","last_undegraded":"2024-03-26T03:53:26.493391+0000","last_fullsized":"2024-03-26T03:53:26.493391+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:21.044199+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.487670+0000","last_change":"2024-03-26T03:51:40.382951+0000","last_active":"2024-03-26T03:53:26.487670+0000","last_peered":"2024-03-26T03:53:26.487670+0000","last_clean":"2024-03-26T03:53:26.487670+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:53:26.487670+0000","last_undegraded":"2024-03-26T03:53:26.487670+0000","last_fullsized":"2024-03-26T03:53:26.487670+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:40.377234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:40.377234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":475,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:28.150136+0000","last_change":"2024-03-26T03:53:21.716954+0000","last_active":"2024-03-26T03:53:28.150136+0000","last_peered":"2024-03-26T03:53:28.150136+0000","last_clean":"2024-03-26T03:53:28.150136+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:53:28.150136+0000","last_undegraded":"2024-03-26T03:53:28.150136+0000","last_fullsized":"2024-03-26T03:53:28.150136+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:53:21.716822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:21.716822+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:29.626939+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":191,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493630+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:53:26.493630+0000","last_peered":"2024-03-26T03:53:26.493630+0000","last_clean":"2024-03-26T03:53:26.493630+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:53:26.493630+0000","last_undegraded":"2024-03-26T03:53:26.493630+0000","last_fullsized":"2024-03-26T03:53:26.493630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.887919+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499324+0000","last_change":"2024-03-26T03:52:25.471379+0000","last_active":"2024-03-26T03:53:26.499324+0000","last_peered":"2024-03-26T03:53:26.499324+0000","last_clean":"2024-03-26T03:53:26.499324+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:53:26.499324+0000","last_undegraded":"2024-03-26T03:53:26.499324+0000","last_fullsized":"2024-03-26T03:53:26.499324+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.471302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.471302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:32.035263+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"140'10942","reported_seq":11119,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.162327+0000","last_change":"2024-03-26T03:52:31.921264+0000","last_active":"2024-03-26T03:53:29.162327+0000","last_peered":"2024-03-26T03:53:29.162327+0000","last_clean":"2024-03-26T03:53:29.162327+0000","last_became_active":"2024-03-26T03:50:44.593854+0000","last_became_peered":"2024-03-26T03:50:44.593854+0000","last_unstale":"2024-03-26T03:53:29.162327+0000","last_undegraded":"2024-03-26T03:53:29.162327+0000","last_fullsized":"2024-03-26T03:53:29.162327+0000","mapping_epoch":95,"log_start":"120'5500","ondisk_log_start":"120'5500","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7468","last_scrub_stamp":"2024-03-26T03:52:31.921157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.921157+0000","objects_scrubbed":6796,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:52.185172+0000","scrub_duration":1567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215220224,"num_objects":3284,"num_object_clones":0,"num_object_copies":9852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10942,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":170,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482294+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:53:26.482294+0000","last_peered":"2024-03-26T03:53:26.482294+0000","last_clean":"2024-03-26T03:53:26.482294+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:53:26.482294+0000","last_undegraded":"2024-03-26T03:53:26.482294+0000","last_fullsized":"2024-03-26T03:53:26.482294+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.478472+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":176,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493671+0000","last_change":"2024-03-26T03:51:20.065167+0000","last_active":"2024-03-26T03:53:26.493671+0000","last_peered":"2024-03-26T03:53:26.493671+0000","last_clean":"2024-03-26T03:53:26.493671+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:53:26.493671+0000","last_undegraded":"2024-03-26T03:53:26.493671+0000","last_fullsized":"2024-03-26T03:53:26.493671+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:20.065038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:20.065038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":221,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486975+0000","last_change":"2024-03-26T03:51:32.540653+0000","last_active":"2024-03-26T03:53:26.486975+0000","last_peered":"2024-03-26T03:53:26.486975+0000","last_clean":"2024-03-26T03:53:26.486975+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:53:26.486975+0000","last_undegraded":"2024-03-26T03:53:26.486975+0000","last_fullsized":"2024-03-26T03:53:26.486975+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:51:32.540268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.540268+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":107,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.487080+0000","last_change":"2024-03-26T03:51:41.399750+0000","last_active":"2024-03-26T03:53:26.487080+0000","last_peered":"2024-03-26T03:53:26.487080+0000","last_clean":"2024-03-26T03:53:26.487080+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:53:26.487080+0000","last_undegraded":"2024-03-26T03:53:26.487080+0000","last_fullsized":"2024-03-26T03:53:26.487080+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.399622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.399622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"140'10839","reported_seq":10980,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.166988+0000","last_change":"2024-03-26T03:53:23.269397+0000","last_active":"2024-03-26T03:53:29.166988+0000","last_peered":"2024-03-26T03:53:29.166988+0000","last_clean":"2024-03-26T03:53:29.166988+0000","last_became_active":"2024-03-26T03:50:45.596978+0000","last_became_peered":"2024-03-26T03:50:45.596978+0000","last_unstale":"2024-03-26T03:53:29.166988+0000","last_undegraded":"2024-03-26T03:53:29.166988+0000","last_fullsized":"2024-03-26T03:53:29.166988+0000","mapping_epoch":96,"log_start":"125'6200","ondisk_log_start":"125'6200","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10476","last_scrub_stamp":"2024-03-26T03:53:23.269247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.269247+0000","objects_scrubbed":3611,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:37.151488+0000","scrub_duration":790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211222528,"num_objects":3223,"num_object_clones":0,"num_object_copies":9669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10839,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486298+0000","last_change":"2024-03-26T03:53:16.573026+0000","last_active":"2024-03-26T03:53:26.486298+0000","last_peered":"2024-03-26T03:53:26.486298+0000","last_clean":"2024-03-26T03:53:26.486298+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:53:26.486298+0000","last_undegraded":"2024-03-26T03:53:26.486298+0000","last_fullsized":"2024-03-26T03:53:26.486298+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:32.584061+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499073+0000","last_change":"2024-03-26T03:52:29.399095+0000","last_active":"2024-03-26T03:53:26.499073+0000","last_peered":"2024-03-26T03:53:26.499073+0000","last_clean":"2024-03-26T03:53:26.499073+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:53:26.499073+0000","last_undegraded":"2024-03-26T03:53:26.499073+0000","last_fullsized":"2024-03-26T03:53:26.499073+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.398965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.398965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:35.994443+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":199,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481787+0000","last_change":"2024-03-26T03:51:17.091083+0000","last_active":"2024-03-26T03:53:26.481787+0000","last_peered":"2024-03-26T03:53:26.481787+0000","last_clean":"2024-03-26T03:53:26.481787+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:53:26.481787+0000","last_undegraded":"2024-03-26T03:53:26.481787+0000","last_fullsized":"2024-03-26T03:53:26.481787+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:51:17.090921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.090921+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":237,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.486773+0000","last_change":"2024-03-26T03:51:15.205153+0000","last_active":"2024-03-26T03:53:26.486773+0000","last_peered":"2024-03-26T03:53:26.486773+0000","last_clean":"2024-03-26T03:53:26.486773+0000","last_became_active":"2024-03-26T03:50:45.602990+0000","last_became_peered":"2024-03-26T03:50:45.602990+0000","last_unstale":"2024-03-26T03:53:26.486773+0000","last_undegraded":"2024-03-26T03:53:26.486773+0000","last_fullsized":"2024-03-26T03:53:26.486773+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:51:15.205025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:15.205025+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"140'10637","reported_seq":10780,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.495692+0000","last_change":"2024-03-26T03:53:28.940164+0000","last_active":"2024-03-26T03:53:29.495692+0000","last_peered":"2024-03-26T03:53:29.495692+0000","last_clean":"2024-03-26T03:53:29.495692+0000","last_became_active":"2024-03-26T03:50:44.586424+0000","last_became_peered":"2024-03-26T03:50:44.586424+0000","last_unstale":"2024-03-26T03:53:29.495692+0000","last_undegraded":"2024-03-26T03:53:29.495692+0000","last_fullsized":"2024-03-26T03:53:29.495692+0000","mapping_epoch":95,"log_start":"126'6400","ondisk_log_start":"126'6400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10589","last_scrub_stamp":"2024-03-26T03:53:28.939404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.939404+0000","objects_scrubbed":3267,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.880836+0000","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207683584,"num_objects":3169,"num_object_clones":0,"num_object_copies":9507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10637,"num_write_kb":441792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":171,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.490555+0000","last_change":"2024-03-26T03:51:33.934747+0000","last_active":"2024-03-26T03:53:26.490555+0000","last_peered":"2024-03-26T03:53:26.490555+0000","last_clean":"2024-03-26T03:53:26.490555+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:53:26.490555+0000","last_undegraded":"2024-03-26T03:53:26.490555+0000","last_fullsized":"2024-03-26T03:53:26.490555+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.934617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":186,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:27.558217+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:53:27.558217+0000","last_peered":"2024-03-26T03:53:27.558217+0000","last_clean":"2024-03-26T03:53:27.558217+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:53:27.558217+0000","last_undegraded":"2024-03-26T03:53:27.558217+0000","last_fullsized":"2024-03-26T03:53:27.558217+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:48.854398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":230,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481896+0000","last_change":"2024-03-26T03:51:29.300134+0000","last_active":"2024-03-26T03:53:26.481896+0000","last_peered":"2024-03-26T03:53:26.481896+0000","last_clean":"2024-03-26T03:53:26.481896+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:53:26.481896+0000","last_undegraded":"2024-03-26T03:53:26.481896+0000","last_fullsized":"2024-03-26T03:53:26.481896+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:51:29.299953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.299953+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":235,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.512481+0000","last_change":"2024-03-26T03:52:31.483741+0000","last_active":"2024-03-26T03:53:26.512481+0000","last_peered":"2024-03-26T03:53:26.512481+0000","last_clean":"2024-03-26T03:53:26.512481+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:53:26.512481+0000","last_undegraded":"2024-03-26T03:53:26.512481+0000","last_fullsized":"2024-03-26T03:53:26.512481+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:31.483593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.483593+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.632795+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"140'10809","reported_seq":10910,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.505429+0000","last_change":"2024-03-26T03:50:44.593372+0000","last_active":"2024-03-26T03:53:29.505429+0000","last_peered":"2024-03-26T03:53:29.505429+0000","last_clean":"2024-03-26T03:53:29.505429+0000","last_became_active":"2024-03-26T03:50:44.592595+0000","last_became_peered":"2024-03-26T03:50:44.592595+0000","last_unstale":"2024-03-26T03:53:29.505429+0000","last_undegraded":"2024-03-26T03:53:29.505429+0000","last_fullsized":"2024-03-26T03:53:29.505429+0000","mapping_epoch":95,"log_start":"126'6600","ondisk_log_start":"126'6600","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4209,"log_dups_size":0,"ondisk_log_size":4209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210567168,"num_objects":3213,"num_object_clones":0,"num_object_copies":9639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10809,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":157,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481386+0000","last_change":"2024-03-26T03:51:21.386995+0000","last_active":"2024-03-26T03:53:26.481386+0000","last_peered":"2024-03-26T03:53:26.481386+0000","last_clean":"2024-03-26T03:53:26.481386+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:53:26.481386+0000","last_undegraded":"2024-03-26T03:53:26.481386+0000","last_fullsized":"2024-03-26T03:53:26.481386+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.386911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.386911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496793+0000","last_change":"2024-03-26T03:52:45.556559+0000","last_active":"2024-03-26T03:53:26.496793+0000","last_peered":"2024-03-26T03:53:26.496793+0000","last_clean":"2024-03-26T03:53:26.496793+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:53:26.496793+0000","last_undegraded":"2024-03-26T03:53:26.496793+0000","last_fullsized":"2024-03-26T03:53:26.496793+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.193240+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":108,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.489055+0000","last_change":"2024-03-26T03:52:46.432028+0000","last_active":"2024-03-26T03:53:26.489055+0000","last_peered":"2024-03-26T03:53:26.489055+0000","last_clean":"2024-03-26T03:53:26.489055+0000","last_became_active":"2024-03-26T03:51:24.676652+0000","last_became_peered":"2024-03-26T03:51:24.676652+0000","last_unstale":"2024-03-26T03:53:26.489055+0000","last_undegraded":"2024-03-26T03:53:26.489055+0000","last_fullsized":"2024-03-26T03:53:26.489055+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:52:46.429387+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.429387+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:54.072480+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481767+0000","last_change":"2024-03-26T03:52:37.885150+0000","last_active":"2024-03-26T03:53:26.481767+0000","last_peered":"2024-03-26T03:53:26.481767+0000","last_clean":"2024-03-26T03:53:26.481767+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:53:26.481767+0000","last_undegraded":"2024-03-26T03:53:26.481767+0000","last_fullsized":"2024-03-26T03:53:26.481767+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:45.707219+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"140'10754","reported_seq":10853,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.479601+0000","last_change":"2024-03-26T03:50:45.583826+0000","last_active":"2024-03-26T03:53:29.479601+0000","last_peered":"2024-03-26T03:53:29.479601+0000","last_clean":"2024-03-26T03:53:29.479601+0000","last_became_active":"2024-03-26T03:50:45.583350+0000","last_became_peered":"2024-03-26T03:50:45.583350+0000","last_unstale":"2024-03-26T03:53:29.479601+0000","last_undegraded":"2024-03-26T03:53:29.479601+0000","last_fullsized":"2024-03-26T03:53:29.479601+0000","mapping_epoch":96,"log_start":"126'6500","ondisk_log_start":"126'6500","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207618048,"num_objects":3168,"num_object_clones":0,"num_object_copies":9504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3168,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10754,"num_write_kb":445504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":173,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:28.635935+0000","last_change":"2024-03-26T03:53:28.635973+0000","last_active":"2024-03-26T03:53:28.635935+0000","last_peered":"2024-03-26T03:53:28.635935+0000","last_clean":"2024-03-26T03:53:28.635935+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:53:28.635935+0000","last_undegraded":"2024-03-26T03:53:28.635935+0000","last_fullsized":"2024-03-26T03:53:28.635935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:36.514232+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":235,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482140+0000","last_change":"2024-03-26T03:52:20.706812+0000","last_active":"2024-03-26T03:53:26.482140+0000","last_peered":"2024-03-26T03:53:26.482140+0000","last_clean":"2024-03-26T03:53:26.482140+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:53:26.482140+0000","last_undegraded":"2024-03-26T03:53:26.482140+0000","last_fullsized":"2024-03-26T03:53:26.482140+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:20.706714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:20.706714+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:29.973564+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.509589+0000","last_change":"2024-03-26T03:51:26.426050+0000","last_active":"2024-03-26T03:53:26.509589+0000","last_peered":"2024-03-26T03:53:26.509589+0000","last_clean":"2024-03-26T03:53:26.509589+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:53:26.509589+0000","last_undegraded":"2024-03-26T03:53:26.509589+0000","last_fullsized":"2024-03-26T03:53:26.509589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:26.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:26.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":197,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.485266+0000","last_change":"2024-03-26T03:51:38.907371+0000","last_active":"2024-03-26T03:53:26.485266+0000","last_peered":"2024-03-26T03:53:26.485266+0000","last_clean":"2024-03-26T03:53:26.485266+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:53:26.485266+0000","last_undegraded":"2024-03-26T03:53:26.485266+0000","last_fullsized":"2024-03-26T03:53:26.485266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:38.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.907089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"140'10649","reported_seq":10749,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.348749+0000","last_change":"2024-03-26T03:50:45.591630+0000","last_active":"2024-03-26T03:53:29.348749+0000","last_peered":"2024-03-26T03:53:29.348749+0000","last_clean":"2024-03-26T03:53:29.348749+0000","last_became_active":"2024-03-26T03:50:45.591306+0000","last_became_peered":"2024-03-26T03:50:45.591306+0000","last_unstale":"2024-03-26T03:53:29.348749+0000","last_undegraded":"2024-03-26T03:53:29.348749+0000","last_fullsized":"2024-03-26T03:53:29.348749+0000","mapping_epoch":96,"log_start":"126'6400","ondisk_log_start":"126'6400","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210632728,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10649,"num_write_kb":443585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":163,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.492776+0000","last_change":"2024-03-26T03:51:19.390232+0000","last_active":"2024-03-26T03:53:26.492776+0000","last_peered":"2024-03-26T03:53:26.492776+0000","last_clean":"2024-03-26T03:53:26.492776+0000","last_became_active":"2024-03-26T03:50:45.595009+0000","last_became_peered":"2024-03-26T03:50:45.595009+0000","last_unstale":"2024-03-26T03:53:26.492776+0000","last_undegraded":"2024-03-26T03:53:26.492776+0000","last_fullsized":"2024-03-26T03:53:26.492776+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.390111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.390111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":183,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.499275+0000","last_change":"2024-03-26T03:52:42.284044+0000","last_active":"2024-03-26T03:53:26.499275+0000","last_peered":"2024-03-26T03:53:26.499275+0000","last_clean":"2024-03-26T03:53:26.499275+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:53:26.499275+0000","last_undegraded":"2024-03-26T03:53:26.499275+0000","last_fullsized":"2024-03-26T03:53:26.499275+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.283958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:50.455074+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":177,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481625+0000","last_change":"2024-03-26T03:51:25.157523+0000","last_active":"2024-03-26T03:53:26.481625+0000","last_peered":"2024-03-26T03:53:26.481625+0000","last_clean":"2024-03-26T03:53:26.481625+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:53:26.481625+0000","last_undegraded":"2024-03-26T03:53:26.481625+0000","last_fullsized":"2024-03-26T03:53:26.481625+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.157414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.157414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":224,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481966+0000","last_change":"2024-03-26T03:51:31.354516+0000","last_active":"2024-03-26T03:53:26.481966+0000","last_peered":"2024-03-26T03:53:26.481966+0000","last_clean":"2024-03-26T03:53:26.481966+0000","last_became_active":"2024-03-26T03:50:45.596131+0000","last_became_peered":"2024-03-26T03:50:45.596131+0000","last_unstale":"2024-03-26T03:53:26.481966+0000","last_undegraded":"2024-03-26T03:53:26.481966+0000","last_fullsized":"2024-03-26T03:53:26.481966+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:31.354413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:31.354413+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"140'10791","reported_seq":10966,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.314323+0000","last_change":"2024-03-26T03:52:39.118362+0000","last_active":"2024-03-26T03:53:29.314323+0000","last_peered":"2024-03-26T03:53:29.314323+0000","last_clean":"2024-03-26T03:53:29.314323+0000","last_became_active":"2024-03-26T03:50:44.590889+0000","last_became_peered":"2024-03-26T03:50:44.590889+0000","last_unstale":"2024-03-26T03:53:29.314323+0000","last_undegraded":"2024-03-26T03:53:29.314323+0000","last_fullsized":"2024-03-26T03:53:29.314323+0000","mapping_epoch":95,"log_start":"120'5300","ondisk_log_start":"120'5300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7740","last_scrub_stamp":"2024-03-26T03:52:39.118187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.118187+0000","objects_scrubbed":6400,"log_size":5491,"log_dups_size":0,"ondisk_log_size":5491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:40.537245+0000","scrub_duration":2793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":215810048,"num_objects":3293,"num_object_clones":0,"num_object_copies":9879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10791,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":173,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.495647+0000","last_change":"2024-03-26T03:52:44.431735+0000","last_active":"2024-03-26T03:53:26.495647+0000","last_peered":"2024-03-26T03:53:26.495647+0000","last_clean":"2024-03-26T03:53:26.495647+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:53:26.495647+0000","last_undegraded":"2024-03-26T03:53:26.495647+0000","last_fullsized":"2024-03-26T03:53:26.495647+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.431608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.431608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.078115+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":168,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493998+0000","last_change":"2024-03-26T03:52:30.927185+0000","last_active":"2024-03-26T03:53:26.493998+0000","last_peered":"2024-03-26T03:53:26.493998+0000","last_clean":"2024-03-26T03:53:26.493998+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:53:26.493998+0000","last_undegraded":"2024-03-26T03:53:26.493998+0000","last_fullsized":"2024-03-26T03:53:26.493998+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:30.926620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.926620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:33.001795+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":107,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481455+0000","last_change":"2024-03-26T03:51:23.188460+0000","last_active":"2024-03-26T03:53:26.481455+0000","last_peered":"2024-03-26T03:53:26.481455+0000","last_clean":"2024-03-26T03:53:26.481455+0000","last_became_active":"2024-03-26T03:50:45.593583+0000","last_became_peered":"2024-03-26T03:50:45.593583+0000","last_unstale":"2024-03-26T03:53:26.481455+0000","last_undegraded":"2024-03-26T03:53:26.481455+0000","last_fullsized":"2024-03-26T03:53:26.481455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.188316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.188316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":201,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482083+0000","last_change":"2024-03-26T03:51:32.392253+0000","last_active":"2024-03-26T03:53:26.482083+0000","last_peered":"2024-03-26T03:53:26.482083+0000","last_clean":"2024-03-26T03:53:26.482083+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:53:26.482083+0000","last_undegraded":"2024-03-26T03:53:26.482083+0000","last_fullsized":"2024-03-26T03:53:26.482083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:32.392163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.392163+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"140'10904","reported_seq":11073,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:29.510029+0000","last_change":"2024-03-26T03:52:46.641388+0000","last_active":"2024-03-26T03:53:29.510029+0000","last_peered":"2024-03-26T03:53:29.510029+0000","last_clean":"2024-03-26T03:53:29.510029+0000","last_became_active":"2024-03-26T03:50:44.587505+0000","last_became_peered":"2024-03-26T03:50:44.587505+0000","last_unstale":"2024-03-26T03:53:29.510029+0000","last_undegraded":"2024-03-26T03:53:29.510029+0000","last_fullsized":"2024-03-26T03:53:29.510029+0000","mapping_epoch":95,"log_start":"124'6100","ondisk_log_start":"124'6100","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'8232","last_scrub_stamp":"2024-03-26T03:52:46.640967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.640967+0000","objects_scrubbed":5921,"log_size":4804,"log_dups_size":0,"ondisk_log_size":4804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:51.138786+0000","scrub_duration":3071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207093760,"num_objects":3160,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":165,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.482096+0000","last_change":"2024-03-26T03:51:41.329652+0000","last_active":"2024-03-26T03:53:26.482096+0000","last_peered":"2024-03-26T03:53:26.482096+0000","last_clean":"2024-03-26T03:53:26.482096+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:53:26.482096+0000","last_undegraded":"2024-03-26T03:53:26.482096+0000","last_fullsized":"2024-03-26T03:53:26.482096+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.329573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.329573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":243,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.496066+0000","last_change":"2024-03-26T03:53:17.373928+0000","last_active":"2024-03-26T03:53:26.496066+0000","last_peered":"2024-03-26T03:53:26.496066+0000","last_clean":"2024-03-26T03:53:26.496066+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:53:26.496066+0000","last_undegraded":"2024-03-26T03:53:26.496066+0000","last_fullsized":"2024-03-26T03:53:26.496066+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:53:17.373796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.373796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.915867+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":169,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.493430+0000","last_change":"2024-03-26T03:51:32.274566+0000","last_active":"2024-03-26T03:53:26.493430+0000","last_peered":"2024-03-26T03:53:26.493430+0000","last_clean":"2024-03-26T03:53:26.493430+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:53:26.493430+0000","last_undegraded":"2024-03-26T03:53:26.493430+0000","last_fullsized":"2024-03-26T03:53:26.493430+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.274349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.274349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":206,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-03-26T03:53:26.481659+0000","last_change":"2024-03-26T03:52:31.734237+0000","last_active":"2024-03-26T03:53:26.481659+0000","last_peered":"2024-03-26T03:53:26.481659+0000","last_clean":"2024-03-26T03:53:26.481659+0000","last_became_active":"2024-03-26T03:50:45.598403+0000","last_became_peered":"2024-03-26T03:50:45.598403+0000","last_unstale":"2024-03-26T03:53:26.481659+0000","last_undegraded":"2024-03-26T03:53:26.481659+0000","last_fullsized":"2024-03-26T03:53:26.481659+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:46.994530+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3380215832,"num_objects":51579,"num_object_clones":0,"num_object_copies":154737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51579,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":173125,"num_write_kb":7190465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10147999744,"data_stored":10147987528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75625,"ondisk_log_size":75625,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070525,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1127408,"kb_used_data":1042088,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":92617744,"statfs":{"total":95995035648,"available":94840569856,"internally_reserved":0,"allocated":1067098112,"data_stored":1066717065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":87357898},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1808716,"kb_used_data":1678788,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91936436,"statfs":{"total":95995035648,"available":94142910464,"internally_reserved":0,"allocated":1719078912,"data_stored":1718668370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1807108,"kb_used_data":1681532,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":91938044,"statfs":{"total":95995035648,"available":94144557056,"internally_reserved":0,"allocated":1721888768,"data_stored":1721487387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1785624,"kb_used_data":1655760,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":91959528,"statfs":{"total":95995035648,"available":94166556672,"internally_reserved":0,"allocated":1695498240,"data_stored":1695139271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2198588,"kb_used_data":2058932,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":91546564,"statfs":{"total":95995035648,"available":93743681536,"internally_reserved":0,"allocated":2108346368,"data_stored":2107952339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":142801351},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953548,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1144116,"kb_used_data":1054632,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":92601036,"statfs":{"total":95995035648,"available":94823460864,"internally_reserved":0,"allocated":1079943168,"data_stored":1079559893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659206,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":915808,"kb_used_data":844376,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":92829344,"statfs":{"total":95995035648,"available":95057248256,"internally_reserved":0,"allocated":864641024,"data_stored":864274171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1688604672,"data_stored":1688600600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1696731136,"data_stored":1696727064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1693384704,"data_stored":1693384704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2095845376,"data_stored":2095841304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1064566784,"data_stored":1064566784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":854851584,"data_stored":854851584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1054015488,"data_stored":1054015488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:53:36.376 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-26T03:53:36.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:36.450 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:36.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:36.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:36.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:36.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:36.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:36.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.842 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:36.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[37501]: pgmap v192: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T03:53:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T03:53:36.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T03:53:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[29446]: pgmap v192: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T03:53:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T03:53:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:36 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T03:53:36.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:36.970 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:36.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:37.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:36 smithi086 ceph-mon[32965]: pgmap v192: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-26T03:53:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:36 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T03:53:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:36 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T03:53:37.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:37.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:37.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:37.254 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:37.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.269 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:37.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:37.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:37.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:37.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:37.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:37.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:37.607 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:53:37.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:37.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:37.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:37.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:37.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:37.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3715820355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3715820355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:37 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:37.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:37.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:37.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:37.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:38.005 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:53:38.006 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:53:38.006 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:53:38.006 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:53:38.006 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:53:38.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:38.068 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:38.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:37 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T03:53:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:37 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T03:53:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3715820355' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:37 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:53:38.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:38.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:38.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:38.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:38.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:38.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:38.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:38.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:38.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:38.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:38.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:38.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: pgmap v193: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 943 op/s 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[37501]: osdmap e142: 8 total, 8 up, 7 in 2024-03-26T03:53:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: pgmap v193: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 943 op/s 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:53:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:38 smithi006 ceph-mon[29446]: osdmap e142: 8 total, 8 up, 7 in 2024-03-26T03:53:38.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:38.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:38.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:38.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:39.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: pgmap v193: 148 pgs: 148 active+clean; 2.9 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 943 op/s 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T03:53:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T03:53:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:53:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:38 smithi086 ceph-mon[32965]: osdmap e142: 8 total, 8 up, 7 in 2024-03-26T03:53:39.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:39.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:39.244 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:39.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:39.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.372 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:39.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:39.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:39.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:39.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:53:39.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:39.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:39.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:39.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:39.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:39.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:39 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:39.911 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:39.911 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:39 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:39.911 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:39 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:39.924 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:39.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:40.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:40.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:40.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:40.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:40.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:40.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:40.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:40.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:40.462 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:40.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:40.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:40.608 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:40.686 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:40.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.760 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:40.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:40 smithi086 ceph-mon[32965]: pgmap v195: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-26T03:53:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:40 smithi086 ceph-mon[32965]: osdmap e143: 8 total, 8 up, 7 in 2024-03-26T03:53:40.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:40 smithi006 ceph-mon[37501]: pgmap v195: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-26T03:53:40.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:40 smithi006 ceph-mon[37501]: osdmap e143: 8 total, 8 up, 7 in 2024-03-26T03:53:40.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:40 smithi006 ceph-mon[29446]: pgmap v195: 148 pgs: 148 active+clean; 2.8 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 713 op/s 2024-03-26T03:53:40.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:40 smithi006 ceph-mon[29446]: osdmap e143: 8 total, 8 up, 7 in 2024-03-26T03:53:40.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:40.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:40.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:40.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:41.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:41.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:41.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:41.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:41.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:41.377 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:53:41.377 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:53:41.377 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:53:41.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:53:41.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:41.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:41.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:41.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:41.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:41.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:41.697 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:41.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:41.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:41.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[37501]: osdmap e144: 8 total, 8 up, 7 in 2024-03-26T03:53:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:41 smithi006 ceph-mon[29446]: osdmap e144: 8 total, 8 up, 7 in 2024-03-26T03:53:41.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:41.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:41.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:41.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:42.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:42.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:42.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:53:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:41 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:41 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:41 smithi086 ceph-mon[32965]: osdmap e144: 8 total, 8 up, 7 in 2024-03-26T03:53:42.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:42.200 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:42.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:42.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:42.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:42.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:42.625 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:42.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:42.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:42.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:42.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:42.913 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:42.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:42.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:42 smithi006 ceph-mon[29446]: pgmap v198: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:42.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:42 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:53:42.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:42 smithi006 ceph-mon[37501]: pgmap v198: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:42.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:42 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:53:43.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:43.038 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:42 smithi086 ceph-mon[32965]: pgmap v198: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:42 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:53:43.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.193 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:43.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:43.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:43.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:43.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:43.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:43.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:43.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:43.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:43.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:43.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:43.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:43.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:43.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:43.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:44.078 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:53:44.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:44.119 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:53:44.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.125 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:44.168 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:53:44.208 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:53:44.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:44.247 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:53:44.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:44.292 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:53:44.336 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:44.336 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:53:44.374 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:53:44.379 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:53:44.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:44.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:44.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:44.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:44.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:44.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:44.647 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:44.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:44.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:44.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:44.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:44 smithi086 ceph-mon[32965]: pgmap v199: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1428145206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:44.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:44.844 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:44.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:44.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:44.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:44.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:45.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:45.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:45.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:45.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:44 smithi006 ceph-mon[37501]: pgmap v199: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:45.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1428145206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:45.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:44 smithi006 ceph-mon[29446]: pgmap v199: 148 pgs: 148 active+clean; 2.5 GiB data, 8.5 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 852 op/s 2024-03-26T03:53:45.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1428145206' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:53:45.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:45.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:45.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:45.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.391 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:45.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:45.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:45.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:45.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:45.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:45.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:45.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:45.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:45.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:45.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:45.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:46.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:46.042 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:46.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:46.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:46.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:46.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:46.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:46.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:46.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:46.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:46.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:46.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:46.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:46.727 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:46.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:46.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:46.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:46.929 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:46 smithi006 ceph-mon[37501]: pgmap v200: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-26T03:53:46.929 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:46 smithi006 ceph-mon[29446]: pgmap v200: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-26T03:53:46.942 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:46.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:47.038 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.039 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:53:47.039 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:53:47.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:46 smithi086 ceph-mon[32965]: pgmap v200: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-26T03:53:47.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:47.181 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:47.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:47.320 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:47.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:47.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:47.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:47.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:47.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:47.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:47.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:47.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:47.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:47.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:47.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: osdmap e145: 8 total, 8 up, 7 in 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:53:47.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: osdmap e145: 8 total, 8 up, 7 in 2024-03-26T03:53:47.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:53:47.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:47 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:53:47.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:47.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:47.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:48.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:53:48.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: osdmap e145: 8 total, 8 up, 7 in 2024-03-26T03:53:48.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:53:48.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:47 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:53:48.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:48.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:48.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:48.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:48.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:48.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:53:48.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:48.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:48.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:48.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:53:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.648 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:48.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:48.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:48.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:48.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:48.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:48.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:48.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:49.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:49.071 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:49.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:49.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:48 smithi086 ceph-mon[32965]: pgmap v202: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 669 B/s rd, 0 B/s wr, 627 op/s 2024-03-26T03:53:49.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:48 smithi006 ceph-mon[37501]: pgmap v202: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 669 B/s rd, 0 B/s wr, 627 op/s 2024-03-26T03:53:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:48 smithi006 ceph-mon[29446]: pgmap v202: 148 pgs: 148 active+clean; 2.3 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 669 B/s rd, 0 B/s wr, 627 op/s 2024-03-26T03:53:49.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:49.214 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:49.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:49.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:49.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:49.500 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:49.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.678 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:49.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:49.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:49.813 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:49.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:49.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:49.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:49.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:49.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:50.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:49 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:50.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:49 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:50.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:49 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:50.018 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:49 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:50.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:49 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:49 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:50.087 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:50.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:50.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:50.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:50.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:50.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:50.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:50.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:50.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:50.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:50.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:50.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:50.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:50.890 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:50.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:50.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:50.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:51.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:51.032 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:50 smithi006 ceph-mon[37501]: pgmap v203: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s 2024-03-26T03:53:51.032 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:50 smithi006 ceph-mon[37501]: osdmap e146: 8 total, 8 up, 7 in 2024-03-26T03:53:51.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:50 smithi006 ceph-mon[29446]: pgmap v203: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s 2024-03-26T03:53:51.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:50 smithi006 ceph-mon[29446]: osdmap e146: 8 total, 8 up, 7 in 2024-03-26T03:53:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:50 smithi086 ceph-mon[32965]: pgmap v203: 148 pgs: 148 active+clean; 2.2 GiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 667 op/s 2024-03-26T03:53:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:50 smithi086 ceph-mon[32965]: osdmap e146: 8 total, 8 up, 7 in 2024-03-26T03:53:51.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:51.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:51.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:51.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:51.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:51.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:51.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:51.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:51.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:51.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:51.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:51.778 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:51.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:51.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:51.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:51.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:52.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:52.056 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:53:52.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:52.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:52.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:52.274 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:52.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:52.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:52.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:52.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:52.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:52.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:52.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:52.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:52.795 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:53:52.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[29446]: pgmap v205: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:53:52.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:52.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:52.795 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:52.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[37501]: pgmap v205: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:53:52.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:52.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:52.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:52.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:52.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:52.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:53.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:53.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:52 smithi086 ceph-mon[32965]: pgmap v205: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-26T03:53:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:52 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:52 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:53:53.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:53.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:53.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:53.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:53.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:53.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:53.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:53.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:53.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:53.530 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 147, 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-03-26T03:43:14.534401+0000", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-03-26T03:53:52.773572+0000", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-03-26T03:48:30.373726+0000", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-03-26T03:50:44.564680+0000", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5806080, 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "nodeep-scrub", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "noscrub", 2024-03-26T03:53:53.531 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 18, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 7, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 8, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "luminous", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-26T03:53:53.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:46:30.183658+0000", 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.533 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 1, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 1, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 1, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 1, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 1, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.534 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "21", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.535 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.536 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-03-26T03:53:53.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 7.8899998664855957, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 7.8899998664855957, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 3, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 3, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "datapool", 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:41.622497+0000", 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.539 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 3, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 3, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 3, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 3, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 3, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.540 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 69, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.541 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.542 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.75, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2, 2024-03-26T03:53:53.543 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".rgw.root", 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:42.191818+0000", 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.544 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "67", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.547 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.548 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.549 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 4, 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:53:53.550 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:44.079106+0000", 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:53:53.551 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.553 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.554 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.555 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:53:53.556 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 5, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:46.035261+0000", 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.557 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.558 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "71", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.559 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.560 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:53:53.561 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 6, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:48.086810+0000", 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.562 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.563 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "73", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.564 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.565 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:53:53.566 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_bias": 4 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 7, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_0", 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:50:43.457541+0000", 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:53:53.568 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 16, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 16, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 16, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 16, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 16, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.569 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "101", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:53:53.570 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:53:53.571 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "rados": {} 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:53:53.572 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 1, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 1, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 1 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:53:53.573 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e", 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 8, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 96, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-03-26T03:53:53.574 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.575 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.576 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/1246039834", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6802/1246039834", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6804/1246039834", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6803/1246039834", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "753f7459-62ec-447b-a0da-64d06e813f16", 2024-03-26T03:53:53.577 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.578 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.578 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.578 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.578 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.578 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 13, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 96, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.580 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6805/260311411", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6806/260311411", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6808/260311411", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6807/260311411", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.581 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b7d7409f-341a-4711-8570-92fe9082a528", 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 18, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 110, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.582 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:53:53.583 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.584 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/3103276794", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6810/3103276794", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6812/3103276794", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6811/3103276794", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6", 2024-03-26T03:53:53.585 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 25, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 96, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-03-26T03:53:53.586 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.587 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6813/2795883342", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6814/2795883342", 2024-03-26T03:53:53.588 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6816/2795883342", 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6815/2795883342", 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8", 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 32, 2024-03-26T03:53:53.589 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 96, 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6800", 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.590 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6801", 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6803", 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6802", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6800/931538411", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6801/931538411", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6803/931538411", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6802/931538411", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.592 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc", 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 41, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 96, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.593 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6804", 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 529279948 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6805", 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 529279948 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.594 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6807", 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 529279948 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6806", 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 529279948 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.595 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6804/529279948", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6805/529279948", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6807/529279948", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6806/529279948", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "58b58555-59a3-4620-a15f-024f7041713f", 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.596 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 50, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 95, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6808", 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6809", 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6811", 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.598 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6810", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6808/1580133777", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6809/1580133777", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6811/1580133777", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6810/1580133777", 2024-03-26T03:53:53.599 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7", 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 59, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 99, 2024-03-26T03:53:53.600 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6812", 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6813", 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:53:53.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6815", 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:53:53.602 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6814", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6812/3905426306", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6813/3905426306", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6815/3905426306", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6814/3905426306", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.603 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:06.303060+0000", 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.604 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:40.193548+0000", 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:52:46.753786+0000", 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:31.707829+0000", 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.606 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:48.891392+0000", 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:45.947207+0000", 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.607 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:51.976637+0000", 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:53:39.143882+0000", 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.608 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [], 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [ 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.0", 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 3, 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.b", 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.609 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 5 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.17", 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.1f", 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:53:53.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [], 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": { 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1978459317": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1771670410": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/3964053112": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/14180211": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/2547319348": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.611 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/601470125": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/3870917014": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1892511221": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/435737617": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/93567168": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/2150730999": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/1166209788": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1248481299": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1832156537": "2024-03-27T03:44:16.095969+0000" 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-03-26T03:53:53.612 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [], 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:53:53.613 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:53:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:53.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:53.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:53.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:53.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:53.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:53.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:53.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:53 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:53 smithi086 ceph-mon[32965]: osdmap e147: 8 total, 8 up, 7 in 2024-03-26T03:53:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4175709992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:53:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:54.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[37501]: osdmap e147: 8 total, 8 up, 7 in 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4175709992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2266245881' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[29446]: osdmap e147: 8 total, 8 up, 7 in 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4175709992' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:53:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:53:54.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:54.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:54.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:54.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:54.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:54.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:54.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:54.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:54.607 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:53:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:53:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:53:54.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:54.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:54.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:54.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:53:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:53:54.804 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:54.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:54 smithi006 ceph-mon[37501]: pgmap v207: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 781 B/s rd, 0 B/s wr, 769 op/s 2024-03-26T03:53:54.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:54 smithi006 ceph-mon[29446]: pgmap v207: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 781 B/s rd, 0 B/s wr, 769 op/s 2024-03-26T03:53:54.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:54.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:54.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:54.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:55.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:54 smithi086 ceph-mon[32965]: pgmap v207: 148 pgs: 148 active+clean; 1.9 GiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 781 B/s rd, 0 B/s wr, 769 op/s 2024-03-26T03:53:55.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:55.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:55.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:55.328 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:55.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:55.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:55.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:55.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:55.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:55.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:55.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:55.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:55.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:55.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:55.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:55.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:55.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:56.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:56.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.129 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:53:56.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:56.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:56.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:56.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:56.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:56.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:56.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:56.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:56.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:56.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:56.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:56.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:56.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:56 smithi006 ceph-mon[37501]: pgmap v208: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T03:53:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:56 smithi006 ceph-mon[29446]: pgmap v208: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T03:53:56.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:56.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:56.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:57.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:57.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:57.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:56 smithi086 ceph-mon[32965]: pgmap v208: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T03:53:57.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:53:57.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:57.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:53:57.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:57.320 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:57.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:57.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:57.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:57.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:57.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:57.596 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:57.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:57.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:57.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:57.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:57.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:57.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:57.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:57.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:58.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:57 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:53:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:57 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:53:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:57 smithi086 ceph-mon[32965]: osdmap e148: 8 total, 8 up, 7 in 2024-03-26T03:53:58.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:58.115 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:53:58.116 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:53:58.116 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[29446]: osdmap e148: 8 total, 8 up, 7 in 2024-03-26T03:53:58.117 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:53:58.117 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:53:58.117 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:57 smithi006 ceph-mon[37501]: osdmap e148: 8 total, 8 up, 7 in 2024-03-26T03:53:58.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:58.174 INFO:tasks.thrashosds.thrasher:j is {'epoch': 147, 'fsid': 'de980954-eb22-11ee-95d0-87774f69a715', 'created': '2024-03-26T03:43:14.534401+0000', 'modified': '2024-03-26T03:53:52.773572+0000', 'last_up_change': '2024-03-26T03:48:30.373726+0000', 'last_in_change': '2024-03-26T03:50:44.564680+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T03:46:30.183658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T03:48:41.622497+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T03:48:42.191818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T03:48:44.079106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T03:48:46.035261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T03:48:48.086810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-26T03:50:43.457541+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1d1c84cf-5fa7-437e-8004-3b7aca831d6e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6801', 'nonce': 1246039834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 1246039834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 1246039834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6803', 'nonce': 1246039834}]}, 'public_addr': '172.21.15.6:6801/1246039834', 'cluster_addr': '172.21.15.6:6802/1246039834', 'heartbeat_back_addr': '172.21.15.6:6804/1246039834', 'heartbeat_front_addr': '172.21.15.6:6803/1246039834', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '753f7459-62ec-447b-a0da-64d06e813f16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6805', 'nonce': 260311411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 260311411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 260311411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6807', 'nonce': 260311411}]}, 'public_addr': '172.21.15.6:6805/260311411', 'cluster_addr': '172.21.15.6:6806/260311411', 'heartbeat_back_addr': '172.21.15.6:6808/260311411', 'heartbeat_front_addr': '172.21.15.6:6807/260311411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b7d7409f-341a-4711-8570-92fe9082a528', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 110, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6809', 'nonce': 3103276794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 3103276794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 3103276794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6811', 'nonce': 3103276794}]}, 'public_addr': '172.21.15.6:6809/3103276794', 'cluster_addr': '172.21.15.6:6810/3103276794', 'heartbeat_back_addr': '172.21.15.6:6812/3103276794', 'heartbeat_front_addr': '172.21.15.6:6811/3103276794', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a91459a0-dee4-420c-ae2f-940756c985a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6813', 'nonce': 2795883342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 2795883342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2795883342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6815', 'nonce': 2795883342}]}, 'public_addr': '172.21.15.6:6813/2795883342', 'cluster_addr': '172.21.15.6:6814/2795883342', 'heartbeat_back_addr': '172.21.15.6:6816/2795883342', 'heartbeat_front_addr': '172.21.15.6:6815/2795883342', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '087da212-7bf7-4982-b7cc-11f3f66c7ff8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 931538411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 931538411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6803', 'nonce': 931538411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6802', 'nonce': 931538411}]}, 'public_addr': '172.21.15.86:6800/931538411', 'cluster_addr': '172.21.15.86:6801/931538411', 'heartbeat_back_addr': '172.21.15.86:6803/931538411', 'heartbeat_front_addr': '172.21.15.86:6802/931538411', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7a688745-b213-4bbc-a735-42ffd2a5bfdc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 529279948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 529279948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6807', 'nonce': 529279948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6806', 'nonce': 529279948}]}, 'public_addr': '172.21.15.86:6804/529279948', 'cluster_addr': '172.21.15.86:6805/529279948', 'heartbeat_back_addr': '172.21.15.86:6807/529279948', 'heartbeat_front_addr': '172.21.15.86:6806/529279948', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '58b58555-59a3-4620-a15f-024f7041713f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 1580133777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 1580133777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6811', 'nonce': 1580133777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6810', 'nonce': 1580133777}]}, 'public_addr': '172.21.15.86:6808/1580133777', 'cluster_addr': '172.21.15.86:6809/1580133777', 'heartbeat_back_addr': '172.21.15.86:6811/1580133777', 'heartbeat_front_addr': '172.21.15.86:6810/1580133777', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '49b1833d-0a24-4168-a195-47b8ac1d9bc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 3905426306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 3905426306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6815', 'nonce': 3905426306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6814', 'nonce': 3905426306}]}, 'public_addr': '172.21.15.86:6812/3905426306', 'cluster_addr': '172.21.15.86:6813/3905426306', 'heartbeat_back_addr': '172.21.15.86:6815/3905426306', 'heartbeat_front_addr': '172.21.15.86:6814/3905426306', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:06.303060+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:40.193548+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:52:46.753786+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:31.707829+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:48.891392+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:45.947207+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:53:51.976637+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:53:39.143882+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.0', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.6:0/1978459317': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/1771670410': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/3964053112': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/14180211': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:6800/2547319348': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/601470125': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:6800/3870917014': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1892511221': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/435737617': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/93567168': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/2150730999': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:6800/1166209788': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/1248481299': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1832156537': '2024-03-27T03:44:16.095969+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:53:58.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:53:58.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:53:58.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.300 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:53:58.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:58.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:58.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:58.598 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:58.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:58.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:58.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:58.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:58.869 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:58.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:58.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:53:58.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:58.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:53:59.071 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:53:59.071 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:53:59.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:53:58 smithi006 ceph-mon[37501]: pgmap v210: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 654 op/s 2024-03-26T03:53:59.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:53:58 smithi006 ceph-mon[29446]: pgmap v210: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 654 op/s 2024-03-26T03:53:59.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:53:59.136 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:59.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:53:59.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:59.275 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:59.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:53:58 smithi086 ceph-mon[32965]: pgmap v210: 148 pgs: 148 active+clean; 1.6 GiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 B/s wr, 654 op/s 2024-03-26T03:53:59.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:53:59.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:59.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:53:59.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:53:59.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:53:59.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:53:59.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:53:59.829 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:53:59.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:53:59.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:53:59.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:53:59.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:53:59.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:00.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:00.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.098 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:00.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:00.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:00.241 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:00.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:00.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:54:00.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:00.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:00.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:00.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:00.660 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:00.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:00.719 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:54:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:00.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:00.776 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:54:00.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:00.821 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:54:00.864 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:54:00.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:00.904 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:54:00.943 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:54:00.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:00.949 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:00.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:00.985 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:54:01.024 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:54:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:54:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:54:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:54:01.037 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:54:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:54:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:54:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:54:01.038 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:54:01.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:01.083 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:00 smithi006 ceph-mon[37501]: pgmap v211: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 719 op/s 2024-03-26T03:54:01.083 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3677555105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:01.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:00 smithi006 ceph-mon[29446]: pgmap v211: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 719 op/s 2024-03-26T03:54:01.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3677555105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:01.104 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:01.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:01.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:00 smithi086 ceph-mon[32965]: pgmap v211: 148 pgs: 148 active+clean; 1.6 GiB data, 5.6 GiB used, 620 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 719 op/s 2024-03-26T03:54:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3677555105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:01.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:01.380 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:01.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:54:01.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:01.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:01.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:01.680 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:01.712 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:01.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:01.845 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:01.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:01.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:02.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:02.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:02.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:02.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:02.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:02.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:02.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:02.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:02.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:02.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:02.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:02.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:02.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:02.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:02.865 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:02.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:02.915 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:02 smithi006 ceph-mon[37501]: pgmap v212: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:54:02.915 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:02 smithi006 ceph-mon[29446]: pgmap v212: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:54:02.967 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:03.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:03.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:03.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.153 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:03.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:03.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:03.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:02 smithi086 ceph-mon[32965]: pgmap v212: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-03-26T03:54:03.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:03.399 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:03.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.447 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:03.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:03.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:03.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:03.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:03.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:03.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:03.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:03.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:03.848 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:03.909 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:54:03.912 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:54:03.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:03.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:03.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:04.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:04.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:04.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:04.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:04.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:04.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:04.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:04.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:04.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:04.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:04.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:04.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:54:04.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:04.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:04.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:04.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:04.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:04.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:04.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:04.921 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:04 smithi006 ceph-mon[37501]: pgmap v213: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-26T03:54:04.923 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:04 smithi006 ceph-mon[29446]: pgmap v213: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-26T03:54:04.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:05.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:05.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:05.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:05.204 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:05.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:05.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:04 smithi086 ceph-mon[32965]: pgmap v213: 148 pgs: 148 active+clean; 1.4 GiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 977 op/s 2024-03-26T03:54:05.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:05.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:05.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.490 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:05.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:05.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:05.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:05.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:05.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.789 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:05.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:05.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:05.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:05.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:05.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:06.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:06.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:06.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:06.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:05 smithi006 ceph-mon[29446]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:05 smithi006 ceph-mon[37501]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:06.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:06.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:06.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:06.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:06.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:06.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:05 smithi086 ceph-mon[32965]: from='client.24793 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:06.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:06.439 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:06.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:06.553 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":213,"stamp":"2024-03-26T03:54:03.268266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1449925056,"num_objects":22346,"num_object_clones":0,"num_object_copies":67038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22346,"num_whiteouts":0,"num_read":1528,"num_read_kb":1342,"num_write":203395,"num_write_kb":7192926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":2362855,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76218,"ondisk_log_size":76218,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":5244856,"kb_used_data":4402544,"kb_used_omap":10,"kb_used_meta":841461,"kb_avail":650971208,"statfs":{"total":671965249536,"available":666594516992,"internally_reserved":0,"allocated":4508205056,"data_stored":4505640525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11158,"internal_metadata":861656170},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-640155648,"num_objects":-9768,"num_object_clones":0,"num_object_copies":-29304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9768,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9768,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004245"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807352+0000","last_change":"2024-03-26T03:53:29.435229+0000","last_active":"2024-03-26T03:53:56.807352+0000","last_peered":"2024-03-26T03:53:56.807352+0000","last_clean":"2024-03-26T03:53:56.807352+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:53:56.807352+0000","last_undegraded":"2024-03-26T03:53:56.807352+0000","last_fullsized":"2024-03-26T03:53:56.807352+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:49.505917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825515+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:53:56.825515+0000","last_peered":"2024-03-26T03:53:56.825515+0000","last_clean":"2024-03-26T03:53:56.825515+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:53:56.825515+0000","last_undegraded":"2024-03-26T03:53:56.825515+0000","last_fullsized":"2024-03-26T03:53:56.825515+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.054186+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826521+0000","last_change":"2024-03-26T03:51:39.331101+0000","last_active":"2024-03-26T03:53:56.826521+0000","last_peered":"2024-03-26T03:53:56.826521+0000","last_clean":"2024-03-26T03:53:56.826521+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:53:56.826521+0000","last_undegraded":"2024-03-26T03:53:56.826521+0000","last_fullsized":"2024-03-26T03:53:56.826521+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.330345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.330345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656847+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":235,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808909+0000","last_change":"2024-03-26T03:52:33.804342+0000","last_active":"2024-03-26T03:53:56.808909+0000","last_peered":"2024-03-26T03:53:56.808909+0000","last_clean":"2024-03-26T03:53:56.808909+0000","last_became_active":"2024-03-26T03:50:45.598168+0000","last_became_peered":"2024-03-26T03:50:45.598168+0000","last_unstale":"2024-03-26T03:53:56.808909+0000","last_undegraded":"2024-03-26T03:53:56.808909+0000","last_fullsized":"2024-03-26T03:53:56.808909+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484700+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827403+0000","last_change":"2024-03-26T03:52:43.442740+0000","last_active":"2024-03-26T03:53:56.827403+0000","last_peered":"2024-03-26T03:53:56.827403+0000","last_clean":"2024-03-26T03:53:56.827403+0000","last_became_active":"2024-03-26T03:50:45.597941+0000","last_became_peered":"2024-03-26T03:50:45.597941+0000","last_unstale":"2024-03-26T03:53:56.827403+0000","last_undegraded":"2024-03-26T03:53:56.827403+0000","last_fullsized":"2024-03-26T03:53:56.827403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524350+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807281+0000","last_change":"2024-03-26T03:53:13.396637+0000","last_active":"2024-03-26T03:53:56.807281+0000","last_peered":"2024-03-26T03:53:56.807281+0000","last_clean":"2024-03-26T03:53:56.807281+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:53:56.807281+0000","last_undegraded":"2024-03-26T03:53:56.807281+0000","last_fullsized":"2024-03-26T03:53:56.807281+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.396551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.396551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:31.835565+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827574+0000","last_change":"2024-03-26T03:51:21.399264+0000","last_active":"2024-03-26T03:53:56.827574+0000","last_peered":"2024-03-26T03:53:56.827574+0000","last_clean":"2024-03-26T03:53:56.827574+0000","last_became_active":"2024-03-26T03:50:45.581989+0000","last_became_peered":"2024-03-26T03:50:45.581989+0000","last_unstale":"2024-03-26T03:53:56.827574+0000","last_undegraded":"2024-03-26T03:53:56.827574+0000","last_fullsized":"2024-03-26T03:53:56.827574+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.398535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.398535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524172+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":262,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.823084+0000","last_change":"2024-03-26T03:53:26.317218+0000","last_active":"2024-03-26T03:53:56.823084+0000","last_peered":"2024-03-26T03:53:56.823084+0000","last_clean":"2024-03-26T03:53:56.823084+0000","last_became_active":"2024-03-26T03:48:46.020449+0000","last_became_peered":"2024-03-26T03:48:46.020449+0000","last_unstale":"2024-03-26T03:53:56.823084+0000","last_undegraded":"2024-03-26T03:53:56.823084+0000","last_fullsized":"2024-03-26T03:53:56.823084+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":11,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:53.761677+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818846+0000","last_change":"2024-03-26T03:53:22.532263+0000","last_active":"2024-03-26T03:53:56.818846+0000","last_peered":"2024-03-26T03:53:56.818846+0000","last_clean":"2024-03-26T03:53:56.818846+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:53:56.818846+0000","last_undegraded":"2024-03-26T03:53:56.818846+0000","last_fullsized":"2024-03-26T03:53:56.818846+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:22.532151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:22.532151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:28.487704+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819254+0000","last_change":"2024-03-26T03:53:24.585622+0000","last_active":"2024-03-26T03:53:56.819254+0000","last_peered":"2024-03-26T03:53:56.819254+0000","last_clean":"2024-03-26T03:53:56.819254+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:53:56.819254+0000","last_undegraded":"2024-03-26T03:53:56.819254+0000","last_fullsized":"2024-03-26T03:53:56.819254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.585508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.585508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:28.460358+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819633+0000","last_change":"2024-03-26T03:53:18.575050+0000","last_active":"2024-03-26T03:53:56.819633+0000","last_peered":"2024-03-26T03:53:56.819633+0000","last_clean":"2024-03-26T03:53:56.819633+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:53:56.819633+0000","last_undegraded":"2024-03-26T03:53:56.819633+0000","last_fullsized":"2024-03-26T03:53:56.819633+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:34.696377+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":238,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807568+0000","last_change":"2024-03-26T03:53:34.457938+0000","last_active":"2024-03-26T03:53:56.807568+0000","last_peered":"2024-03-26T03:53:56.807568+0000","last_clean":"2024-03-26T03:53:56.807568+0000","last_became_active":"2024-03-26T03:48:46.020868+0000","last_became_peered":"2024-03-26T03:48:46.020868+0000","last_unstale":"2024-03-26T03:53:56.807568+0000","last_undegraded":"2024-03-26T03:53:56.807568+0000","last_fullsized":"2024-03-26T03:53:56.807568+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.741751+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819067+0000","last_change":"2024-03-26T03:52:26.427599+0000","last_active":"2024-03-26T03:53:56.819067+0000","last_peered":"2024-03-26T03:53:56.819067+0000","last_clean":"2024-03-26T03:53:56.819067+0000","last_became_active":"2024-03-26T03:48:50.039653+0000","last_became_peered":"2024-03-26T03:48:50.039653+0000","last_unstale":"2024-03-26T03:53:56.819067+0000","last_undegraded":"2024-03-26T03:53:56.819067+0000","last_fullsized":"2024-03-26T03:53:56.819067+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.427501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.427501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.518585+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.814994+0000","last_change":"2024-03-26T03:52:52.666181+0000","last_active":"2024-03-26T03:53:56.814994+0000","last_peered":"2024-03-26T03:53:56.814994+0000","last_clean":"2024-03-26T03:53:56.814994+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:53:56.814994+0000","last_undegraded":"2024-03-26T03:53:56.814994+0000","last_fullsized":"2024-03-26T03:53:56.814994+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.666107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.666107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.715049+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819315+0000","last_change":"2024-03-26T03:52:28.376930+0000","last_active":"2024-03-26T03:53:56.819315+0000","last_peered":"2024-03-26T03:53:56.819315+0000","last_clean":"2024-03-26T03:53:56.819315+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:53:56.819315+0000","last_undegraded":"2024-03-26T03:53:56.819315+0000","last_fullsized":"2024-03-26T03:53:56.819315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:28.376798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:28.376798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.543920+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807863+0000","last_change":"2024-03-26T03:52:38.915695+0000","last_active":"2024-03-26T03:53:56.807863+0000","last_peered":"2024-03-26T03:53:56.807863+0000","last_clean":"2024-03-26T03:53:56.807863+0000","last_became_active":"2024-03-26T03:48:46.023427+0000","last_became_peered":"2024-03-26T03:48:46.023427+0000","last_unstale":"2024-03-26T03:53:56.807863+0000","last_undegraded":"2024-03-26T03:53:56.807863+0000","last_fullsized":"2024-03-26T03:53:56.807863+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.912180+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808457+0000","last_change":"2024-03-26T03:52:51.658215+0000","last_active":"2024-03-26T03:53:56.808457+0000","last_peered":"2024-03-26T03:53:56.808457+0000","last_clean":"2024-03-26T03:53:56.808457+0000","last_became_active":"2024-03-26T03:50:45.588396+0000","last_became_peered":"2024-03-26T03:50:45.588396+0000","last_unstale":"2024-03-26T03:53:56.808457+0000","last_undegraded":"2024-03-26T03:53:56.808457+0000","last_fullsized":"2024-03-26T03:53:56.808457+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:17.128768+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.830917+0000","last_change":"2024-03-26T03:53:30.866289+0000","last_active":"2024-03-26T03:53:56.830917+0000","last_peered":"2024-03-26T03:53:56.830917+0000","last_clean":"2024-03-26T03:53:56.830917+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:53:56.830917+0000","last_undegraded":"2024-03-26T03:53:56.830917+0000","last_fullsized":"2024-03-26T03:53:56.830917+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.866047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.866047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.871487+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818479+0000","last_change":"2024-03-26T03:53:13.631462+0000","last_active":"2024-03-26T03:53:56.818479+0000","last_peered":"2024-03-26T03:53:56.818479+0000","last_clean":"2024-03-26T03:53:56.818479+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:53:56.818479+0000","last_undegraded":"2024-03-26T03:53:56.818479+0000","last_fullsized":"2024-03-26T03:53:56.818479+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.631351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.631351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:41.275139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":222,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819173+0000","last_change":"2024-03-26T03:52:45.404305+0000","last_active":"2024-03-26T03:53:56.819173+0000","last_peered":"2024-03-26T03:53:56.819173+0000","last_clean":"2024-03-26T03:53:56.819173+0000","last_became_active":"2024-03-26T03:48:46.023026+0000","last_became_peered":"2024-03-26T03:48:46.023026+0000","last_unstale":"2024-03-26T03:53:56.819173+0000","last_undegraded":"2024-03-26T03:53:56.819173+0000","last_fullsized":"2024-03-26T03:53:56.819173+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:45.403401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.403401+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.188496+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.829042+0000","last_change":"2024-03-26T03:51:38.222500+0000","last_active":"2024-03-26T03:53:56.829042+0000","last_peered":"2024-03-26T03:53:56.829042+0000","last_clean":"2024-03-26T03:53:56.829042+0000","last_became_active":"2024-03-26T03:50:45.591442+0000","last_became_peered":"2024-03-26T03:50:45.591442+0000","last_unstale":"2024-03-26T03:53:56.829042+0000","last_undegraded":"2024-03-26T03:53:56.829042+0000","last_fullsized":"2024-03-26T03:53:56.829042+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.222375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524323+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807965+0000","last_change":"2024-03-26T03:52:49.690105+0000","last_active":"2024-03-26T03:53:56.807965+0000","last_peered":"2024-03-26T03:53:56.807965+0000","last_clean":"2024-03-26T03:53:56.807965+0000","last_became_active":"2024-03-26T03:48:48.030427+0000","last_became_peered":"2024-03-26T03:48:48.030427+0000","last_unstale":"2024-03-26T03:53:56.807965+0000","last_undegraded":"2024-03-26T03:53:56.807965+0000","last_fullsized":"2024-03-26T03:53:56.807965+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:49.690025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.690025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.171459+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807596+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:53:56.807596+0000","last_peered":"2024-03-26T03:53:56.807596+0000","last_clean":"2024-03-26T03:53:56.807596+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:53:56.807596+0000","last_undegraded":"2024-03-26T03:53:56.807596+0000","last_fullsized":"2024-03-26T03:53:56.807596+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.802874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":224,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827896+0000","last_change":"2024-03-26T03:52:35.473032+0000","last_active":"2024-03-26T03:53:56.827896+0000","last_peered":"2024-03-26T03:53:56.827896+0000","last_clean":"2024-03-26T03:53:56.827896+0000","last_became_active":"2024-03-26T03:48:46.026751+0000","last_became_peered":"2024-03-26T03:48:46.026751+0000","last_unstale":"2024-03-26T03:53:56.827896+0000","last_undegraded":"2024-03-26T03:53:56.827896+0000","last_fullsized":"2024-03-26T03:53:56.827896+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.269829+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815653+0000","last_change":"2024-03-26T03:52:23.708646+0000","last_active":"2024-03-26T03:53:56.815653+0000","last_peered":"2024-03-26T03:53:56.815653+0000","last_clean":"2024-03-26T03:53:56.815653+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:53:56.815653+0000","last_undegraded":"2024-03-26T03:53:56.815653+0000","last_fullsized":"2024-03-26T03:53:56.815653+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:52:23.708510+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.708510+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808754+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:53:56.808754+0000","last_peered":"2024-03-26T03:53:56.808754+0000","last_clean":"2024-03-26T03:53:56.808754+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:53:56.808754+0000","last_undegraded":"2024-03-26T03:53:56.808754+0000","last_fullsized":"2024-03-26T03:53:56.808754+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484666+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825419+0000","last_change":"2024-03-26T03:53:13.785620+0000","last_active":"2024-03-26T03:53:56.825419+0000","last_peered":"2024-03-26T03:53:56.825419+0000","last_clean":"2024-03-26T03:53:56.825419+0000","last_became_active":"2024-03-26T03:48:44.017832+0000","last_became_peered":"2024-03-26T03:48:44.017832+0000","last_unstale":"2024-03-26T03:53:56.825419+0000","last_undegraded":"2024-03-26T03:53:56.825419+0000","last_fullsized":"2024-03-26T03:53:56.825419+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:13.785265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.785265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:16.531268+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":216,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.821847+0000","last_change":"2024-03-26T03:52:42.591753+0000","last_active":"2024-03-26T03:53:56.821847+0000","last_peered":"2024-03-26T03:53:56.821847+0000","last_clean":"2024-03-26T03:53:56.821847+0000","last_became_active":"2024-03-26T03:48:46.210377+0000","last_became_peered":"2024-03-26T03:48:46.210377+0000","last_unstale":"2024-03-26T03:53:56.821847+0000","last_undegraded":"2024-03-26T03:53:56.821847+0000","last_fullsized":"2024-03-26T03:53:56.821847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:52:42.591503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.591503+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.661591+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":180,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.830597+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:53:56.830597+0000","last_peered":"2024-03-26T03:53:56.830597+0000","last_clean":"2024-03-26T03:53:56.830597+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:53:56.830597+0000","last_undegraded":"2024-03-26T03:53:56.830597+0000","last_fullsized":"2024-03-26T03:53:56.830597+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:07.860041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822614+0000","last_change":"2024-03-26T03:51:33.548319+0000","last_active":"2024-03-26T03:53:56.822614+0000","last_peered":"2024-03-26T03:53:56.822614+0000","last_clean":"2024-03-26T03:53:56.822614+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:53:56.822614+0000","last_undegraded":"2024-03-26T03:53:56.822614+0000","last_fullsized":"2024-03-26T03:53:56.822614+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:51:33.548192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656936+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.824910+0000","last_change":"2024-03-26T03:51:35.260411+0000","last_active":"2024-03-26T03:53:56.824910+0000","last_peered":"2024-03-26T03:53:56.824910+0000","last_clean":"2024-03-26T03:53:56.824910+0000","last_became_active":"2024-03-26T03:48:44.015546+0000","last_became_peered":"2024-03-26T03:48:44.015546+0000","last_unstale":"2024-03-26T03:53:56.824910+0000","last_undegraded":"2024-03-26T03:53:56.824910+0000","last_fullsized":"2024-03-26T03:53:56.824910+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:35.258167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:35.258167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524225+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819803+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:53:56.819803+0000","last_peered":"2024-03-26T03:53:56.819803+0000","last_clean":"2024-03-26T03:53:56.819803+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:53:56.819803+0000","last_undegraded":"2024-03-26T03:53:56.819803+0000","last_fullsized":"2024-03-26T03:53:56.819803+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.838957+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.814902+0000","last_change":"2024-03-26T03:52:34.605587+0000","last_active":"2024-03-26T03:53:56.814902+0000","last_peered":"2024-03-26T03:53:56.814902+0000","last_clean":"2024-03-26T03:53:56.814902+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:53:56.814902+0000","last_undegraded":"2024-03-26T03:53:56.814902+0000","last_fullsized":"2024-03-26T03:53:56.814902+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:34.605509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:34.605509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":123,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807305+0000","last_change":"2024-03-26T03:51:39.355828+0000","last_active":"2024-03-26T03:53:56.807305+0000","last_peered":"2024-03-26T03:53:56.807305+0000","last_clean":"2024-03-26T03:53:56.807305+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:53:56.807305+0000","last_undegraded":"2024-03-26T03:53:56.807305+0000","last_fullsized":"2024-03-26T03:53:56.807305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:39.355728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:39.355728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484539+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826716+0000","last_change":"2024-03-26T03:52:23.578870+0000","last_active":"2024-03-26T03:53:56.826716+0000","last_peered":"2024-03-26T03:53:56.826716+0000","last_clean":"2024-03-26T03:53:56.826716+0000","last_became_active":"2024-03-26T03:48:44.011741+0000","last_became_peered":"2024-03-26T03:48:44.011741+0000","last_unstale":"2024-03-26T03:53:56.826716+0000","last_undegraded":"2024-03-26T03:53:56.826716+0000","last_fullsized":"2024-03-26T03:53:56.826716+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.657035+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":209,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825005+0000","last_change":"2024-03-26T03:53:32.271527+0000","last_active":"2024-03-26T03:53:56.825005+0000","last_peered":"2024-03-26T03:53:56.825005+0000","last_clean":"2024-03-26T03:53:56.825005+0000","last_became_active":"2024-03-26T03:50:45.587808+0000","last_became_peered":"2024-03-26T03:50:45.587808+0000","last_unstale":"2024-03-26T03:53:56.825005+0000","last_undegraded":"2024-03-26T03:53:56.825005+0000","last_fullsized":"2024-03-26T03:53:56.825005+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:48.643622+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827672+0000","last_change":"2024-03-26T03:53:17.352408+0000","last_active":"2024-03-26T03:53:56.827672+0000","last_peered":"2024-03-26T03:53:56.827672+0000","last_clean":"2024-03-26T03:53:56.827672+0000","last_became_active":"2024-03-26T03:50:45.589899+0000","last_became_peered":"2024-03-26T03:50:45.589899+0000","last_unstale":"2024-03-26T03:53:56.827672+0000","last_undegraded":"2024-03-26T03:53:56.827672+0000","last_fullsized":"2024-03-26T03:53:56.827672+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.352186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.352186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:30.380486+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815109+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:53:56.815109+0000","last_peered":"2024-03-26T03:53:56.815109+0000","last_clean":"2024-03-26T03:53:56.815109+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:53:56.815109+0000","last_undegraded":"2024-03-26T03:53:56.815109+0000","last_fullsized":"2024-03-26T03:53:56.815109+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.913379+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819589+0000","last_change":"2024-03-26T03:52:39.291143+0000","last_active":"2024-03-26T03:53:56.819589+0000","last_peered":"2024-03-26T03:53:56.819589+0000","last_clean":"2024-03-26T03:53:56.819589+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:53:56.819589+0000","last_undegraded":"2024-03-26T03:53:56.819589+0000","last_fullsized":"2024-03-26T03:53:56.819589+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:39.291063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.291063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":246,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822960+0000","last_change":"2024-03-26T03:53:19.344881+0000","last_active":"2024-03-26T03:53:56.822960+0000","last_peered":"2024-03-26T03:53:56.822960+0000","last_clean":"2024-03-26T03:53:56.822960+0000","last_became_active":"2024-03-26T03:50:45.595599+0000","last_became_peered":"2024-03-26T03:50:45.595599+0000","last_unstale":"2024-03-26T03:53:56.822960+0000","last_undegraded":"2024-03-26T03:53:56.822960+0000","last_fullsized":"2024-03-26T03:53:56.822960+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:25.336032+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815313+0000","last_change":"2024-03-26T03:52:26.596430+0000","last_active":"2024-03-26T03:53:56.815313+0000","last_peered":"2024-03-26T03:53:56.815313+0000","last_clean":"2024-03-26T03:53:56.815313+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:53:56.815313+0000","last_undegraded":"2024-03-26T03:53:56.815313+0000","last_fullsized":"2024-03-26T03:53:56.815313+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:26.596311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:26.596311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.657062+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827320+0000","last_change":"2024-03-26T03:52:40.410756+0000","last_active":"2024-03-26T03:53:56.827320+0000","last_peered":"2024-03-26T03:53:56.827320+0000","last_clean":"2024-03-26T03:53:56.827320+0000","last_became_active":"2024-03-26T03:48:48.031266+0000","last_became_peered":"2024-03-26T03:48:48.031266+0000","last_unstale":"2024-03-26T03:53:56.827320+0000","last_undegraded":"2024-03-26T03:53:56.827320+0000","last_fullsized":"2024-03-26T03:53:56.827320+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.947379+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815407+0000","last_change":"2024-03-26T03:53:24.672838+0000","last_active":"2024-03-26T03:53:56.815407+0000","last_peered":"2024-03-26T03:53:56.815407+0000","last_clean":"2024-03-26T03:53:56.815407+0000","last_became_active":"2024-03-26T03:48:44.019085+0000","last_became_peered":"2024-03-26T03:48:44.019085+0000","last_unstale":"2024-03-26T03:53:56.815407+0000","last_undegraded":"2024-03-26T03:53:56.815407+0000","last_fullsized":"2024-03-26T03:53:56.815407+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.120373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":220,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.823223+0000","last_change":"2024-03-26T03:51:29.988370+0000","last_active":"2024-03-26T03:53:56.823223+0000","last_peered":"2024-03-26T03:53:56.823223+0000","last_clean":"2024-03-26T03:53:56.823223+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:53:56.823223+0000","last_undegraded":"2024-03-26T03:53:56.823223+0000","last_fullsized":"2024-03-26T03:53:56.823223+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-03-26T03:51:29.988111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.988111+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.789876+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822833+0000","last_change":"2024-03-26T03:52:41.432452+0000","last_active":"2024-03-26T03:53:56.822833+0000","last_peered":"2024-03-26T03:53:56.822833+0000","last_clean":"2024-03-26T03:53:56.822833+0000","last_became_active":"2024-03-26T03:48:50.043494+0000","last_became_peered":"2024-03-26T03:48:50.043494+0000","last_unstale":"2024-03-26T03:53:56.822833+0000","last_undegraded":"2024-03-26T03:53:56.822833+0000","last_fullsized":"2024-03-26T03:53:56.822833+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:41.432332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:41.432332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524379+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822997+0000","last_change":"2024-03-26T03:52:40.835637+0000","last_active":"2024-03-26T03:53:56.822997+0000","last_peered":"2024-03-26T03:53:56.822997+0000","last_clean":"2024-03-26T03:53:56.822997+0000","last_became_active":"2024-03-26T03:48:48.030199+0000","last_became_peered":"2024-03-26T03:48:48.030199+0000","last_unstale":"2024-03-26T03:53:56.822997+0000","last_undegraded":"2024-03-26T03:53:56.822997+0000","last_fullsized":"2024-03-26T03:53:56.822997+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.791892+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807583+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:53:56.807583+0000","last_peered":"2024-03-26T03:53:56.807583+0000","last_clean":"2024-03-26T03:53:56.807583+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:53:56.807583+0000","last_undegraded":"2024-03-26T03:53:56.807583+0000","last_fullsized":"2024-03-26T03:53:56.807583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.223788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":226,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808061+0000","last_change":"2024-03-26T03:53:33.392149+0000","last_active":"2024-03-26T03:53:56.808061+0000","last_peered":"2024-03-26T03:53:56.808061+0000","last_clean":"2024-03-26T03:53:56.808061+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:53:56.808061+0000","last_undegraded":"2024-03-26T03:53:56.808061+0000","last_fullsized":"2024-03-26T03:53:56.808061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:51.222885+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.821616+0000","last_change":"2024-03-26T03:53:36.618514+0000","last_active":"2024-03-26T03:53:56.821616+0000","last_peered":"2024-03-26T03:53:56.821616+0000","last_clean":"2024-03-26T03:53:56.821616+0000","last_became_active":"2024-03-26T03:48:50.045853+0000","last_became_peered":"2024-03-26T03:48:50.045853+0000","last_unstale":"2024-03-26T03:53:56.821616+0000","last_undegraded":"2024-03-26T03:53:56.821616+0000","last_fullsized":"2024-03-26T03:53:56.821616+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:49.017884+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":265,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819720+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:53:56.819720+0000","last_peered":"2024-03-26T03:53:56.819720+0000","last_clean":"2024-03-26T03:53:56.819720+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:53:56.819720+0000","last_undegraded":"2024-03-26T03:53:56.819720+0000","last_fullsized":"2024-03-26T03:53:56.819720+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.768552+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815155+0000","last_change":"2024-03-26T03:53:20.739662+0000","last_active":"2024-03-26T03:53:56.815155+0000","last_peered":"2024-03-26T03:53:56.815155+0000","last_clean":"2024-03-26T03:53:56.815155+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:53:56.815155+0000","last_undegraded":"2024-03-26T03:53:56.815155+0000","last_fullsized":"2024-03-26T03:53:56.815155+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.739547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.739547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:25.034838+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":220,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826886+0000","last_change":"2024-03-26T03:51:42.190694+0000","last_active":"2024-03-26T03:53:56.826886+0000","last_peered":"2024-03-26T03:53:56.826886+0000","last_clean":"2024-03-26T03:53:56.826886+0000","last_became_active":"2024-03-26T03:48:46.209411+0000","last_became_peered":"2024-03-26T03:48:46.209411+0000","last_unstale":"2024-03-26T03:53:56.826886+0000","last_undegraded":"2024-03-26T03:53:56.826886+0000","last_fullsized":"2024-03-26T03:53:56.826886+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:51:42.190557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:42.190557+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524256+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808767+0000","last_change":"2024-03-26T03:51:34.417542+0000","last_active":"2024-03-26T03:53:56.808767+0000","last_peered":"2024-03-26T03:53:56.808767+0000","last_clean":"2024-03-26T03:53:56.808767+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:53:56.808767+0000","last_undegraded":"2024-03-26T03:53:56.808767+0000","last_fullsized":"2024-03-26T03:53:56.808767+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:34.417463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.417463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484456+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":249,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827271+0000","last_change":"2024-03-26T03:51:34.258348+0000","last_active":"2024-03-26T03:53:56.827271+0000","last_peered":"2024-03-26T03:53:56.827271+0000","last_clean":"2024-03-26T03:53:56.827271+0000","last_became_active":"2024-03-26T03:48:48.028697+0000","last_became_peered":"2024-03-26T03:48:48.028697+0000","last_unstale":"2024-03-26T03:53:56.827271+0000","last_undegraded":"2024-03-26T03:53:56.827271+0000","last_fullsized":"2024-03-26T03:53:56.827271+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.816158+0000","last_change":"2024-03-26T03:53:12.601302+0000","last_active":"2024-03-26T03:53:56.816158+0000","last_peered":"2024-03-26T03:53:56.816158+0000","last_clean":"2024-03-26T03:53:56.816158+0000","last_became_active":"2024-03-26T03:48:44.015207+0000","last_became_peered":"2024-03-26T03:48:44.015207+0000","last_unstale":"2024-03-26T03:53:56.816158+0000","last_undegraded":"2024-03-26T03:53:56.816158+0000","last_fullsized":"2024-03-26T03:53:56.816158+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:53:12.600988+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:12.600988+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:29.368784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819818+0000","last_change":"2024-03-26T03:53:27.547640+0000","last_active":"2024-03-26T03:53:56.819818+0000","last_peered":"2024-03-26T03:53:56.819818+0000","last_clean":"2024-03-26T03:53:56.819818+0000","last_became_active":"2024-03-26T03:48:46.210334+0000","last_became_peered":"2024-03-26T03:48:46.210334+0000","last_unstale":"2024-03-26T03:53:56.819818+0000","last_undegraded":"2024-03-26T03:53:56.819818+0000","last_fullsized":"2024-03-26T03:53:56.819818+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.015778+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819412+0000","last_change":"2024-03-26T03:53:18.332085+0000","last_active":"2024-03-26T03:53:56.819412+0000","last_peered":"2024-03-26T03:53:56.819412+0000","last_clean":"2024-03-26T03:53:56.819412+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:53:56.819412+0000","last_undegraded":"2024-03-26T03:53:56.819412+0000","last_fullsized":"2024-03-26T03:53:56.819412+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:53:18.331943+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.331943+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:26.743313+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.828895+0000","last_change":"2024-03-26T03:53:23.353045+0000","last_active":"2024-03-26T03:53:56.828895+0000","last_peered":"2024-03-26T03:53:56.828895+0000","last_clean":"2024-03-26T03:53:56.828895+0000","last_became_active":"2024-03-26T03:48:48.031771+0000","last_became_peered":"2024-03-26T03:48:48.031771+0000","last_unstale":"2024-03-26T03:53:56.828895+0000","last_undegraded":"2024-03-26T03:53:56.828895+0000","last_fullsized":"2024-03-26T03:53:56.828895+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:23.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:26.709176+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.814885+0000","last_change":"2024-03-26T03:53:29.687145+0000","last_active":"2024-03-26T03:53:56.814885+0000","last_peered":"2024-03-26T03:53:56.814885+0000","last_clean":"2024-03-26T03:53:56.814885+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:53:56.814885+0000","last_undegraded":"2024-03-26T03:53:56.814885+0000","last_fullsized":"2024-03-26T03:53:56.814885+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.687067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.687067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:33.606575+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":229,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822450+0000","last_change":"2024-03-26T03:52:37.650277+0000","last_active":"2024-03-26T03:53:56.822450+0000","last_peered":"2024-03-26T03:53:56.822450+0000","last_clean":"2024-03-26T03:53:56.822450+0000","last_became_active":"2024-03-26T03:48:46.209538+0000","last_became_peered":"2024-03-26T03:48:46.209538+0000","last_unstale":"2024-03-26T03:53:56.822450+0000","last_undegraded":"2024-03-26T03:53:56.822450+0000","last_fullsized":"2024-03-26T03:53:56.822450+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.102642+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815362+0000","last_change":"2024-03-26T03:53:17.657919+0000","last_active":"2024-03-26T03:53:56.815362+0000","last_peered":"2024-03-26T03:53:56.815362+0000","last_clean":"2024-03-26T03:53:56.815362+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:53:56.815362+0000","last_undegraded":"2024-03-26T03:53:56.815362+0000","last_fullsized":"2024-03-26T03:53:56.815362+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.376109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819656+0000","last_change":"2024-03-26T03:53:29.537877+0000","last_active":"2024-03-26T03:53:56.819656+0000","last_peered":"2024-03-26T03:53:56.819656+0000","last_clean":"2024-03-26T03:53:56.819656+0000","last_became_active":"2024-03-26T03:48:48.029308+0000","last_became_peered":"2024-03-26T03:48:48.029308+0000","last_unstale":"2024-03-26T03:53:56.819656+0000","last_undegraded":"2024-03-26T03:53:56.819656+0000","last_fullsized":"2024-03-26T03:53:56.819656+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:57.680160+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":123,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818334+0000","last_change":"2024-03-26T03:51:38.327055+0000","last_active":"2024-03-26T03:53:56.818334+0000","last_peered":"2024-03-26T03:53:56.818334+0000","last_clean":"2024-03-26T03:53:56.818334+0000","last_became_active":"2024-03-26T03:50:45.593133+0000","last_became_peered":"2024-03-26T03:50:45.593133+0000","last_unstale":"2024-03-26T03:53:56.818334+0000","last_undegraded":"2024-03-26T03:53:56.818334+0000","last_fullsized":"2024-03-26T03:53:56.818334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:38.326923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.326923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656814+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":227,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.829127+0000","last_change":"2024-03-26T03:52:33.466952+0000","last_active":"2024-03-26T03:53:56.829127+0000","last_peered":"2024-03-26T03:53:56.829127+0000","last_clean":"2024-03-26T03:53:56.829127+0000","last_became_active":"2024-03-26T03:50:24.531528+0000","last_became_peered":"2024-03-26T03:50:24.531528+0000","last_unstale":"2024-03-26T03:53:56.829127+0000","last_undegraded":"2024-03-26T03:53:56.829127+0000","last_fullsized":"2024-03-26T03:53:56.829127+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524409+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"148'12437","reported_seq":12621,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.369402+0000","last_change":"2024-03-26T03:52:44.145650+0000","last_active":"2024-03-26T03:53:59.369402+0000","last_peered":"2024-03-26T03:53:59.369402+0000","last_clean":"2024-03-26T03:53:59.369402+0000","last_became_active":"2024-03-26T03:50:45.595232+0000","last_became_peered":"2024-03-26T03:50:45.595232+0000","last_unstale":"2024-03-26T03:53:59.369402+0000","last_undegraded":"2024-03-26T03:53:59.369402+0000","last_fullsized":"2024-03-26T03:53:59.369402+0000","mapping_epoch":96,"log_start":"134'8200","ondisk_log_start":"134'8200","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"132'7919","last_scrub_stamp":"2024-03-26T03:52:44.145396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.145396+0000","objects_scrubbed":5918,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.921446+0000","scrub_duration":2302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12437,"num_write_kb":440960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819568+0000","last_change":"2024-03-26T03:52:50.408368+0000","last_active":"2024-03-26T03:53:56.819568+0000","last_peered":"2024-03-26T03:53:56.819568+0000","last_clean":"2024-03-26T03:53:56.819568+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:53:56.819568+0000","last_undegraded":"2024-03-26T03:53:56.819568+0000","last_fullsized":"2024-03-26T03:53:56.819568+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.408254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.408254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:02.251022+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":205,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815152+0000","last_change":"2024-03-26T03:52:32.622645+0000","last_active":"2024-03-26T03:53:56.815152+0000","last_peered":"2024-03-26T03:53:56.815152+0000","last_clean":"2024-03-26T03:53:56.815152+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:53:56.815152+0000","last_undegraded":"2024-03-26T03:53:56.815152+0000","last_fullsized":"2024-03-26T03:53:56.815152+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:52:32.622512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.622512+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":264,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819122+0000","last_change":"2024-03-26T03:52:22.438654+0000","last_active":"2024-03-26T03:53:56.819122+0000","last_peered":"2024-03-26T03:53:56.819122+0000","last_clean":"2024-03-26T03:53:56.819122+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:53:56.819122+0000","last_undegraded":"2024-03-26T03:53:56.819122+0000","last_fullsized":"2024-03-26T03:53:56.819122+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:52:22.438565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:22.438565+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.518566+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":275,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815159+0000","last_change":"2024-03-26T03:53:19.706639+0000","last_active":"2024-03-26T03:53:56.815159+0000","last_peered":"2024-03-26T03:53:56.815159+0000","last_clean":"2024-03-26T03:53:56.815159+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:53:56.815159+0000","last_undegraded":"2024-03-26T03:53:56.815159+0000","last_fullsized":"2024-03-26T03:53:56.815159+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:19.706557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.706557+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:24.796992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"148'12550","reported_seq":12667,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.523539+0000","last_change":"2024-03-26T03:50:44.592931+0000","last_active":"2024-03-26T03:53:59.523539+0000","last_peered":"2024-03-26T03:53:59.523539+0000","last_clean":"2024-03-26T03:53:59.523539+0000","last_became_active":"2024-03-26T03:50:44.592128+0000","last_became_peered":"2024-03-26T03:50:44.592128+0000","last_unstale":"2024-03-26T03:53:59.523539+0000","last_undegraded":"2024-03-26T03:53:59.523539+0000","last_fullsized":"2024-03-26T03:53:59.523539+0000","mapping_epoch":95,"log_start":"134'8300","ondisk_log_start":"134'8300","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91750400,"num_objects":1400,"num_object_clones":0,"num_object_copies":4200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12550,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827488+0000","last_change":"2024-03-26T03:51:23.414926+0000","last_active":"2024-03-26T03:53:56.827488+0000","last_peered":"2024-03-26T03:53:56.827488+0000","last_clean":"2024-03-26T03:53:56.827488+0000","last_became_active":"2024-03-26T03:48:50.043236+0000","last_became_peered":"2024-03-26T03:48:50.043236+0000","last_unstale":"2024-03-26T03:53:56.827488+0000","last_undegraded":"2024-03-26T03:53:56.827488+0000","last_fullsized":"2024-03-26T03:53:56.827488+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.414852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.414852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815904+0000","last_change":"2024-03-26T03:52:31.638104+0000","last_active":"2024-03-26T03:53:56.815904+0000","last_peered":"2024-03-26T03:53:56.815904+0000","last_clean":"2024-03-26T03:53:56.815904+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:53:56.815904+0000","last_undegraded":"2024-03-26T03:53:56.815904+0000","last_fullsized":"2024-03-26T03:53:56.815904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.637987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.637987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":236,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807644+0000","last_change":"2024-03-26T03:52:30.697090+0000","last_active":"2024-03-26T03:53:56.807644+0000","last_peered":"2024-03-26T03:53:56.807644+0000","last_clean":"2024-03-26T03:53:56.807644+0000","last_became_active":"2024-03-26T03:48:46.020974+0000","last_became_peered":"2024-03-26T03:48:46.020974+0000","last_unstale":"2024-03-26T03:53:56.807644+0000","last_undegraded":"2024-03-26T03:53:56.807644+0000","last_fullsized":"2024-03-26T03:53:56.807644+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484630+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815328+0000","last_change":"2024-03-26T03:53:19.352440+0000","last_active":"2024-03-26T03:53:56.815328+0000","last_peered":"2024-03-26T03:53:56.815328+0000","last_clean":"2024-03-26T03:53:56.815328+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:53:56.815328+0000","last_undegraded":"2024-03-26T03:53:56.815328+0000","last_fullsized":"2024-03-26T03:53:56.815328+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:19.352319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.352319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:45.533994+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"148'12995","reported_seq":13187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.587830+0000","last_change":"2024-03-26T03:52:36.221752+0000","last_active":"2024-03-26T03:53:59.587830+0000","last_peered":"2024-03-26T03:53:59.587830+0000","last_clean":"2024-03-26T03:53:59.587830+0000","last_became_active":"2024-03-26T03:50:44.585858+0000","last_became_peered":"2024-03-26T03:50:44.585858+0000","last_unstale":"2024-03-26T03:53:59.587830+0000","last_undegraded":"2024-03-26T03:53:59.587830+0000","last_fullsized":"2024-03-26T03:53:59.587830+0000","mapping_epoch":95,"log_start":"131'8200","ondisk_log_start":"131'8200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"130'7838","last_scrub_stamp":"2024-03-26T03:52:36.221537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:36.221537+0000","objects_scrubbed":6697,"log_size":4795,"log_dups_size":0,"ondisk_log_size":4795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.686425+0000","scrub_duration":2663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94830592,"num_objects":1447,"num_object_clones":0,"num_object_copies":4341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12995,"num_write_kb":462144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.814678+0000","last_change":"2024-03-26T03:53:36.664308+0000","last_active":"2024-03-26T03:53:56.814678+0000","last_peered":"2024-03-26T03:53:56.814678+0000","last_clean":"2024-03-26T03:53:56.814678+0000","last_became_active":"2024-03-26T03:48:50.033651+0000","last_became_peered":"2024-03-26T03:48:50.033651+0000","last_unstale":"2024-03-26T03:53:56.814678+0000","last_undegraded":"2024-03-26T03:53:56.814678+0000","last_fullsized":"2024-03-26T03:53:56.814678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.664206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.664206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:57.777780+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819887+0000","last_change":"2024-03-26T03:53:35.307506+0000","last_active":"2024-03-26T03:53:56.819887+0000","last_peered":"2024-03-26T03:53:56.819887+0000","last_clean":"2024-03-26T03:53:56.819887+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:53:56.819887+0000","last_undegraded":"2024-03-26T03:53:56.819887+0000","last_fullsized":"2024-03-26T03:53:56.819887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:35.307371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:35.307371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:37.812565+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815596+0000","last_change":"2024-03-26T03:52:35.600199+0000","last_active":"2024-03-26T03:53:56.815596+0000","last_peered":"2024-03-26T03:53:56.815596+0000","last_clean":"2024-03-26T03:53:56.815596+0000","last_became_active":"2024-03-26T03:50:45.594682+0000","last_became_peered":"2024-03-26T03:50:45.594682+0000","last_unstale":"2024-03-26T03:53:56.815596+0000","last_undegraded":"2024-03-26T03:53:56.815596+0000","last_fullsized":"2024-03-26T03:53:56.815596+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:35.600091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.600091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:57.681359+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.822844+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:53:56.822844+0000","last_peered":"2024-03-26T03:53:56.822844+0000","last_clean":"2024-03-26T03:53:56.822844+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:53:56.822844+0000","last_undegraded":"2024-03-26T03:53:56.822844+0000","last_fullsized":"2024-03-26T03:53:56.822844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.791858+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"148'12680","reported_seq":12872,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.474418+0000","last_change":"2024-03-26T03:52:33.625525+0000","last_active":"2024-03-26T03:53:59.474418+0000","last_peered":"2024-03-26T03:53:59.474418+0000","last_clean":"2024-03-26T03:53:59.474418+0000","last_became_active":"2024-03-26T03:50:44.589615+0000","last_became_peered":"2024-03-26T03:50:44.589615+0000","last_unstale":"2024-03-26T03:53:59.474418+0000","last_undegraded":"2024-03-26T03:53:59.474418+0000","last_fullsized":"2024-03-26T03:53:59.474418+0000","mapping_epoch":95,"log_start":"128'7200","ondisk_log_start":"128'7200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7504","last_scrub_stamp":"2024-03-26T03:52:33.625304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.625304+0000","objects_scrubbed":6650,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.188459+0000","scrub_duration":2355,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12680,"num_write_kb":450432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819510+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:53:56.819510+0000","last_peered":"2024-03-26T03:53:56.819510+0000","last_clean":"2024-03-26T03:53:56.819510+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:53:56.819510+0000","last_undegraded":"2024-03-26T03:53:56.819510+0000","last_fullsized":"2024-03-26T03:53:56.819510+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:59.605441+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818561+0000","last_change":"2024-03-26T03:52:42.382709+0000","last_active":"2024-03-26T03:53:56.818561+0000","last_peered":"2024-03-26T03:53:56.818561+0000","last_clean":"2024-03-26T03:53:56.818561+0000","last_became_active":"2024-03-26T03:48:48.029096+0000","last_became_peered":"2024-03-26T03:48:48.029096+0000","last_unstale":"2024-03-26T03:53:56.818561+0000","last_undegraded":"2024-03-26T03:53:56.818561+0000","last_fullsized":"2024-03-26T03:53:56.818561+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.382447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.382447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:08.004037+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819366+0000","last_change":"2024-03-26T03:52:31.320265+0000","last_active":"2024-03-26T03:53:56.819366+0000","last_peered":"2024-03-26T03:53:56.819366+0000","last_clean":"2024-03-26T03:53:56.819366+0000","last_became_active":"2024-03-26T03:50:45.592202+0000","last_became_peered":"2024-03-26T03:50:45.592202+0000","last_unstale":"2024-03-26T03:53:56.819366+0000","last_undegraded":"2024-03-26T03:53:56.819366+0000","last_fullsized":"2024-03-26T03:53:56.819366+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:31.320181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.320181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":235,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818691+0000","last_change":"2024-03-26T03:51:34.486120+0000","last_active":"2024-03-26T03:53:56.818691+0000","last_peered":"2024-03-26T03:53:56.818691+0000","last_clean":"2024-03-26T03:53:56.818691+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:53:56.818691+0000","last_undegraded":"2024-03-26T03:53:56.818691+0000","last_fullsized":"2024-03-26T03:53:56.818691+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:34.485979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.485979+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656881+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"148'12807","reported_seq":12990,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.632268+0000","last_change":"2024-03-26T03:52:48.148490+0000","last_active":"2024-03-26T03:53:59.632268+0000","last_peered":"2024-03-26T03:53:59.632268+0000","last_clean":"2024-03-26T03:53:59.632268+0000","last_became_active":"2024-03-26T03:50:45.590035+0000","last_became_peered":"2024-03-26T03:50:45.590035+0000","last_unstale":"2024-03-26T03:53:59.632268+0000","last_undegraded":"2024-03-26T03:53:59.632268+0000","last_fullsized":"2024-03-26T03:53:59.632268+0000","mapping_epoch":96,"log_start":"131'8000","ondisk_log_start":"131'8000","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8409","last_scrub_stamp":"2024-03-26T03:52:48.148171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.148171+0000","objects_scrubbed":5895,"log_size":4807,"log_dups_size":0,"ondisk_log_size":4807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.767336+0000","scrub_duration":3557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93126656,"num_objects":1421,"num_object_clones":0,"num_object_copies":4263,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1421,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12807,"num_write_kb":455296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.828363+0000","last_change":"2024-03-26T03:51:25.463769+0000","last_active":"2024-03-26T03:53:56.828363+0000","last_peered":"2024-03-26T03:53:56.828363+0000","last_clean":"2024-03-26T03:53:56.828363+0000","last_became_active":"2024-03-26T03:48:50.043467+0000","last_became_peered":"2024-03-26T03:48:50.043467+0000","last_unstale":"2024-03-26T03:53:56.828363+0000","last_undegraded":"2024-03-26T03:53:56.828363+0000","last_fullsized":"2024-03-26T03:53:56.828363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.463664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.463664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524144+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":222,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819310+0000","last_change":"2024-03-26T03:53:37.581465+0000","last_active":"2024-03-26T03:53:56.819310+0000","last_peered":"2024-03-26T03:53:56.819310+0000","last_clean":"2024-03-26T03:53:56.819310+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:53:56.819310+0000","last_undegraded":"2024-03-26T03:53:56.819310+0000","last_fullsized":"2024-03-26T03:53:56.819310+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.527591+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807715+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:53:56.807715+0000","last_peered":"2024-03-26T03:53:56.807715+0000","last_clean":"2024-03-26T03:53:56.807715+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:53:56.807715+0000","last_undegraded":"2024-03-26T03:53:56.807715+0000","last_fullsized":"2024-03-26T03:53:56.807715+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"73'34","reported_seq":312,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.824987+0000","last_change":"2024-03-26T03:51:34.973521+0000","last_active":"2024-03-26T03:53:56.824987+0000","last_peered":"2024-03-26T03:53:56.824987+0000","last_clean":"2024-03-26T03:53:56.824987+0000","last_became_active":"2024-03-26T03:48:46.026747+0000","last_became_peered":"2024-03-26T03:48:46.026747+0000","last_unstale":"2024-03-26T03:53:56.824987+0000","last_undegraded":"2024-03-26T03:53:56.824987+0000","last_fullsized":"2024-03-26T03:53:56.824987+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:51:34.973366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.973366+0000","objects_scrubbed":11,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.789817+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.816878+0000","last_change":"2024-03-26T03:52:25.587745+0000","last_active":"2024-03-26T03:53:56.816878+0000","last_peered":"2024-03-26T03:53:56.816878+0000","last_clean":"2024-03-26T03:53:56.816878+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:53:56.816878+0000","last_undegraded":"2024-03-26T03:53:56.816878+0000","last_fullsized":"2024-03-26T03:53:56.816878+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.587502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.587502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.657007+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"148'12698","reported_seq":12888,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.610149+0000","last_change":"2024-03-26T03:52:33.050130+0000","last_active":"2024-03-26T03:53:59.610149+0000","last_peered":"2024-03-26T03:53:59.610149+0000","last_clean":"2024-03-26T03:53:59.610149+0000","last_became_active":"2024-03-26T03:50:44.584865+0000","last_became_peered":"2024-03-26T03:50:44.584865+0000","last_unstale":"2024-03-26T03:53:59.610149+0000","last_undegraded":"2024-03-26T03:53:59.610149+0000","last_fullsized":"2024-03-26T03:53:59.610149+0000","mapping_epoch":95,"log_start":"131'7900","ondisk_log_start":"131'7900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7431","last_scrub_stamp":"2024-03-26T03:52:33.049847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.049847+0000","objects_scrubbed":6680,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:03.693912+0000","scrub_duration":2449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88735744,"num_objects":1354,"num_object_clones":0,"num_object_copies":4062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12698,"num_write_kb":449664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815726+0000","last_change":"2024-03-26T03:52:44.692881+0000","last_active":"2024-03-26T03:53:56.815726+0000","last_peered":"2024-03-26T03:53:56.815726+0000","last_clean":"2024-03-26T03:53:56.815726+0000","last_became_active":"2024-03-26T03:50:45.583936+0000","last_became_peered":"2024-03-26T03:50:45.583936+0000","last_unstale":"2024-03-26T03:53:56.815726+0000","last_undegraded":"2024-03-26T03:53:56.815726+0000","last_fullsized":"2024-03-26T03:53:56.815726+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.026793+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.828352+0000","last_change":"2024-03-26T03:53:34.835708+0000","last_active":"2024-03-26T03:53:56.828352+0000","last_peered":"2024-03-26T03:53:56.828352+0000","last_clean":"2024-03-26T03:53:56.828352+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:53:56.828352+0000","last_undegraded":"2024-03-26T03:53:56.828352+0000","last_fullsized":"2024-03-26T03:53:56.828352+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.835581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.835581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:03.222485+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":210,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.817118+0000","last_change":"2024-03-26T03:52:40.285553+0000","last_active":"2024-03-26T03:53:56.817118+0000","last_peered":"2024-03-26T03:53:56.817118+0000","last_clean":"2024-03-26T03:53:56.817118+0000","last_became_active":"2024-03-26T03:48:43.011433+0000","last_became_peered":"2024-03-26T03:48:43.011433+0000","last_unstale":"2024-03-26T03:53:56.817118+0000","last_undegraded":"2024-03-26T03:53:56.817118+0000","last_fullsized":"2024-03-26T03:53:56.817118+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.285400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.285400+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.875607+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":227,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826767+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:53:56.826767+0000","last_peered":"2024-03-26T03:53:56.826767+0000","last_clean":"2024-03-26T03:53:56.826767+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:53:56.826767+0000","last_undegraded":"2024-03-26T03:53:56.826767+0000","last_fullsized":"2024-03-26T03:53:56.826767+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.791949+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.817344+0000","last_change":"2024-03-26T03:53:32.388419+0000","last_active":"2024-03-26T03:53:56.817344+0000","last_peered":"2024-03-26T03:53:56.817344+0000","last_clean":"2024-03-26T03:53:56.817344+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:53:56.817344+0000","last_undegraded":"2024-03-26T03:53:56.817344+0000","last_fullsized":"2024-03-26T03:53:56.817344+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.387809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.387809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.271413+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"148'12512","reported_seq":12693,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.611314+0000","last_change":"2024-03-26T03:52:49.471047+0000","last_active":"2024-03-26T03:53:59.611314+0000","last_peered":"2024-03-26T03:53:59.611314+0000","last_clean":"2024-03-26T03:53:59.611314+0000","last_became_active":"2024-03-26T03:50:44.588356+0000","last_became_peered":"2024-03-26T03:50:44.588356+0000","last_unstale":"2024-03-26T03:53:59.611314+0000","last_undegraded":"2024-03-26T03:53:59.611314+0000","last_fullsized":"2024-03-26T03:53:59.611314+0000","mapping_epoch":95,"log_start":"131'7700","ondisk_log_start":"131'7700","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"134'8243","last_scrub_stamp":"2024-03-26T03:52:49.470707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.470707+0000","objects_scrubbed":5693,"log_size":4812,"log_dups_size":0,"ondisk_log_size":4812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:11.948331+0000","scrub_duration":2903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12512,"num_write_kb":443328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819648+0000","last_change":"2024-03-26T03:53:25.552202+0000","last_active":"2024-03-26T03:53:56.819648+0000","last_peered":"2024-03-26T03:53:56.819648+0000","last_clean":"2024-03-26T03:53:56.819648+0000","last_became_active":"2024-03-26T03:48:50.039606+0000","last_became_peered":"2024-03-26T03:48:50.039606+0000","last_unstale":"2024-03-26T03:53:56.819648+0000","last_undegraded":"2024-03-26T03:53:56.819648+0000","last_fullsized":"2024-03-26T03:53:56.819648+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:50.318624+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":456,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.814881+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:53:56.814881+0000","last_peered":"2024-03-26T03:53:56.814881+0000","last_clean":"2024-03-26T03:53:56.814881+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:53:56.814881+0000","last_undegraded":"2024-03-26T03:53:56.814881+0000","last_fullsized":"2024-03-26T03:53:56.814881+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:14.983642+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827137+0000","last_change":"2024-03-26T03:51:18.413053+0000","last_active":"2024-03-26T03:53:56.827137+0000","last_peered":"2024-03-26T03:53:56.827137+0000","last_clean":"2024-03-26T03:53:56.827137+0000","last_became_active":"2024-03-26T03:50:45.592343+0000","last_became_peered":"2024-03-26T03:50:45.592343+0000","last_unstale":"2024-03-26T03:53:56.827137+0000","last_undegraded":"2024-03-26T03:53:56.827137+0000","last_fullsized":"2024-03-26T03:53:56.827137+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:18.412940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:18.412940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524031+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808240+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:53:56.808240+0000","last_peered":"2024-03-26T03:53:56.808240+0000","last_clean":"2024-03-26T03:53:56.808240+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:53:56.808240+0000","last_undegraded":"2024-03-26T03:53:56.808240+0000","last_fullsized":"2024-03-26T03:53:56.808240+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484648+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815654+0000","last_change":"2024-03-26T03:53:28.678428+0000","last_active":"2024-03-26T03:53:56.815654+0000","last_peered":"2024-03-26T03:53:56.815654+0000","last_clean":"2024-03-26T03:53:56.815654+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:53:56.815654+0000","last_undegraded":"2024-03-26T03:53:56.815654+0000","last_fullsized":"2024-03-26T03:53:56.815654+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:35.290355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"148'12850","reported_seq":13015,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.195881+0000","last_change":"2024-03-26T03:53:19.691120+0000","last_active":"2024-03-26T03:53:59.195881+0000","last_peered":"2024-03-26T03:53:59.195881+0000","last_clean":"2024-03-26T03:53:59.195881+0000","last_became_active":"2024-03-26T03:50:44.586504+0000","last_became_peered":"2024-03-26T03:50:44.586504+0000","last_unstale":"2024-03-26T03:53:59.195881+0000","last_undegraded":"2024-03-26T03:53:59.195881+0000","last_fullsized":"2024-03-26T03:53:59.195881+0000","mapping_epoch":95,"log_start":"130'7800","ondisk_log_start":"130'7800","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10430","last_scrub_stamp":"2024-03-26T03:53:19.691027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.691027+0000","objects_scrubbed":3868,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:22.357406+0000","scrub_duration":1022,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93192192,"num_objects":1422,"num_object_clones":0,"num_object_copies":4266,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1422,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12850,"num_write_kb":456704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":180,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.824623+0000","last_change":"2024-03-26T03:53:20.692581+0000","last_active":"2024-03-26T03:53:56.824623+0000","last_peered":"2024-03-26T03:53:56.824623+0000","last_clean":"2024-03-26T03:53:56.824623+0000","last_became_active":"2024-03-26T03:48:50.043374+0000","last_became_peered":"2024-03-26T03:48:50.043374+0000","last_unstale":"2024-03-26T03:53:56.824623+0000","last_undegraded":"2024-03-26T03:53:56.824623+0000","last_fullsized":"2024-03-26T03:53:56.824623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:21.044199+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819235+0000","last_change":"2024-03-26T03:53:33.645572+0000","last_active":"2024-03-26T03:53:56.819235+0000","last_peered":"2024-03-26T03:53:56.819235+0000","last_clean":"2024-03-26T03:53:56.819235+0000","last_became_active":"2024-03-26T03:48:44.011223+0000","last_became_peered":"2024-03-26T03:48:44.011223+0000","last_unstale":"2024-03-26T03:53:56.819235+0000","last_undegraded":"2024-03-26T03:53:56.819235+0000","last_fullsized":"2024-03-26T03:53:56.819235+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:33.645447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.645447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:50.785544+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":521,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:58.180177+0000","last_change":"2024-03-26T03:53:21.716954+0000","last_active":"2024-03-26T03:53:58.180177+0000","last_peered":"2024-03-26T03:53:58.180177+0000","last_clean":"2024-03-26T03:53:58.180177+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:53:58.180177+0000","last_undegraded":"2024-03-26T03:53:58.180177+0000","last_fullsized":"2024-03-26T03:53:58.180177+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:53:21.716822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:21.716822+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:29.626939+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.821728+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:53:56.821728+0000","last_peered":"2024-03-26T03:53:56.821728+0000","last_clean":"2024-03-26T03:53:56.821728+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:53:56.821728+0000","last_undegraded":"2024-03-26T03:53:56.821728+0000","last_fullsized":"2024-03-26T03:53:56.821728+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:56.887919+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819905+0000","last_change":"2024-03-26T03:52:25.471379+0000","last_active":"2024-03-26T03:53:56.819905+0000","last_peered":"2024-03-26T03:53:56.819905+0000","last_clean":"2024-03-26T03:53:56.819905+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:53:56.819905+0000","last_undegraded":"2024-03-26T03:53:56.819905+0000","last_fullsized":"2024-03-26T03:53:56.819905+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:25.471302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:25.471302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"148'12793","reported_seq":12987,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.177342+0000","last_change":"2024-03-26T03:52:31.921264+0000","last_active":"2024-03-26T03:53:59.177342+0000","last_peered":"2024-03-26T03:53:59.177342+0000","last_clean":"2024-03-26T03:53:59.177342+0000","last_became_active":"2024-03-26T03:50:44.593854+0000","last_became_peered":"2024-03-26T03:50:44.593854+0000","last_unstale":"2024-03-26T03:53:59.177342+0000","last_undegraded":"2024-03-26T03:53:59.177342+0000","last_fullsized":"2024-03-26T03:53:59.177342+0000","mapping_epoch":95,"log_start":"128'7400","ondisk_log_start":"128'7400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"129'7468","last_scrub_stamp":"2024-03-26T03:52:31.921157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.921157+0000","objects_scrubbed":6796,"log_size":5393,"log_dups_size":0,"ondisk_log_size":5393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93913088,"num_objects":1433,"num_object_clones":0,"num_object_copies":4299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12793,"num_write_kb":455232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825601+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:53:56.825601+0000","last_peered":"2024-03-26T03:53:56.825601+0000","last_clean":"2024-03-26T03:53:56.825601+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:53:56.825601+0000","last_undegraded":"2024-03-26T03:53:56.825601+0000","last_fullsized":"2024-03-26T03:53:56.825601+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.791919+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":200,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.824855+0000","last_change":"2024-03-26T03:53:31.824781+0000","last_active":"2024-03-26T03:53:56.824855+0000","last_peered":"2024-03-26T03:53:56.824855+0000","last_clean":"2024-03-26T03:53:56.824855+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:53:56.824855+0000","last_undegraded":"2024-03-26T03:53:56.824855+0000","last_fullsized":"2024-03-26T03:53:56.824855+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:34.207903+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":237,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.818560+0000","last_change":"2024-03-26T03:51:32.540653+0000","last_active":"2024-03-26T03:53:56.818560+0000","last_peered":"2024-03-26T03:53:56.818560+0000","last_clean":"2024-03-26T03:53:56.818560+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:53:56.818560+0000","last_undegraded":"2024-03-26T03:53:56.818560+0000","last_fullsized":"2024-03-26T03:53:56.818560+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:51:32.540268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.540268+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656968+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":123,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.816068+0000","last_change":"2024-03-26T03:51:41.399750+0000","last_active":"2024-03-26T03:53:56.816068+0000","last_peered":"2024-03-26T03:53:56.816068+0000","last_clean":"2024-03-26T03:53:56.816068+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:53:56.816068+0000","last_undegraded":"2024-03-26T03:53:56.816068+0000","last_fullsized":"2024-03-26T03:53:56.816068+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.399622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.399622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.656909+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"148'12670","reported_seq":12830,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.185084+0000","last_change":"2024-03-26T03:53:23.269397+0000","last_active":"2024-03-26T03:53:59.185084+0000","last_peered":"2024-03-26T03:53:59.185084+0000","last_clean":"2024-03-26T03:53:59.185084+0000","last_became_active":"2024-03-26T03:50:45.596978+0000","last_became_peered":"2024-03-26T03:50:45.596978+0000","last_unstale":"2024-03-26T03:53:59.185084+0000","last_undegraded":"2024-03-26T03:53:59.185084+0000","last_fullsized":"2024-03-26T03:53:59.185084+0000","mapping_epoch":96,"log_start":"132'8100","ondisk_log_start":"132'8100","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"139'10476","last_scrub_stamp":"2024-03-26T03:53:23.269247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.269247+0000","objects_scrubbed":3611,"log_size":4570,"log_dups_size":0,"ondisk_log_size":4570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:37.151488+0000","scrub_duration":790,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91226112,"num_objects":1392,"num_object_clones":0,"num_object_copies":4176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1392,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815410+0000","last_change":"2024-03-26T03:53:16.573026+0000","last_active":"2024-03-26T03:53:56.815410+0000","last_peered":"2024-03-26T03:53:56.815410+0000","last_clean":"2024-03-26T03:53:56.815410+0000","last_became_active":"2024-03-26T03:48:50.044967+0000","last_became_peered":"2024-03-26T03:48:50.044967+0000","last_unstale":"2024-03-26T03:53:56.815410+0000","last_undegraded":"2024-03-26T03:53:56.815410+0000","last_fullsized":"2024-03-26T03:53:56.815410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:32.584061+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819723+0000","last_change":"2024-03-26T03:52:29.399095+0000","last_active":"2024-03-26T03:53:56.819723+0000","last_peered":"2024-03-26T03:53:56.819723+0000","last_clean":"2024-03-26T03:53:56.819723+0000","last_became_active":"2024-03-26T03:48:44.014667+0000","last_became_peered":"2024-03-26T03:48:44.014667+0000","last_unstale":"2024-03-26T03:53:56.819723+0000","last_undegraded":"2024-03-26T03:53:56.819723+0000","last_fullsized":"2024-03-26T03:53:56.819723+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:29.398965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:29.398965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":215,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.823080+0000","last_change":"2024-03-26T03:51:17.091083+0000","last_active":"2024-03-26T03:53:56.823080+0000","last_peered":"2024-03-26T03:53:56.823080+0000","last_clean":"2024-03-26T03:53:56.823080+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:53:56.823080+0000","last_undegraded":"2024-03-26T03:53:56.823080+0000","last_fullsized":"2024-03-26T03:53:56.823080+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:51:17.090921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:17.090921+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.789780+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"100'9","reported_seq":267,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.816583+0000","last_change":"2024-03-26T03:53:29.559475+0000","last_active":"2024-03-26T03:53:56.816583+0000","last_peered":"2024-03-26T03:53:56.816583+0000","last_clean":"2024-03-26T03:53:56.816583+0000","last_became_active":"2024-03-26T03:50:45.602990+0000","last_became_peered":"2024-03-26T03:50:45.602990+0000","last_unstale":"2024-03-26T03:53:56.816583+0000","last_undegraded":"2024-03-26T03:53:56.816583+0000","last_fullsized":"2024-03-26T03:53:56.816583+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:49.863675+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"148'12407","reported_seq":12566,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.534726+0000","last_change":"2024-03-26T03:53:28.940164+0000","last_active":"2024-03-26T03:53:59.534726+0000","last_peered":"2024-03-26T03:53:59.534726+0000","last_clean":"2024-03-26T03:53:59.534726+0000","last_became_active":"2024-03-26T03:50:44.586424+0000","last_became_peered":"2024-03-26T03:50:44.586424+0000","last_unstale":"2024-03-26T03:53:59.534726+0000","last_undegraded":"2024-03-26T03:53:59.534726+0000","last_fullsized":"2024-03-26T03:53:59.534726+0000","mapping_epoch":95,"log_start":"134'8200","ondisk_log_start":"134'8200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10589","last_scrub_stamp":"2024-03-26T03:53:28.939404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.939404+0000","objects_scrubbed":3267,"log_size":4207,"log_dups_size":0,"ondisk_log_size":4207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.880836+0000","scrub_duration":1668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12407,"num_write_kb":441792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815464+0000","last_change":"2024-03-26T03:53:34.666684+0000","last_active":"2024-03-26T03:53:56.815464+0000","last_peered":"2024-03-26T03:53:56.815464+0000","last_clean":"2024-03-26T03:53:56.815464+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:53:56.815464+0000","last_undegraded":"2024-03-26T03:53:56.815464+0000","last_fullsized":"2024-03-26T03:53:56.815464+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.666607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.666607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:51.753812+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":202,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.821736+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:53:56.821736+0000","last_peered":"2024-03-26T03:53:56.821736+0000","last_clean":"2024-03-26T03:53:56.821736+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:53:56.821736+0000","last_undegraded":"2024-03-26T03:53:56.821736+0000","last_fullsized":"2024-03-26T03:53:56.821736+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:48.854398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":246,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807589+0000","last_change":"2024-03-26T03:51:29.300134+0000","last_active":"2024-03-26T03:53:56.807589+0000","last_peered":"2024-03-26T03:53:56.807589+0000","last_clean":"2024-03-26T03:53:56.807589+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:53:56.807589+0000","last_undegraded":"2024-03-26T03:53:56.807589+0000","last_fullsized":"2024-03-26T03:53:56.807589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:51:29.299953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:29.299953+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484474+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":257,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825263+0000","last_change":"2024-03-26T03:52:31.483741+0000","last_active":"2024-03-26T03:53:56.825263+0000","last_peered":"2024-03-26T03:53:56.825263+0000","last_clean":"2024-03-26T03:53:56.825263+0000","last_became_active":"2024-03-26T03:48:48.028257+0000","last_became_peered":"2024-03-26T03:48:48.028257+0000","last_unstale":"2024-03-26T03:53:56.825263+0000","last_undegraded":"2024-03-26T03:53:56.825263+0000","last_fullsized":"2024-03-26T03:53:56.825263+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:31.483593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.483593+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"148'12647","reported_seq":12802,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.533694+0000","last_change":"2024-03-26T03:53:34.507595+0000","last_active":"2024-03-26T03:53:59.533694+0000","last_peered":"2024-03-26T03:53:59.533694+0000","last_clean":"2024-03-26T03:53:59.533694+0000","last_became_active":"2024-03-26T03:50:44.592595+0000","last_became_peered":"2024-03-26T03:50:44.592595+0000","last_unstale":"2024-03-26T03:53:59.533694+0000","last_undegraded":"2024-03-26T03:53:59.533694+0000","last_fullsized":"2024-03-26T03:53:59.533694+0000","mapping_epoch":95,"log_start":"134'8400","ondisk_log_start":"134'8400","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"141'11108","last_scrub_stamp":"2024-03-26T03:53:34.507336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.507336+0000","objects_scrubbed":2955,"log_size":4247,"log_dups_size":0,"ondisk_log_size":4247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:03.308769+0000","scrub_duration":1304,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90112000,"num_objects":1375,"num_object_clones":0,"num_object_copies":4125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12647,"num_write_kb":448704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":173,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807470+0000","last_change":"2024-03-26T03:51:21.386995+0000","last_active":"2024-03-26T03:53:56.807470+0000","last_peered":"2024-03-26T03:53:56.807470+0000","last_clean":"2024-03-26T03:53:56.807470+0000","last_became_active":"2024-03-26T03:48:50.043453+0000","last_became_peered":"2024-03-26T03:48:50.043453+0000","last_unstale":"2024-03-26T03:53:56.807470+0000","last_undegraded":"2024-03-26T03:53:56.807470+0000","last_fullsized":"2024-03-26T03:53:56.807470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:21.386911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:21.386911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484491+0000","scrub_duration":164,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.828696+0000","last_change":"2024-03-26T03:52:45.556559+0000","last_active":"2024-03-26T03:53:56.828696+0000","last_peered":"2024-03-26T03:53:56.828696+0000","last_clean":"2024-03-26T03:53:56.828696+0000","last_became_active":"2024-03-26T03:48:44.013269+0000","last_became_peered":"2024-03-26T03:48:44.013269+0000","last_unstale":"2024-03-26T03:53:56.828696+0000","last_undegraded":"2024-03-26T03:53:56.828696+0000","last_fullsized":"2024-03-26T03:53:56.828696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.193240+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":124,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.815220+0000","last_change":"2024-03-26T03:52:46.432028+0000","last_active":"2024-03-26T03:53:56.815220+0000","last_peered":"2024-03-26T03:53:56.815220+0000","last_clean":"2024-03-26T03:53:56.815220+0000","last_became_active":"2024-03-26T03:51:24.676652+0000","last_became_peered":"2024-03-26T03:51:24.676652+0000","last_unstale":"2024-03-26T03:53:56.815220+0000","last_undegraded":"2024-03-26T03:53:56.815220+0000","last_fullsized":"2024-03-26T03:53:56.815220+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:52:46.429387+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.429387+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.188512+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807350+0000","last_change":"2024-03-26T03:52:37.885150+0000","last_active":"2024-03-26T03:53:56.807350+0000","last_peered":"2024-03-26T03:53:56.807350+0000","last_clean":"2024-03-26T03:53:56.807350+0000","last_became_active":"2024-03-26T03:48:48.029207+0000","last_became_peered":"2024-03-26T03:48:48.029207+0000","last_unstale":"2024-03-26T03:53:56.807350+0000","last_undegraded":"2024-03-26T03:53:56.807350+0000","last_fullsized":"2024-03-26T03:53:56.807350+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484682+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"148'12588","reported_seq":12703,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.548938+0000","last_change":"2024-03-26T03:50:45.583826+0000","last_active":"2024-03-26T03:53:59.548938+0000","last_peered":"2024-03-26T03:53:59.548938+0000","last_clean":"2024-03-26T03:53:59.548938+0000","last_became_active":"2024-03-26T03:50:45.583350+0000","last_became_peered":"2024-03-26T03:50:45.583350+0000","last_unstale":"2024-03-26T03:53:59.548938+0000","last_undegraded":"2024-03-26T03:53:59.548938+0000","last_fullsized":"2024-03-26T03:53:59.548938+0000","mapping_epoch":96,"log_start":"134'8400","ondisk_log_start":"134'8400","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:50:43.560190+0000","objects_scrubbed":0,"log_size":4188,"log_dups_size":0,"ondisk_log_size":4188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87425024,"num_objects":1334,"num_object_clones":0,"num_object_copies":4002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12588,"num_write_kb":445504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819483+0000","last_change":"2024-03-26T03:53:28.635973+0000","last_active":"2024-03-26T03:53:56.819483+0000","last_peered":"2024-03-26T03:53:56.819483+0000","last_clean":"2024-03-26T03:53:56.819483+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:53:56.819483+0000","last_undegraded":"2024-03-26T03:53:56.819483+0000","last_fullsized":"2024-03-26T03:53:56.819483+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:36.514232+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":257,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808566+0000","last_change":"2024-03-26T03:52:20.706812+0000","last_active":"2024-03-26T03:53:56.808566+0000","last_peered":"2024-03-26T03:53:56.808566+0000","last_clean":"2024-03-26T03:53:56.808566+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:53:56.808566+0000","last_undegraded":"2024-03-26T03:53:56.808566+0000","last_fullsized":"2024-03-26T03:53:56.808566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:20.706714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:20.706714+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484612+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825146+0000","last_change":"2024-03-26T03:51:26.426050+0000","last_active":"2024-03-26T03:53:56.825146+0000","last_peered":"2024-03-26T03:53:56.825146+0000","last_clean":"2024-03-26T03:53:56.825146+0000","last_became_active":"2024-03-26T03:48:44.016013+0000","last_became_peered":"2024-03-26T03:48:44.016013+0000","last_unstale":"2024-03-26T03:53:56.825146+0000","last_undegraded":"2024-03-26T03:53:56.825146+0000","last_fullsized":"2024-03-26T03:53:56.825146+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:26.425925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:26.425925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524198+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826670+0000","last_change":"2024-03-26T03:51:38.907371+0000","last_active":"2024-03-26T03:53:56.826670+0000","last_peered":"2024-03-26T03:53:56.826670+0000","last_clean":"2024-03-26T03:53:56.826670+0000","last_became_active":"2024-03-26T03:48:46.026227+0000","last_became_peered":"2024-03-26T03:48:46.026227+0000","last_unstale":"2024-03-26T03:53:56.826670+0000","last_undegraded":"2024-03-26T03:53:56.826670+0000","last_fullsized":"2024-03-26T03:53:56.826670+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:51:38.907089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:38.907089+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.789847+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"148'12492","reported_seq":12647,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.410174+0000","last_change":"2024-03-26T03:53:31.080444+0000","last_active":"2024-03-26T03:53:59.410174+0000","last_peered":"2024-03-26T03:53:59.410174+0000","last_clean":"2024-03-26T03:53:59.410174+0000","last_became_active":"2024-03-26T03:50:45.591306+0000","last_became_peered":"2024-03-26T03:50:45.591306+0000","last_unstale":"2024-03-26T03:53:59.410174+0000","last_undegraded":"2024-03-26T03:53:59.410174+0000","last_fullsized":"2024-03-26T03:53:59.410174+0000","mapping_epoch":96,"log_start":"134'8300","ondisk_log_start":"134'8300","created":95,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'10751","last_scrub_stamp":"2024-03-26T03:53:31.080052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.080052+0000","objects_scrubbed":3147,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.105203+0000","scrub_duration":1283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89849880,"num_objects":1372,"num_object_clones":0,"num_object_copies":4116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1372,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12492,"num_write_kb":443585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.826751+0000","last_change":"2024-03-26T03:51:19.390232+0000","last_active":"2024-03-26T03:53:56.826751+0000","last_peered":"2024-03-26T03:53:56.826751+0000","last_clean":"2024-03-26T03:53:56.826751+0000","last_became_active":"2024-03-26T03:50:45.595009+0000","last_became_peered":"2024-03-26T03:50:45.595009+0000","last_unstale":"2024-03-26T03:53:56.826751+0000","last_undegraded":"2024-03-26T03:53:56.826751+0000","last_fullsized":"2024-03-26T03:53:56.826751+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:19.390111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:19.390111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524058+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.819820+0000","last_change":"2024-03-26T03:52:42.284044+0000","last_active":"2024-03-26T03:53:56.819820+0000","last_peered":"2024-03-26T03:53:56.819820+0000","last_clean":"2024-03-26T03:53:56.819820+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:53:56.819820+0000","last_undegraded":"2024-03-26T03:53:56.819820+0000","last_fullsized":"2024-03-26T03:53:56.819820+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.283958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":193,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807416+0000","last_change":"2024-03-26T03:51:25.157523+0000","last_active":"2024-03-26T03:53:56.807416+0000","last_peered":"2024-03-26T03:53:56.807416+0000","last_clean":"2024-03-26T03:53:56.807416+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:53:56.807416+0000","last_undegraded":"2024-03-26T03:53:56.807416+0000","last_fullsized":"2024-03-26T03:53:56.807416+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:25.157414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:51:25.157414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484570+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":248,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808367+0000","last_change":"2024-03-26T03:53:37.345947+0000","last_active":"2024-03-26T03:53:56.808367+0000","last_peered":"2024-03-26T03:53:56.808367+0000","last_clean":"2024-03-26T03:53:56.808367+0000","last_became_active":"2024-03-26T03:50:45.596131+0000","last_became_peered":"2024-03-26T03:50:45.596131+0000","last_unstale":"2024-03-26T03:53:56.808367+0000","last_undegraded":"2024-03-26T03:53:56.808367+0000","last_fullsized":"2024-03-26T03:53:56.808367+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:03.919769+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"148'12742","reported_seq":12933,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.450691+0000","last_change":"2024-03-26T03:52:39.118362+0000","last_active":"2024-03-26T03:53:59.450691+0000","last_peered":"2024-03-26T03:53:59.450691+0000","last_clean":"2024-03-26T03:53:59.450691+0000","last_became_active":"2024-03-26T03:50:44.590889+0000","last_became_peered":"2024-03-26T03:50:44.590889+0000","last_unstale":"2024-03-26T03:53:59.450691+0000","last_undegraded":"2024-03-26T03:53:59.450691+0000","last_fullsized":"2024-03-26T03:53:59.450691+0000","mapping_epoch":95,"log_start":"128'7200","ondisk_log_start":"128'7200","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'7740","last_scrub_stamp":"2024-03-26T03:52:39.118187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.118187+0000","objects_scrubbed":6400,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.188478+0000","scrub_duration":2793,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87949312,"num_objects":1342,"num_object_clones":0,"num_object_copies":4026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":189,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.817635+0000","last_change":"2024-03-26T03:52:44.431735+0000","last_active":"2024-03-26T03:53:56.817635+0000","last_peered":"2024-03-26T03:53:56.817635+0000","last_clean":"2024-03-26T03:53:56.817635+0000","last_became_active":"2024-03-26T03:48:50.043838+0000","last_became_peered":"2024-03-26T03:48:50.043838+0000","last_unstale":"2024-03-26T03:53:56.817635+0000","last_undegraded":"2024-03-26T03:53:56.817635+0000","last_fullsized":"2024-03-26T03:53:56.817635+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.431608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.431608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:09.078115+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.825122+0000","last_change":"2024-03-26T03:52:30.927185+0000","last_active":"2024-03-26T03:53:56.825122+0000","last_peered":"2024-03-26T03:53:56.825122+0000","last_clean":"2024-03-26T03:53:56.825122+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:53:56.825122+0000","last_undegraded":"2024-03-26T03:53:56.825122+0000","last_fullsized":"2024-03-26T03:53:56.825122+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:30.926620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.926620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.789902+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":123,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807488+0000","last_change":"2024-03-26T03:51:23.188460+0000","last_active":"2024-03-26T03:53:56.807488+0000","last_peered":"2024-03-26T03:53:56.807488+0000","last_clean":"2024-03-26T03:53:56.807488+0000","last_became_active":"2024-03-26T03:50:45.593583+0000","last_became_peered":"2024-03-26T03:50:45.593583+0000","last_unstale":"2024-03-26T03:53:56.807488+0000","last_undegraded":"2024-03-26T03:53:56.807488+0000","last_fullsized":"2024-03-26T03:53:56.807488+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:23.188316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:23.188316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484510+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":217,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808300+0000","last_change":"2024-03-26T03:51:32.392253+0000","last_active":"2024-03-26T03:53:56.808300+0000","last_peered":"2024-03-26T03:53:56.808300+0000","last_clean":"2024-03-26T03:53:56.808300+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:53:56.808300+0000","last_undegraded":"2024-03-26T03:53:56.808300+0000","last_fullsized":"2024-03-26T03:53:56.808300+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:51:32.392163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.392163+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484437+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"148'12720","reported_seq":12905,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:59.631865+0000","last_change":"2024-03-26T03:52:46.641388+0000","last_active":"2024-03-26T03:53:59.631865+0000","last_peered":"2024-03-26T03:53:59.631865+0000","last_clean":"2024-03-26T03:53:59.631865+0000","last_became_active":"2024-03-26T03:50:44.587505+0000","last_became_peered":"2024-03-26T03:50:44.587505+0000","last_unstale":"2024-03-26T03:53:59.631865+0000","last_undegraded":"2024-03-26T03:53:59.631865+0000","last_fullsized":"2024-03-26T03:53:59.631865+0000","mapping_epoch":95,"log_start":"131'7900","ondisk_log_start":"131'7900","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"133'8232","last_scrub_stamp":"2024-03-26T03:52:46.640967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:43.560190+0000","last_clean_scrub_stamp":"2024-03-26T03:52:46.640967+0000","objects_scrubbed":5921,"log_size":4820,"log_dups_size":0,"ondisk_log_size":4820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:01.686391+0000","scrub_duration":3071,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":1344,"num_object_clones":0,"num_object_copies":4032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12720,"num_write_kb":450048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":181,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.808547+0000","last_change":"2024-03-26T03:51:41.329652+0000","last_active":"2024-03-26T03:53:56.808547+0000","last_peered":"2024-03-26T03:53:56.808547+0000","last_clean":"2024-03-26T03:53:56.808547+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:53:56.808547+0000","last_undegraded":"2024-03-26T03:53:56.808547+0000","last_fullsized":"2024-03-26T03:53:56.808547+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:41.329573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:51:41.329573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484592+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":265,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.817914+0000","last_change":"2024-03-26T03:53:17.373928+0000","last_active":"2024-03-26T03:53:56.817914+0000","last_peered":"2024-03-26T03:53:56.817914+0000","last_clean":"2024-03-26T03:53:56.817914+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:53:56.817914+0000","last_undegraded":"2024-03-26T03:53:56.817914+0000","last_fullsized":"2024-03-26T03:53:56.817914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:53:17.373796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.373796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.915867+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":185,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.827228+0000","last_change":"2024-03-26T03:51:32.274566+0000","last_active":"2024-03-26T03:53:56.827228+0000","last_peered":"2024-03-26T03:53:56.827228+0000","last_clean":"2024-03-26T03:53:56.827228+0000","last_became_active":"2024-03-26T03:48:44.013601+0000","last_became_peered":"2024-03-26T03:48:44.013601+0000","last_unstale":"2024-03-26T03:53:56.827228+0000","last_undegraded":"2024-03-26T03:53:56.827228+0000","last_fullsized":"2024-03-26T03:53:56.827228+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:51:32.274349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:51:32.274349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:53:58.524116+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":222,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-26T03:53:56.807872+0000","last_change":"2024-03-26T03:52:31.734237+0000","last_active":"2024-03-26T03:53:56.807872+0000","last_peered":"2024-03-26T03:53:56.807872+0000","last_clean":"2024-03-26T03:53:56.807872+0000","last_became_active":"2024-03-26T03:50:45.598403+0000","last_became_peered":"2024-03-26T03:50:45.598403+0000","last_unstale":"2024-03-26T03:53:56.807872+0000","last_undegraded":"2024-03-26T03:53:56.807872+0000","last_fullsized":"2024-03-26T03:53:56.807872+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:00.484719+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1449328664,"num_objects":22116,"num_object_clones":0,"num_object_copies":66348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22116,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202588,"num_write_kb":7190465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4359729152,"data_stored":4359716936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75388,"ondisk_log_size":75388,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":303,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":983,"num_read_kb":708,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":93,"num_bytes_recovered":1364,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":616,"ondisk_log_size":616,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070531,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":566284,"kb_used_data":458756,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93178868,"statfs":{"total":95995035648,"available":95415160832,"internally_reserved":0,"allocated":469766144,"data_stored":469403892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":110098890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574946,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":865128,"kb_used_data":735200,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92880024,"statfs":{"total":95995035648,"available":95109144576,"internally_reserved":0,"allocated":752844800,"data_stored":752453053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862176,"kb_used_data":736408,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":92882976,"statfs":{"total":95995035648,"available":95112167424,"internally_reserved":0,"allocated":754081792,"data_stored":753699206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411421,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":844148,"kb_used_data":714284,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":92901004,"statfs":{"total":95995035648,"available":95130628096,"internally_reserved":0,"allocated":731426816,"data_stored":731086642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088280,"kb_used_data":922448,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":92656872,"statfs":{"total":95995035648,"available":94880636928,"internally_reserved":0,"allocated":944586752,"data_stored":944211518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953554,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579924,"kb_used_data":468100,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93165228,"statfs":{"total":95995035648,"available":95401193472,"internally_reserved":0,"allocated":479334400,"data_stored":478969920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":114358725},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659212,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438916,"kb_used_data":367348,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93306236,"statfs":{"total":95995035648,"available":95545585664,"internally_reserved":0,"allocated":376164352,"data_stored":375816294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":720179200,"data_stored":720175128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":733483008,"data_stored":733478936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":720896000,"data_stored":720896000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":902696960,"data_stored":902692888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":463798272,"data_stored":463798272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":364576768,"data_stored":364576768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":454098944,"data_stored":454098944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:54:06.558 INFO:tasks.thrashosds.thrasher:Setting 3.1f to [7, 3, 0, 2, 4, 6] 2024-03-26T03:54:06.559 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.1f', '7', '3', '0', '2', '4', '6'] 2024-03-26T03:54:06.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 3.1f 7 3 0 2 4 6 2024-03-26T03:54:06.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:06.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:06.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:06.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:06.834 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:54:06.834 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:54:06.834 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:54:06.835 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:54:06.835 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:54:06.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:06.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:06.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:07.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:07.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:07.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:06 smithi006 ceph-mon[37501]: pgmap v214: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:54:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:06 smithi006 ceph-mon[29446]: pgmap v214: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:54:07.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:07.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:07.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:07.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:06 smithi086 ceph-mon[32965]: pgmap v214: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 987 op/s 2024-03-26T03:54:07.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:07.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:07.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:07.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:07.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:07.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:07.693 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:07.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:07.831 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:07.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:07.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:07.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:54:07.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:07.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:07.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:07.993 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:08.013 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:08.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:08.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:08.219 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:07 smithi006 ceph-mon[37501]: pgmap v215: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 944 op/s 2024-03-26T03:54:08.219 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:07 smithi006 ceph-mon[29446]: pgmap v215: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 944 op/s 2024-03-26T03:54:08.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:08.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:07 smithi086 ceph-mon[32965]: pgmap v215: 148 pgs: 148 active+clean; 1.0 GiB data, 4.1 GiB used, 622 GiB / 626 GiB avail; 978 B/s rd, 0 B/s wr, 944 op/s 2024-03-26T03:54:08.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:08.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:08.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:08.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:08.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:08.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:08.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:08.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:08.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:08.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:08.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:09.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:09.080 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:08 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:09.081 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:08 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:09.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:09.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:09.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:09.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:09.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:08 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:09.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:09.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:09.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:09.503 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:09.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:09.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.648 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:09.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:09.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:09.786 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:09.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:09.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:09.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:09.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:09.921 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:54:10.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:10.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:10.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:10.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:10.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:10.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:54:10.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:09 smithi006 ceph-mon[37501]: pgmap v216: 148 pgs: 148 active+clean; 1014 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s 2024-03-26T03:54:10.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:54:10.202 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:09 smithi006 ceph-mon[29446]: pgmap v216: 148 pgs: 148 active+clean; 1014 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s 2024-03-26T03:54:10.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:10.325 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:54:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:09 smithi086 ceph-mon[32965]: pgmap v216: 148 pgs: 148 active+clean; 1014 MiB data, 3.8 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 898 op/s 2024-03-26T03:54:10.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:10.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.465 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:10.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:10.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:10.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:10.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:10.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:10.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:10.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:10.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:10.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:10.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:11.028 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:11.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:11.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:54:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[37501]: osdmap e149: 8 total, 8 up, 7 in 2024-03-26T03:54:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:54:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:10 smithi006 ceph-mon[29446]: osdmap e149: 8 total, 8 up, 7 in 2024-03-26T03:54:11.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:11.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:11.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:11.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:10 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:10 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2866281810' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:54:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:10 smithi086 ceph-mon[32965]: osdmap e149: 8 total, 8 up, 7 in 2024-03-26T03:54:11.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:11.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:11.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:11.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.542 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:11.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:11.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:11.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:11.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:11.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:11.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:11.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:11.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:11.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:12.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:12.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:12.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:12.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[37501]: pgmap v218: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[29446]: pgmap v218: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T03:54:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:11 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T03:54:12.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:12.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:12.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:12.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:11 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T03:54:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:11 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T03:54:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:11 smithi086 ceph-mon[32965]: pgmap v218: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 995 op/s 2024-03-26T03:54:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:11 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T03:54:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:11 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T03:54:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:12.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:12.464 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:12.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:12.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:12.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:12.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:12.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:12.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:12.866 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:12.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:12.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:12.993 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:12.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:13.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:13.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:13.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:13.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T03:54:13.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T03:54:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:12 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T03:54:13.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:13.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:13.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:12 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T03:54:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:12 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T03:54:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:12 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T03:54:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:12 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T03:54:13.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:13.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:13.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:13.548 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:13.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:13.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:13.685 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:13.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:13.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:13.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:13.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:13.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:54:13.955 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:13.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:13.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:14.019 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[37501]: pgmap v219: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-26T03:54:14.019 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T03:54:14.019 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T03:54:14.019 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:54:14.020 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[29446]: pgmap v219: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-26T03:54:14.020 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T03:54:14.020 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:13 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T03:54:14.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:14.069 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:54:14.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:14.120 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:54:14.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:14.186 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:54:14.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:14.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:14.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:54:14.282 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:54:14.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:14.322 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:14.325 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:54:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:14.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:13 smithi086 ceph-mon[32965]: pgmap v219: 148 pgs: 148 active+clean; 761 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 996 op/s 2024-03-26T03:54:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:13 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T03:54:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:13 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T03:54:14.375 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:54:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:54:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:54:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:54:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:54:14.398 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:54:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:54:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:54:14.399 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:54:14.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:14.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:14.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:14.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:14.621 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:14.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:14.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:14.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:14.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:14.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:14.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:14.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:15.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:15.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:15.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:15.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:15.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:15.251 INFO:teuthology.orchestra.run.smithi006.stderr:set 3.1f pg_upmap_items mapping to [7->3,0->2,4->6] 2024-03-26T03:54:15.251 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: 7.a scrub starts 2024-03-26T03:54:15.251 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2445363140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:15.251 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: 7.a scrub ok 2024-03-26T03:54:15.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:15.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3252957205' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T03:54:15.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T03:54:15.256 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: 7.a scrub starts 2024-03-26T03:54:15.256 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2445363140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:15.256 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: 7.a scrub ok 2024-03-26T03:54:15.256 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:15.256 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3252957205' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.257 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.257 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T03:54:15.257 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:14 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T03:54:15.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:15.306 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: 7.a scrub starts 2024-03-26T03:54:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2445363140' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: 7.a scrub ok 2024-03-26T03:54:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3252957205' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]: dispatch 2024-03-26T03:54:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T03:54:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:14 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T03:54:15.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:15.460 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:15.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:15.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:15.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:15.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:15.726 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:15.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:15.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:15.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:15.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:15.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:16.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:16.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:16.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:16.264 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:16.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:16.412 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:16.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:16.514 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:16.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T03:54:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T03:54:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 7.c scrub starts 2024-03-26T03:54:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 7.c scrub ok 2024-03-26T03:54:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]': finished 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: osdmap e150: 8 total, 8 up, 7 in 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: pgmap v221: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T03:54:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:16 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T03:54:16.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T03:54:16.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 7.c scrub starts 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 7.c scrub ok 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]': finished 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: osdmap e150: 8 total, 8 up, 7 in 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: pgmap v221: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T03:54:16.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 7.c scrub starts 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 7.c scrub ok 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.1f", "id": [7, 3, 0, 2, 4, 6]}]': finished 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: osdmap e150: 8 total, 8 up, 7 in 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: pgmap v221: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T03:54:16.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:16 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T03:54:16.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:16.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:16.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:16.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:16.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:16.948 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:16.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:17.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:17.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:17.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:17.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:17.216 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:17.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:17.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:17.504 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:17.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:17.528 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T03:54:17.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T03:54:17.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:54:17.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:17 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:54:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T03:54:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T03:54:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T03:54:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T03:54:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T03:54:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T03:54:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:54:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:17 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:54:17.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:17.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:17.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:17.770 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:17.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:17.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:17.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:17.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:17.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:18.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.051 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:18.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:18.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.221 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:18.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:18.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:18.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:18.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:18.364 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: osdmap e151: 8 total, 8 up, 7 in 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: pgmap v223: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 650 op/s 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 6.2 scrub starts 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 6.2 scrub ok 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T03:54:18.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: osdmap e151: 8 total, 8 up, 7 in 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: pgmap v223: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 650 op/s 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 6.2 scrub starts 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 6.2 scrub ok 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T03:54:18.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:18 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T03:54:18.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:18.463 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:54:18.463 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:54:18.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:18.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:18.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T03:54:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T03:54:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: osdmap e151: 8 total, 8 up, 7 in 2024-03-26T03:54:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: pgmap v223: 148 pgs: 148 active+clean; 462 MiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 695 B/s rd, 0 B/s wr, 650 op/s 2024-03-26T03:54:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 6.2 scrub starts 2024-03-26T03:54:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 6.2 scrub ok 2024-03-26T03:54:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T03:54:18.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:18 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T03:54:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.616 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:18.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:18.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:18.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:18.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:18.898 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:18.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:18.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:19.044 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:19.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:19.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:19.185 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:19.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:19.326 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:19.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:19.474 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:19.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T03:54:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T03:54:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T03:54:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T03:54:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T03:54:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T03:54:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T03:54:19.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:19 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T03:54:19.614 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T03:54:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T03:54:19.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:19 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T03:54:19.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:19.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:19.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:19.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:19.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:19.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:54:19.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:19.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:19.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:20.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:20.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:20.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:20.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:20.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:20.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:20.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: pgmap v224: 148 pgs: 148 active+clean; 404 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T03:54:20.564 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T03:54:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T03:54:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: pgmap v224: 148 pgs: 148 active+clean; 404 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s 2024-03-26T03:54:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T03:54:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:20 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T03:54:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: pgmap v224: 148 pgs: 148 active+clean; 404 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 714 op/s 2024-03-26T03:54:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T03:54:20.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:20 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T03:54:20.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:20.709 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:20.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:20.848 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:20.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:20.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:20.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:20.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:21.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:21.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:21.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:21.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:21.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:21.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:21 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T03:54:21.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:21 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T03:54:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:21 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T03:54:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:21 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T03:54:21.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:21.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:21.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:21 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T03:54:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:21 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T03:54:21.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:21.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:21.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:21.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:21.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:21.966 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:21.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:22.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:22.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:54:22.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:22.104 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Removed 112351 objects 2024-03-26T03:54:22.105 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Clean up completed and total clean up time :115.822 2024-03-26T03:54:22.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:22.217 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:22.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:22.342 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:22.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:22.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:22.449 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:54:22.449 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:54:22.449 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 7] 2024-03-26T03:54:22.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd out 3 2024-03-26T03:54:22.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:22.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:22.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T03:54:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T03:54:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T03:54:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T03:54:22.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 7.8 scrub starts 2024-03-26T03:54:22.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 7.8 scrub ok 2024-03-26T03:54:22.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: pgmap v225: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:54:22.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T03:54:22.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:22 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T03:54:22.592 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:22.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:22.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T03:54:22.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 7.8 scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 7.8 scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: pgmap v225: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 7.8 scrub starts 2024-03-26T03:54:22.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 7.8 scrub ok 2024-03-26T03:54:22.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: pgmap v225: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-26T03:54:22.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T03:54:22.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:22 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T03:54:22.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:22.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:22.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:22.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:22.825 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:22.845 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:22.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:22.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:22.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:22.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:22.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:23.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:23.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:23.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:23.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:23.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:23.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:23.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:23.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:23.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:23.473 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:23.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:23.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.600 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:23.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:23.622 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-26T03:54:23.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-26T03:54:23.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:23.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:23.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:23.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:23.850 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:23.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:23.914 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:23.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:23.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:23.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:23.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:24.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:24.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:24.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:24.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:24.310 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:54:24.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:24.353 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:24.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:24.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:24.481 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:24.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:24.572 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 7.e scrub starts 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 7.e scrub ok 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: pgmap v226: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 636 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1769380991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T03:54:24.573 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 7.e scrub starts 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 7.e scrub ok 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: pgmap v226: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 636 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1769380991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:24.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 7.e scrub starts 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 7.e scrub ok 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: pgmap v226: 148 pgs: 148 active+clean; 158 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 636 B/s rd, 0 B/s wr, 605 op/s 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1769380991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:24.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:24.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:24.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:24.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:24.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:24.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:24.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:24.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:24.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:24.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:24.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:25.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:25.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:25.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:25.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.235 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:25.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:25.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:25.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:25.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:25.423 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T03:54:25.423 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T03:54:25.423 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: osdmap e152: 8 total, 8 up, 7 in 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:54:25.424 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: osdmap e152: 8 total, 8 up, 7 in 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T03:54:25.425 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:25 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T03:54:25.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:25.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:25.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:25.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: osdmap e152: 8 total, 8 up, 7 in 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T03:54:25.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:25 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T03:54:25.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:25.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:25.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:25.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:25.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:25.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:25.875 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:25.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:25.989 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:25.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:26.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.115 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:26.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:26.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:26.240 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:26.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:26.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:26.414 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:26.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:26.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:26.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:26 smithi086 ceph-mon[32965]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 921 op/s 2024-03-26T03:54:26.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:26 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T03:54:26.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:26 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T03:54:26.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:26.608 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[37501]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 921 op/s 2024-03-26T03:54:26.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T03:54:26.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T03:54:26.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[29446]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 921 op/s 2024-03-26T03:54:26.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T03:54:26.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:26 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T03:54:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:26.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:26.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:26.741 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:26.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:26.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:26.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:26.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:26.996 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:26.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:27.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:27.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:27.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:27.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:27.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:27.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:27.410 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T03:54:27.410 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[37501]: 4.12 deep-scrub starts 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[37501]: 4.12 deep-scrub ok 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[29446]: 4.12 deep-scrub starts 2024-03-26T03:54:27.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-mon[29446]: 4.12 deep-scrub ok 2024-03-26T03:54:27.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:27.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:27.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:27 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T03:54:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:27 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T03:54:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:27 smithi086 ceph-mon[32965]: 4.12 deep-scrub starts 2024-03-26T03:54:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:27 smithi086 ceph-mon[32965]: 4.12 deep-scrub ok 2024-03-26T03:54:27.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:27.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:27.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:27.752 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:27.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:27.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:54:27.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:27.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:27.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:27.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:27 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T03:54:27.452+0000 7fd465f8b700 -1 mon.a@0(leader).osd e152 definitely_dead 0 2024-03-26T03:54:27.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:28.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:28.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:28.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:28.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:28.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:28.262 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:28.276 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:54:28.297 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:54:28.318 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:54:28.355 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.3. 2024-03-26T03:54:28.362 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:54:28.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:28.387 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:54:28.390 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:28.410 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:54:28.410 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 739 op/s 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2767156358' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/482290675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 739 op/s 2024-03-26T03:54:28.411 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2767156358' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.412 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.412 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T03:54:28.412 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T03:54:28.412 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T03:54:28.412 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/482290675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:28.431 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:54:28.451 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:54:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:54:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:54:28.452 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:54:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:54:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:54:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:54:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:54:28.453 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:54:28.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:28.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:28.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:28.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:28.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:28.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T03:54:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T03:54:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 857 MiB used, 625 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 739 op/s 2024-03-26T03:54:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2767156358' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T03:54:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T03:54:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T03:54:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T03:54:28.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/482290675' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:28.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:28.894 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:28.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:28.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:29.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:29.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:29.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:29.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:29.266 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_0' removed 2024-03-26T03:54:29.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:29.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:29.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:29.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:29.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:29.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T03:54:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T03:54:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T03:54:29.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: osdmap e153: 8 total, 8 up, 6 in 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: 6.e deep-scrub starts 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: 6.e deep-scrub ok 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/772298516' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:54:29.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:29 smithi086 ceph-mon[32965]: osdmap e154: 8 total, 8 up, 6 in 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: osdmap e153: 8 total, 8 up, 6 in 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: 6.e deep-scrub starts 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: 6.e deep-scrub ok 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/772298516' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:54:29.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[29446]: osdmap e154: 8 total, 8 up, 6 in 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: osdmap e153: 8 total, 8 up, 6 in 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: 6.e deep-scrub starts 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: 6.e deep-scrub ok 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/772298516' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:54:29.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:29 smithi006 ceph-mon[37501]: osdmap e154: 8 total, 8 up, 6 in 2024-03-26T03:54:29.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:29.687 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:29.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:29.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:29.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:29.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:29.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:29.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:29.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:30.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:30.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:30.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:30.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:30.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:30.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:30.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:30.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:30.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:30.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:30.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T03:54:30.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T03:54:30.623 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: pgmap v232: 132 pgs: 32 peering, 100 active+clean; 582 KiB data, 699 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: pgmap v232: 132 pgs: 32 peering, 100 active+clean; 582 KiB data, 699 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T03:54:30.624 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:30 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:54:30.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:30.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:30.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:30.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:30.813 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:30.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T03:54:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T03:54:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: pgmap v232: 132 pgs: 32 peering, 100 active+clean; 582 KiB data, 699 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T03:54:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T03:54:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T03:54:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:30 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 7 pgs inactive, 11 pgs peering (PG_AVAILABILITY) 2024-03-26T03:54:30.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:30.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:30.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:30.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:31.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:31.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:31.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:31.134 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:54:31.135 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:54:31.135 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-26T03:54:31.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-26T03:54:31.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:31.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:31.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:31.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:31.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:31.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:31.443 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:31.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:31.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:31.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:31.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:31 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T03:54:31.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:31 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T03:54:31.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:31 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T03:54:31.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:31 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T03:54:31.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:31.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:31.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:31.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:31.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:31 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T03:54:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:31 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T03:54:31.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:31.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:31.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:31.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:32.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:32.087 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:32.093 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:32.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:32.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:32.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:32.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:32.345 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:32.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:32.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:32.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:32.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:32.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: pgmap v233: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 282 B/s, 5 objects/s recovering 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 3.8 deep-scrub starts 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[37501]: 3.8 deep-scrub ok 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T03:54:32.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: pgmap v233: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 282 B/s, 5 objects/s recovering 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 3.8 deep-scrub starts 2024-03-26T03:54:32.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:32 smithi006 ceph-mon[29446]: 3.8 deep-scrub ok 2024-03-26T03:54:32.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:32.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:32.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:32.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T03:54:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: pgmap v233: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 282 B/s, 5 objects/s recovering 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 3.8 deep-scrub starts 2024-03-26T03:54:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:32 smithi086 ceph-mon[32965]: 3.8 deep-scrub ok 2024-03-26T03:54:32.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:32.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:32.932 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:54:32.933 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:54:32.933 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T03:54:32.933 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:54:32.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:32.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:32.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:33.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:33.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:33.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:33.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:33.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 6.6 scrub starts 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 6.6 scrub ok 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 4.14 scrub starts 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[37501]: 4.14 scrub ok 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T03:54:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 6.6 scrub starts 2024-03-26T03:54:33.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 6.6 scrub ok 2024-03-26T03:54:33.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T03:54:33.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T03:54:33.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 4.14 scrub starts 2024-03-26T03:54:33.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:33 smithi006 ceph-mon[29446]: 4.14 scrub ok 2024-03-26T03:54:33.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:33.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:33.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:33.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:33.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:33.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:33.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:33.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 6.6 scrub starts 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 6.6 scrub ok 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 4.14 scrub starts 2024-03-26T03:54:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:33 smithi086 ceph-mon[32965]: 4.14 scrub ok 2024-03-26T03:54:33.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:33.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:33.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:33.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:34.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:54:34.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:34.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:34.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:34.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:34.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:54:34.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:34.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:34.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:34.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[37501]: pgmap v234: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 246 B/s, 5 objects/s recovering 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[29446]: pgmap v234: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 246 B/s, 5 objects/s recovering 2024-03-26T03:54:34.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T03:54:34.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:34 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T03:54:34.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:34.647 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:34.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:34.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:34.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:34.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:34 smithi086 ceph-mon[32965]: pgmap v234: 132 pgs: 1 activating, 1 active+recovering+degraded, 32 peering, 98 active+clean; 582 KiB data, 703 MiB used, 536 GiB / 536 GiB avail; 7/690 objects misplaced (1.014%); 246 B/s, 5 objects/s recovering 2024-03-26T03:54:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:34 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T03:54:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:34 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T03:54:34.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:34.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:34.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:35.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:35.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:35.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:35.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:35.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:35.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:35.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:35.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:35.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:35.407 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:35.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:35.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.532 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:35.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:35.565 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:35.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:35.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 5.0 deep-scrub starts 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[37501]: 5.0 deep-scrub ok 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T03:54:35.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:35.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 5.0 deep-scrub starts 2024-03-26T03:54:35.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:35 smithi006 ceph-mon[29446]: 5.0 deep-scrub ok 2024-03-26T03:54:35.649 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:54:35.649 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:54:35.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:54:35.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:35.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:35.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T03:54:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T03:54:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T03:54:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T03:54:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 5.0 deep-scrub starts 2024-03-26T03:54:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:35 smithi086 ceph-mon[32965]: 5.0 deep-scrub ok 2024-03-26T03:54:35.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:35.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:35.999 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:36.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:36.046 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:36.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:36.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:36.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:36.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:36.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:36.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:36.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:36.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:36.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:36.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: 6.11 deep-scrub starts 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: 6.11 deep-scrub ok 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 397 B/s, 13 objects/s recovering 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T03:54:36.583 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: 6.11 deep-scrub starts 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: 6.11 deep-scrub ok 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 397 B/s, 13 objects/s recovering 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T03:54:36.584 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:36 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T03:54:36.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:36.686 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:36.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:36.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:36.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: 6.11 deep-scrub starts 2024-03-26T03:54:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: 6.11 deep-scrub ok 2024-03-26T03:54:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: pgmap v235: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 397 B/s, 13 objects/s recovering 2024-03-26T03:54:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 8 pgs inactive, 11 pgs peering) 2024-03-26T03:54:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-03-26T03:54:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:54:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T03:54:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:36 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T03:54:36.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:36.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:37.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:37.049 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:37.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:37.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:37.198 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:37.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:37.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:37.323 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:37.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_1' created 2024-03-26T03:54:37.376 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:54:37.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:37.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2622481131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2295704923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.467 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2622481131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2295704923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:37 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:37.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:37.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:37.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:37.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:37.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T03:54:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T03:54:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2622481131' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:54:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2295704923' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:37 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-26T03:54:37.840 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:37.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:37.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:37.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:37.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:38.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:38.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:38.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:38.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:38.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:38.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:38.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:38.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:38.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.477 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:38.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:38.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:38.602 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 457 B/s rd, 0 op/s; 355 B/s, 11 objects/s recovering 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:54:38.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[37501]: osdmap e155: 8 total, 8 up, 6 in 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 457 B/s rd, 0 op/s; 355 B/s, 11 objects/s recovering 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:54:38.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:38 smithi006 ceph-mon[29446]: osdmap e155: 8 total, 8 up, 6 in 2024-03-26T03:54:38.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:38.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:38.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 457 B/s rd, 0 op/s; 355 B/s, 11 objects/s recovering 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-26T03:54:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:38 smithi086 ceph-mon[32965]: osdmap e155: 8 total, 8 up, 6 in 2024-03-26T03:54:38.859 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:38.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:38.985 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:38.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:39.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:39.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:39.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:39.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:39.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:39.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:39.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:39.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:39.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:39.492 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:39.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:39.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:39.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:39.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:39.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:39 smithi006 ceph-mon[37501]: osdmap e156: 8 total, 8 up, 6 in 2024-03-26T03:54:39.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:39 smithi006 ceph-mon[29446]: osdmap e156: 8 total, 8 up, 6 in 2024-03-26T03:54:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:39.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:39.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:39 smithi086 ceph-mon[32965]: osdmap e156: 8 total, 8 up, 6 in 2024-03-26T03:54:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:39.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:39.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:39.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:39.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:39.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:40.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:40.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:40.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:40.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:40.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:40.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:40.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:40.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:40.518 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:40.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:40.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[37501]: pgmap v239: 148 pgs: 8 unknown, 140 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 151 B/s, 8 objects/s recovering 2024-03-26T03:54:40.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[37501]: osdmap e157: 8 total, 8 up, 6 in 2024-03-26T03:54:40.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:54:40.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[29446]: pgmap v239: 148 pgs: 8 unknown, 140 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 151 B/s, 8 objects/s recovering 2024-03-26T03:54:40.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[29446]: osdmap e157: 8 total, 8 up, 6 in 2024-03-26T03:54:40.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:40 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:54:40.643 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:40.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:40.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:40 smithi086 ceph-mon[32965]: pgmap v239: 148 pgs: 8 unknown, 140 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 151 B/s, 8 objects/s recovering 2024-03-26T03:54:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:40 smithi086 ceph-mon[32965]: osdmap e157: 8 total, 8 up, 6 in 2024-03-26T03:54:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:40 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:54:40.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:40.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:40.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:40.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:40.990 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:54:40.990 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:54:40.990 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:54:40.990 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:54:40.990 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:54:40.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:41.009 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:54:41.011 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:54:41.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:41.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:41.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:41.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:41.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:41.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:41.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:41.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:41.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:41.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:41.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:41.535 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:41.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:41.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:41.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:41.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:41.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:41.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:41.815 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:54:41.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:41.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:41.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:42.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:42.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:42.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:42.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:42.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:42.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:42.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:42.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:42.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:42.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:42.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:42.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:42 smithi006 ceph-mon[37501]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:42.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:42 smithi006 ceph-mon[37501]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:42.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:42 smithi006 ceph-mon[29446]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:42.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:42 smithi006 ceph-mon[29446]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:42.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:42.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:42.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:42 smithi086 ceph-mon[32965]: from='client.15018 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:42 smithi086 ceph-mon[32965]: pgmap v241: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:42.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:42.946 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:42.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:43.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:43.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:43.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:43.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:43.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:43.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:43.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:43.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-26T03:54:43.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:43.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:43.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:43.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:43.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:43.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:43.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:43.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:43.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:43.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:43.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:43.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:43.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:44.017 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-03-26T03:54:39.375252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1562,"num_read_kb":1376,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":2366036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":838,"ondisk_log_size":838,"up":420,"acting":420,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":405,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":742000,"kb_used_data":6576,"kb_used_omap":9,"kb_used_meta":735350,"kb_avail":561728912,"statfs":{"total":575970213888,"available":575210405888,"internally_reserved":0,"allocated":6733824,"data_stored":4235906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9565,"internal_metadata":752999075},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":1210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.092308"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":197,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372034+0000","last_change":"2024-03-26T03:53:29.435229+0000","last_active":"2024-03-26T03:54:38.372034+0000","last_peered":"2024-03-26T03:54:38.372034+0000","last_clean":"2024-03-26T03:54:38.372034+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:54:38.372034+0000","last_undegraded":"2024-03-26T03:54:38.372034+0000","last_fullsized":"2024-03-26T03:54:38.372034+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:49.505917+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.257059+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:54:29.257059+0000","last_peered":"2024-03-26T03:54:29.257059+0000","last_clean":"2024-03-26T03:54:29.257059+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:54:29.257059+0000","last_undegraded":"2024-03-26T03:54:29.257059+0000","last_fullsized":"2024-03-26T03:54:29.257059+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.247510+0000","last_change":"2024-03-26T03:54:15.751346+0000","last_active":"2024-03-26T03:54:29.247510+0000","last_peered":"2024-03-26T03:54:29.247510+0000","last_clean":"2024-03-26T03:54:29.247510+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:54:29.247510+0000","last_undegraded":"2024-03-26T03:54:29.247510+0000","last_fullsized":"2024-03-26T03:54:29.247510+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.598037+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":269,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.371666+0000","last_change":"2024-03-26T03:54:29.547822+0000","last_active":"2024-03-26T03:54:38.371666+0000","last_peered":"2024-03-26T03:54:38.371666+0000","last_clean":"2024-03-26T03:54:38.371666+0000","last_became_active":"2024-03-26T03:54:29.438463+0000","last_became_peered":"2024-03-26T03:54:29.438463+0000","last_unstale":"2024-03-26T03:54:38.371666+0000","last_undegraded":"2024-03-26T03:54:38.371666+0000","last_fullsized":"2024-03-26T03:54:38.371666+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785154+0000","last_change":"2024-03-26T03:54:29.358028+0000","last_active":"2024-03-26T03:54:38.785154+0000","last_peered":"2024-03-26T03:54:38.785154+0000","last_clean":"2024-03-26T03:54:38.785154+0000","last_became_active":"2024-03-26T03:54:29.357724+0000","last_became_peered":"2024-03-26T03:54:29.357724+0000","last_unstale":"2024-03-26T03:54:38.785154+0000","last_undegraded":"2024-03-26T03:54:38.785154+0000","last_fullsized":"2024-03-26T03:54:38.785154+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782768+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.371549+0000","last_change":"2024-03-26T03:53:13.396637+0000","last_active":"2024-03-26T03:54:38.371549+0000","last_peered":"2024-03-26T03:54:38.371549+0000","last_clean":"2024-03-26T03:54:38.371549+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:54:38.371549+0000","last_undegraded":"2024-03-26T03:54:38.371549+0000","last_fullsized":"2024-03-26T03:54:38.371549+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.396551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.396551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":90,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.982447+0000","last_change":"2024-03-26T03:54:29.982467+0000","last_active":"2024-03-26T03:54:29.982447+0000","last_peered":"2024-03-26T03:54:29.982447+0000","last_clean":"2024-03-26T03:54:29.982447+0000","last_became_active":"2024-03-26T03:54:29.377365+0000","last_became_peered":"2024-03-26T03:54:29.377365+0000","last_unstale":"2024-03-26T03:54:29.982447+0000","last_undegraded":"2024-03-26T03:54:29.982447+0000","last_fullsized":"2024-03-26T03:54:29.982447+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.982392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.982392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.166800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":128,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.418094+0000","last_change":"2024-03-26T03:54:29.395843+0000","last_active":"2024-03-26T03:54:29.418094+0000","last_peered":"2024-03-26T03:54:29.418094+0000","last_clean":"2024-03-26T03:54:29.418094+0000","last_became_active":"2024-03-26T03:54:29.330558+0000","last_became_peered":"2024-03-26T03:54:29.330558+0000","last_unstale":"2024-03-26T03:54:29.418094+0000","last_undegraded":"2024-03-26T03:54:29.418094+0000","last_fullsized":"2024-03-26T03:54:29.418094+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.232404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":193,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.249325+0000","last_change":"2024-03-26T03:53:22.532263+0000","last_active":"2024-03-26T03:54:29.249325+0000","last_peered":"2024-03-26T03:54:29.249325+0000","last_clean":"2024-03-26T03:54:29.249325+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:54:29.249325+0000","last_undegraded":"2024-03-26T03:54:29.249325+0000","last_fullsized":"2024-03-26T03:54:29.249325+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:22.532151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:22.532151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.376839+0000","last_change":"2024-03-26T03:53:24.585622+0000","last_active":"2024-03-26T03:54:38.376839+0000","last_peered":"2024-03-26T03:54:38.376839+0000","last_clean":"2024-03-26T03:54:38.376839+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:54:38.376839+0000","last_undegraded":"2024-03-26T03:54:38.376839+0000","last_fullsized":"2024-03-26T03:54:38.376839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.585508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.585508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.817831+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378300+0000","last_change":"2024-03-26T03:53:18.575050+0000","last_active":"2024-03-26T03:54:38.378300+0000","last_peered":"2024-03-26T03:54:38.378300+0000","last_clean":"2024-03-26T03:54:38.378300+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:54:38.378300+0000","last_undegraded":"2024-03-26T03:54:38.378300+0000","last_fullsized":"2024-03-26T03:54:38.378300+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.817855+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":274,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372385+0000","last_change":"2024-03-26T03:54:29.551157+0000","last_active":"2024-03-26T03:54:38.372385+0000","last_peered":"2024-03-26T03:54:38.372385+0000","last_clean":"2024-03-26T03:54:38.372385+0000","last_became_active":"2024-03-26T03:54:29.444189+0000","last_became_peered":"2024-03-26T03:54:29.444189+0000","last_unstale":"2024-03-26T03:54:38.372385+0000","last_undegraded":"2024-03-26T03:54:38.372385+0000","last_fullsized":"2024-03-26T03:54:38.372385+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":211,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.375499+0000","last_change":"2024-03-26T03:54:29.289257+0000","last_active":"2024-03-26T03:54:38.375499+0000","last_peered":"2024-03-26T03:54:38.375499+0000","last_clean":"2024-03-26T03:54:38.375499+0000","last_became_active":"2024-03-26T03:54:29.288929+0000","last_became_peered":"2024-03-26T03:54:29.288929+0000","last_unstale":"2024-03-26T03:54:38.375499+0000","last_undegraded":"2024-03-26T03:54:38.375499+0000","last_fullsized":"2024-03-26T03:54:38.375499+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.259326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785144+0000","last_change":"2024-03-26T03:54:11.544712+0000","last_active":"2024-03-26T03:54:38.785144+0000","last_peered":"2024-03-26T03:54:38.785144+0000","last_clean":"2024-03-26T03:54:38.785144+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:54:38.785144+0000","last_undegraded":"2024-03-26T03:54:38.785144+0000","last_fullsized":"2024-03-26T03:54:38.785144+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.544637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.544637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:30.660465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":217,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377360+0000","last_change":"2024-03-26T03:54:32.784964+0000","last_active":"2024-03-26T03:54:38.377360+0000","last_peered":"2024-03-26T03:54:38.377360+0000","last_clean":"2024-03-26T03:54:38.377360+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:54:38.377360+0000","last_undegraded":"2024-03-26T03:54:38.377360+0000","last_fullsized":"2024-03-26T03:54:38.377360+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.924921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":244,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372533+0000","last_change":"2024-03-26T03:54:29.472775+0000","last_active":"2024-03-26T03:54:38.372533+0000","last_peered":"2024-03-26T03:54:38.372533+0000","last_clean":"2024-03-26T03:54:38.372533+0000","last_became_active":"2024-03-26T03:54:29.360309+0000","last_became_peered":"2024-03-26T03:54:29.360309+0000","last_unstale":"2024-03-26T03:54:38.372533+0000","last_undegraded":"2024-03-26T03:54:38.372533+0000","last_fullsized":"2024-03-26T03:54:38.372533+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373205+0000","last_change":"2024-03-26T03:54:29.346617+0000","last_active":"2024-03-26T03:54:38.373205+0000","last_peered":"2024-03-26T03:54:38.373205+0000","last_clean":"2024-03-26T03:54:38.373205+0000","last_became_active":"2024-03-26T03:54:29.346320+0000","last_became_peered":"2024-03-26T03:54:29.346320+0000","last_unstale":"2024-03-26T03:54:38.373205+0000","last_undegraded":"2024-03-26T03:54:38.373205+0000","last_fullsized":"2024-03-26T03:54:38.373205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":196,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.258435+0000","last_change":"2024-03-26T03:53:30.866289+0000","last_active":"2024-03-26T03:54:29.258435+0000","last_peered":"2024-03-26T03:54:29.258435+0000","last_clean":"2024-03-26T03:54:29.258435+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:54:29.258435+0000","last_undegraded":"2024-03-26T03:54:29.258435+0000","last_fullsized":"2024-03-26T03:54:29.258435+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.866047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.866047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.871487+0000","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":213,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.246470+0000","last_change":"2024-03-26T03:53:13.631462+0000","last_active":"2024-03-26T03:54:29.246470+0000","last_peered":"2024-03-26T03:54:29.246470+0000","last_clean":"2024-03-26T03:54:29.246470+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:54:29.246470+0000","last_undegraded":"2024-03-26T03:54:29.246470+0000","last_fullsized":"2024-03-26T03:54:29.246470+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.631351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.631351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:41.275139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":264,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.543230+0000","last_change":"2024-03-26T03:54:29.543230+0000","last_active":"2024-03-26T03:54:29.543230+0000","last_peered":"2024-03-26T03:54:29.543230+0000","last_clean":"2024-03-26T03:54:29.543230+0000","last_became_active":"2024-03-26T03:54:29.346374+0000","last_became_peered":"2024-03-26T03:54:29.346374+0000","last_unstale":"2024-03-26T03:54:29.543230+0000","last_undegraded":"2024-03-26T03:54:29.543230+0000","last_fullsized":"2024-03-26T03:54:29.543230+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.041388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.041388+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:30.279550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":82,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377821+0000","last_change":"2024-03-26T03:54:29.809911+0000","last_active":"2024-03-26T03:54:38.377821+0000","last_peered":"2024-03-26T03:54:38.377821+0000","last_clean":"2024-03-26T03:54:38.377821+0000","last_became_active":"2024-03-26T03:54:29.289421+0000","last_became_peered":"2024-03-26T03:54:29.289421+0000","last_unstale":"2024-03-26T03:54:38.377821+0000","last_undegraded":"2024-03-26T03:54:38.377821+0000","last_fullsized":"2024-03-26T03:54:38.377821+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:48.048135+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372951+0000","last_change":"2024-03-26T03:54:29.360637+0000","last_active":"2024-03-26T03:54:38.372951+0000","last_peered":"2024-03-26T03:54:38.372951+0000","last_clean":"2024-03-26T03:54:38.372951+0000","last_became_active":"2024-03-26T03:54:29.360289+0000","last_became_peered":"2024-03-26T03:54:29.360289+0000","last_unstale":"2024-03-26T03:54:38.372951+0000","last_undegraded":"2024-03-26T03:54:38.372951+0000","last_fullsized":"2024-03-26T03:54:38.372951+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:16.928630+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":217,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372577+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:54:38.372577+0000","last_peered":"2024-03-26T03:54:38.372577+0000","last_clean":"2024-03-26T03:54:38.372577+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:54:38.372577+0000","last_undegraded":"2024-03-26T03:54:38.372577+0000","last_fullsized":"2024-03-26T03:54:38.372577+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.802874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":80,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.440054+0000","last_change":"2024-03-26T03:54:29.404853+0000","last_active":"2024-03-26T03:54:29.440054+0000","last_peered":"2024-03-26T03:54:29.440054+0000","last_clean":"2024-03-26T03:54:29.440054+0000","last_became_active":"2024-03-26T03:54:29.322224+0000","last_became_peered":"2024-03-26T03:54:29.322224+0000","last_unstale":"2024-03-26T03:54:29.440054+0000","last_undegraded":"2024-03-26T03:54:29.440054+0000","last_fullsized":"2024-03-26T03:54:29.440054+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786136+0000","last_change":"2024-03-26T03:54:15.688631+0000","last_active":"2024-03-26T03:54:38.786136+0000","last_peered":"2024-03-26T03:54:38.786136+0000","last_clean":"2024-03-26T03:54:38.786136+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:54:38.786136+0000","last_undegraded":"2024-03-26T03:54:38.786136+0000","last_fullsized":"2024-03-26T03:54:38.786136+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:54:15.688518+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.688518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:42.161568+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":221,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372985+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:54:38.372985+0000","last_peered":"2024-03-26T03:54:38.372985+0000","last_clean":"2024-03-26T03:54:38.372985+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:54:38.372985+0000","last_undegraded":"2024-03-26T03:54:38.372985+0000","last_fullsized":"2024-03-26T03:54:38.372985+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.399795+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.376610+0000","last_change":"2024-03-26T03:54:29.376610+0000","last_active":"2024-03-26T03:54:29.376610+0000","last_peered":"2024-03-26T03:54:29.376610+0000","last_clean":"2024-03-26T03:54:29.376610+0000","last_became_active":"2024-03-26T03:54:29.324000+0000","last_became_peered":"2024-03-26T03:54:29.324000+0000","last_unstale":"2024-03-26T03:54:29.376610+0000","last_undegraded":"2024-03-26T03:54:29.376610+0000","last_fullsized":"2024-03-26T03:54:29.376610+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:13.785265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.785265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":248,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.552098+0000","last_change":"2024-03-26T03:54:29.552098+0000","last_active":"2024-03-26T03:54:29.552098+0000","last_peered":"2024-03-26T03:54:29.552098+0000","last_clean":"2024-03-26T03:54:29.552098+0000","last_became_active":"2024-03-26T03:54:29.521665+0000","last_became_peered":"2024-03-26T03:54:29.521665+0000","last_unstale":"2024-03-26T03:54:29.552098+0000","last_undegraded":"2024-03-26T03:54:29.552098+0000","last_fullsized":"2024-03-26T03:54:29.552098+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:52:42.591503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.591503+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":192,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.257222+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:54:29.257222+0000","last_peered":"2024-03-26T03:54:29.257222+0000","last_clean":"2024-03-26T03:54:29.257222+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:54:29.257222+0000","last_undegraded":"2024-03-26T03:54:29.257222+0000","last_fullsized":"2024-03-26T03:54:29.257222+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.278597+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:54:29.278597+0000","last_peered":"2024-03-26T03:54:29.278597+0000","last_clean":"2024-03-26T03:54:29.278597+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:54:29.278597+0000","last_undegraded":"2024-03-26T03:54:29.278597+0000","last_fullsized":"2024-03-26T03:54:29.278597+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.699157+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:31.825452+0000","last_change":"2024-03-26T03:54:31.825491+0000","last_active":"2024-03-26T03:54:31.825452+0000","last_peered":"2024-03-26T03:54:31.825452+0000","last_clean":"2024-03-26T03:54:31.825452+0000","last_became_active":"2024-03-26T03:54:29.453692+0000","last_became_peered":"2024-03-26T03:54:29.453692+0000","last_unstale":"2024-03-26T03:54:31.825452+0000","last_undegraded":"2024-03-26T03:54:31.825452+0000","last_fullsized":"2024-03-26T03:54:31.825452+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.059388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":213,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378122+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:54:38.378122+0000","last_peered":"2024-03-26T03:54:38.378122+0000","last_clean":"2024-03-26T03:54:38.378122+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:54:38.378122+0000","last_undegraded":"2024-03-26T03:54:38.378122+0000","last_fullsized":"2024-03-26T03:54:38.378122+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.817806+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":205,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785266+0000","last_change":"2024-03-26T03:54:19.645849+0000","last_active":"2024-03-26T03:54:38.785266+0000","last_peered":"2024-03-26T03:54:38.785266+0000","last_clean":"2024-03-26T03:54:38.785266+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:54:38.785266+0000","last_undegraded":"2024-03-26T03:54:38.785266+0000","last_fullsized":"2024-03-26T03:54:38.785266+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:19.645746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.645746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:45.723244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":147,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.371853+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:54:38.371853+0000","last_peered":"2024-03-26T03:54:38.371853+0000","last_clean":"2024-03-26T03:54:38.371853+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:54:38.371853+0000","last_undegraded":"2024-03-26T03:54:38.371853+0000","last_fullsized":"2024-03-26T03:54:38.371853+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.501238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":211,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.518528+0000","last_change":"2024-03-26T03:54:29.518528+0000","last_active":"2024-03-26T03:54:29.518528+0000","last_peered":"2024-03-26T03:54:29.518528+0000","last_clean":"2024-03-26T03:54:29.518528+0000","last_became_active":"2024-03-26T03:54:29.518254+0000","last_became_peered":"2024-03-26T03:54:29.518254+0000","last_unstale":"2024-03-26T03:54:29.518528+0000","last_undegraded":"2024-03-26T03:54:29.518528+0000","last_fullsized":"2024-03-26T03:54:29.518528+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.479576+0000","last_change":"2024-03-26T03:54:29.394685+0000","last_active":"2024-03-26T03:54:29.479576+0000","last_peered":"2024-03-26T03:54:29.479576+0000","last_clean":"2024-03-26T03:54:29.479576+0000","last_became_active":"2024-03-26T03:54:29.345582+0000","last_became_peered":"2024-03-26T03:54:29.345582+0000","last_unstale":"2024-03-26T03:54:29.479576+0000","last_undegraded":"2024-03-26T03:54:29.479576+0000","last_fullsized":"2024-03-26T03:54:29.479576+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:44.279689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":78,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.411473+0000","last_change":"2024-03-26T03:54:29.400638+0000","last_active":"2024-03-26T03:54:29.411473+0000","last_peered":"2024-03-26T03:54:29.411473+0000","last_clean":"2024-03-26T03:54:29.411473+0000","last_became_active":"2024-03-26T03:54:29.361440+0000","last_became_peered":"2024-03-26T03:54:29.361440+0000","last_unstale":"2024-03-26T03:54:29.411473+0000","last_undegraded":"2024-03-26T03:54:29.411473+0000","last_fullsized":"2024-03-26T03:54:29.411473+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.352186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.352186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784973+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:54:38.784973+0000","last_peered":"2024-03-26T03:54:38.784973+0000","last_clean":"2024-03-26T03:54:38.784973+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:54:38.784973+0000","last_undegraded":"2024-03-26T03:54:38.784973+0000","last_fullsized":"2024-03-26T03:54:38.784973+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782830+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":225,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377705+0000","last_change":"2024-03-26T03:54:35.798476+0000","last_active":"2024-03-26T03:54:38.377705+0000","last_peered":"2024-03-26T03:54:38.377705+0000","last_clean":"2024-03-26T03:54:38.377705+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:54:38.377705+0000","last_undegraded":"2024-03-26T03:54:38.377705+0000","last_fullsized":"2024-03-26T03:54:38.377705+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:57.886093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":117,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785117+0000","last_change":"2024-03-26T03:54:29.548373+0000","last_active":"2024-03-26T03:54:38.785117+0000","last_peered":"2024-03-26T03:54:38.785117+0000","last_clean":"2024-03-26T03:54:38.785117+0000","last_became_active":"2024-03-26T03:54:29.357737+0000","last_became_peered":"2024-03-26T03:54:29.357737+0000","last_unstale":"2024-03-26T03:54:38.785117+0000","last_undegraded":"2024-03-26T03:54:38.785117+0000","last_fullsized":"2024-03-26T03:54:38.785117+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.783027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":201,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.243489+0000","last_change":"2024-03-26T03:54:28.830813+0000","last_active":"2024-03-26T03:54:29.243489+0000","last_peered":"2024-03-26T03:54:29.243489+0000","last_clean":"2024-03-26T03:54:29.243489+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:54:29.243489+0000","last_undegraded":"2024-03-26T03:54:29.243489+0000","last_fullsized":"2024-03-26T03:54:29.243489+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.572394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.400093+0000","last_change":"2024-03-26T03:54:29.348291+0000","last_active":"2024-03-26T03:54:29.400093+0000","last_peered":"2024-03-26T03:54:29.400093+0000","last_clean":"2024-03-26T03:54:29.400093+0000","last_became_active":"2024-03-26T03:54:29.344062+0000","last_became_peered":"2024-03-26T03:54:29.344062+0000","last_unstale":"2024-03-26T03:54:29.400093+0000","last_undegraded":"2024-03-26T03:54:29.400093+0000","last_fullsized":"2024-03-26T03:54:29.400093+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785871+0000","last_change":"2024-03-26T03:54:29.530213+0000","last_active":"2024-03-26T03:54:38.785871+0000","last_peered":"2024-03-26T03:54:38.785871+0000","last_clean":"2024-03-26T03:54:38.785871+0000","last_became_active":"2024-03-26T03:54:29.529851+0000","last_became_peered":"2024-03-26T03:54:29.529851+0000","last_unstale":"2024-03-26T03:54:38.785871+0000","last_undegraded":"2024-03-26T03:54:38.785871+0000","last_fullsized":"2024-03-26T03:54:38.785871+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.367821+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"152'13","reported_seq":244,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.277907+0000","last_change":"2024-03-26T03:54:26.861839+0000","last_active":"2024-03-26T03:54:29.277907+0000","last_peered":"2024-03-26T03:54:29.277907+0000","last_clean":"2024-03-26T03:54:29.277907+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:54:29.277907+0000","last_undegraded":"2024-03-26T03:54:29.277907+0000","last_fullsized":"2024-03-26T03:54:29.277907+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:54:26.861674+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:28.880700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.305160+0000","last_change":"2024-03-26T03:54:29.304012+0000","last_active":"2024-03-26T03:54:29.305160+0000","last_peered":"2024-03-26T03:54:29.305160+0000","last_clean":"2024-03-26T03:54:29.305160+0000","last_became_active":"2024-03-26T03:54:29.303365+0000","last_became_peered":"2024-03-26T03:54:29.303365+0000","last_unstale":"2024-03-26T03:54:29.305160+0000","last_undegraded":"2024-03-26T03:54:29.305160+0000","last_fullsized":"2024-03-26T03:54:29.305160+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:41.432332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:41.432332+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":202,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.323517+0000","last_change":"2024-03-26T03:54:29.323517+0000","last_active":"2024-03-26T03:54:29.323517+0000","last_peered":"2024-03-26T03:54:29.323517+0000","last_clean":"2024-03-26T03:54:29.323517+0000","last_became_active":"2024-03-26T03:54:29.319822+0000","last_became_peered":"2024-03-26T03:54:29.319822+0000","last_unstale":"2024-03-26T03:54:29.323517+0000","last_undegraded":"2024-03-26T03:54:29.323517+0000","last_fullsized":"2024-03-26T03:54:29.323517+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372443+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:54:38.372443+0000","last_peered":"2024-03-26T03:54:38.372443+0000","last_clean":"2024-03-26T03:54:38.372443+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:54:38.372443+0000","last_undegraded":"2024-03-26T03:54:38.372443+0000","last_fullsized":"2024-03-26T03:54:38.372443+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.223788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":242,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372707+0000","last_change":"2024-03-26T03:53:33.392149+0000","last_active":"2024-03-26T03:54:38.372707+0000","last_peered":"2024-03-26T03:54:38.372707+0000","last_clean":"2024-03-26T03:54:38.372707+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:54:38.372707+0000","last_undegraded":"2024-03-26T03:54:38.372707+0000","last_fullsized":"2024-03-26T03:54:38.372707+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:51.222885+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":199,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.521980+0000","last_change":"2024-03-26T03:54:29.521980+0000","last_active":"2024-03-26T03:54:29.521980+0000","last_peered":"2024-03-26T03:54:29.521980+0000","last_clean":"2024-03-26T03:54:29.521980+0000","last_became_active":"2024-03-26T03:54:29.520018+0000","last_became_peered":"2024-03-26T03:54:29.520018+0000","last_unstale":"2024-03-26T03:54:29.521980+0000","last_undegraded":"2024-03-26T03:54:29.521980+0000","last_fullsized":"2024-03-26T03:54:29.521980+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:53.774450+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":289,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378077+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:54:38.378077+0000","last_peered":"2024-03-26T03:54:38.378077+0000","last_clean":"2024-03-26T03:54:38.378077+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:54:38.378077+0000","last_undegraded":"2024-03-26T03:54:38.378077+0000","last_fullsized":"2024-03-26T03:54:38.378077+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.817782+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":217,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785289+0000","last_change":"2024-03-26T03:53:20.739662+0000","last_active":"2024-03-26T03:54:38.785289+0000","last_peered":"2024-03-26T03:54:38.785289+0000","last_clean":"2024-03-26T03:54:38.785289+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:54:38.785289+0000","last_undegraded":"2024-03-26T03:54:38.785289+0000","last_fullsized":"2024-03-26T03:54:38.785289+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.739547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.739547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782952+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":94,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:32.911353+0000","last_change":"2024-03-26T03:54:32.911388+0000","last_active":"2024-03-26T03:54:32.911353+0000","last_peered":"2024-03-26T03:54:32.911353+0000","last_clean":"2024-03-26T03:54:32.911353+0000","last_became_active":"2024-03-26T03:54:29.390009+0000","last_became_peered":"2024-03-26T03:54:29.390009+0000","last_unstale":"2024-03-26T03:54:32.911353+0000","last_undegraded":"2024-03-26T03:54:32.911353+0000","last_fullsized":"2024-03-26T03:54:32.911353+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:59.591186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":205,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373057+0000","last_change":"2024-03-26T03:54:15.344511+0000","last_active":"2024-03-26T03:54:38.373057+0000","last_peered":"2024-03-26T03:54:38.373057+0000","last_clean":"2024-03-26T03:54:38.373057+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:54:38.373057+0000","last_undegraded":"2024-03-26T03:54:38.373057+0000","last_fullsized":"2024-03-26T03:54:38.373057+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:20.100911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":48,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786788+0000","last_change":"2024-03-26T03:54:29.542214+0000","last_active":"2024-03-26T03:54:38.786788+0000","last_peered":"2024-03-26T03:54:38.786788+0000","last_clean":"2024-03-26T03:54:38.786788+0000","last_became_active":"2024-03-26T03:54:29.389636+0000","last_became_peered":"2024-03-26T03:54:29.389636+0000","last_unstale":"2024-03-26T03:54:38.786788+0000","last_undegraded":"2024-03-26T03:54:38.786788+0000","last_fullsized":"2024-03-26T03:54:38.786788+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":231,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.455442+0000","last_change":"2024-03-26T03:54:29.455442+0000","last_active":"2024-03-26T03:54:29.455442+0000","last_peered":"2024-03-26T03:54:29.455442+0000","last_clean":"2024-03-26T03:54:29.455442+0000","last_became_active":"2024-03-26T03:54:29.345136+0000","last_became_peered":"2024-03-26T03:54:29.345136+0000","last_unstale":"2024-03-26T03:54:29.455442+0000","last_undegraded":"2024-03-26T03:54:29.455442+0000","last_fullsized":"2024-03-26T03:54:29.455442+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:53:12.600988+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:12.600988+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:32.347923+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":264,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378354+0000","last_change":"2024-03-26T03:54:29.550761+0000","last_active":"2024-03-26T03:54:38.378354+0000","last_peered":"2024-03-26T03:54:38.378354+0000","last_clean":"2024-03-26T03:54:38.378354+0000","last_became_active":"2024-03-26T03:54:29.522025+0000","last_became_peered":"2024-03-26T03:54:29.522025+0000","last_unstale":"2024-03-26T03:54:38.378354+0000","last_undegraded":"2024-03-26T03:54:38.378354+0000","last_fullsized":"2024-03-26T03:54:38.378354+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:41.636278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":203,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.248798+0000","last_change":"2024-03-26T03:53:18.332085+0000","last_active":"2024-03-26T03:54:29.248798+0000","last_peered":"2024-03-26T03:54:29.248798+0000","last_clean":"2024-03-26T03:54:29.248798+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:54:29.248798+0000","last_undegraded":"2024-03-26T03:54:29.248798+0000","last_fullsized":"2024-03-26T03:54:29.248798+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:53:18.331943+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.331943+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.532516+0000","last_change":"2024-03-26T03:54:29.532516+0000","last_active":"2024-03-26T03:54:29.532516+0000","last_peered":"2024-03-26T03:54:29.532516+0000","last_clean":"2024-03-26T03:54:29.532516+0000","last_became_active":"2024-03-26T03:54:29.532138+0000","last_became_peered":"2024-03-26T03:54:29.532138+0000","last_unstale":"2024-03-26T03:54:29.532516+0000","last_undegraded":"2024-03-26T03:54:29.532516+0000","last_fullsized":"2024-03-26T03:54:29.532516+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:23.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784920+0000","last_change":"2024-03-26T03:53:29.687145+0000","last_active":"2024-03-26T03:54:38.784920+0000","last_peered":"2024-03-26T03:54:38.784920+0000","last_clean":"2024-03-26T03:54:38.784920+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:54:38.784920+0000","last_undegraded":"2024-03-26T03:54:38.784920+0000","last_fullsized":"2024-03-26T03:54:38.784920+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.687067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.687067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.783056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":261,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.542309+0000","last_change":"2024-03-26T03:54:29.542309+0000","last_active":"2024-03-26T03:54:29.542309+0000","last_peered":"2024-03-26T03:54:29.542309+0000","last_clean":"2024-03-26T03:54:29.542309+0000","last_became_active":"2024-03-26T03:54:29.521088+0000","last_became_peered":"2024-03-26T03:54:29.521088+0000","last_unstale":"2024-03-26T03:54:29.542309+0000","last_undegraded":"2024-03-26T03:54:29.542309+0000","last_fullsized":"2024-03-26T03:54:29.542309+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785589+0000","last_change":"2024-03-26T03:53:17.657919+0000","last_active":"2024-03-26T03:54:38.785589+0000","last_peered":"2024-03-26T03:54:38.785589+0000","last_clean":"2024-03-26T03:54:38.785589+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:54:38.785589+0000","last_undegraded":"2024-03-26T03:54:38.785589+0000","last_fullsized":"2024-03-26T03:54:38.785589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.376109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377565+0000","last_change":"2024-03-26T03:54:29.288599+0000","last_active":"2024-03-26T03:54:38.377565+0000","last_peered":"2024-03-26T03:54:38.377565+0000","last_clean":"2024-03-26T03:54:38.377565+0000","last_became_active":"2024-03-26T03:54:29.288159+0000","last_became_peered":"2024-03-26T03:54:29.288159+0000","last_unstale":"2024-03-26T03:54:38.377565+0000","last_undegraded":"2024-03-26T03:54:38.377565+0000","last_fullsized":"2024-03-26T03:54:38.377565+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.147231+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":149,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.454612+0000","last_change":"2024-03-26T03:54:29.454612+0000","last_active":"2024-03-26T03:54:29.454612+0000","last_peered":"2024-03-26T03:54:29.454612+0000","last_clean":"2024-03-26T03:54:29.454612+0000","last_became_active":"2024-03-26T03:54:29.454092+0000","last_became_peered":"2024-03-26T03:54:29.454092+0000","last_unstale":"2024-03-26T03:54:29.454612+0000","last_undegraded":"2024-03-26T03:54:29.454612+0000","last_fullsized":"2024-03-26T03:54:29.454612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:17.213229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":67,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786274+0000","last_change":"2024-03-26T03:54:29.485628+0000","last_active":"2024-03-26T03:54:38.786274+0000","last_peered":"2024-03-26T03:54:38.786274+0000","last_clean":"2024-03-26T03:54:38.786274+0000","last_became_active":"2024-03-26T03:54:29.289163+0000","last_became_peered":"2024-03-26T03:54:29.289163+0000","last_unstale":"2024-03-26T03:54:38.786274+0000","last_undegraded":"2024-03-26T03:54:38.786274+0000","last_fullsized":"2024-03-26T03:54:38.786274+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.382229+0000","last_change":"2024-03-26T03:54:38.381911+0000","last_active":"2024-03-26T03:54:38.382229+0000","last_peered":"2024-03-26T03:54:38.382229+0000","last_clean":"2024-03-26T03:54:38.382229+0000","last_became_active":"2024-03-26T03:54:38.381579+0000","last_became_peered":"2024-03-26T03:54:38.381579+0000","last_unstale":"2024-03-26T03:54:38.382229+0000","last_undegraded":"2024-03-26T03:54:38.382229+0000","last_fullsized":"2024-03-26T03:54:38.382229+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:59.528290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":219,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.376333+0000","last_change":"2024-03-26T03:54:10.731222+0000","last_active":"2024-03-26T03:54:38.376333+0000","last_peered":"2024-03-26T03:54:38.376333+0000","last_clean":"2024-03-26T03:54:38.376333+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:54:38.376333+0000","last_undegraded":"2024-03-26T03:54:38.376333+0000","last_fullsized":"2024-03-26T03:54:38.376333+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:37.665420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":229,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786696+0000","last_change":"2024-03-26T03:54:16.696733+0000","last_active":"2024-03-26T03:54:38.786696+0000","last_peered":"2024-03-26T03:54:38.786696+0000","last_clean":"2024-03-26T03:54:38.786696+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:54:38.786696+0000","last_undegraded":"2024-03-26T03:54:38.786696+0000","last_fullsized":"2024-03-26T03:54:38.786696+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:54:16.696602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.696602+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.478810+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":288,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.375411+0000","last_change":"2024-03-26T03:54:19.799307+0000","last_active":"2024-03-26T03:54:38.375411+0000","last_peered":"2024-03-26T03:54:38.375411+0000","last_clean":"2024-03-26T03:54:38.375411+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:54:38.375411+0000","last_undegraded":"2024-03-26T03:54:38.375411+0000","last_fullsized":"2024-03-26T03:54:38.375411+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:35.017235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":299,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785160+0000","last_change":"2024-03-26T03:53:19.706639+0000","last_active":"2024-03-26T03:54:38.785160+0000","last_peered":"2024-03-26T03:54:38.785160+0000","last_clean":"2024-03-26T03:54:38.785160+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:54:38.785160+0000","last_undegraded":"2024-03-26T03:54:38.785160+0000","last_fullsized":"2024-03-26T03:54:38.785160+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:19.706557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.706557+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782920+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.787219+0000","last_change":"2024-03-26T03:54:38.786252+0000","last_active":"2024-03-26T03:54:38.787219+0000","last_peered":"2024-03-26T03:54:38.787219+0000","last_clean":"2024-03-26T03:54:38.787219+0000","last_became_active":"2024-03-26T03:54:38.785773+0000","last_became_peered":"2024-03-26T03:54:38.785773+0000","last_unstale":"2024-03-26T03:54:38.787219+0000","last_undegraded":"2024-03-26T03:54:38.787219+0000","last_fullsized":"2024-03-26T03:54:38.787219+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:00.591678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":25,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.785388+0000","last_change":"2024-03-26T03:54:30.757009+0000","last_active":"2024-03-26T03:54:38.785388+0000","last_peered":"2024-03-26T03:54:38.785388+0000","last_clean":"2024-03-26T03:54:38.785388+0000","last_became_active":"2024-03-26T03:54:29.304565+0000","last_became_peered":"2024-03-26T03:54:29.304565+0000","last_unstale":"2024-03-26T03:54:38.785388+0000","last_undegraded":"2024-03-26T03:54:38.785388+0000","last_fullsized":"2024-03-26T03:54:38.785388+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.498968+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":217,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786428+0000","last_change":"2024-03-26T03:54:18.653671+0000","last_active":"2024-03-26T03:54:38.786428+0000","last_peered":"2024-03-26T03:54:38.786428+0000","last_clean":"2024-03-26T03:54:38.786428+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:54:38.786428+0000","last_undegraded":"2024-03-26T03:54:38.786428+0000","last_fullsized":"2024-03-26T03:54:38.786428+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.653555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.653555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:20.217041+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":274,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373046+0000","last_change":"2024-03-26T03:54:29.541247+0000","last_active":"2024-03-26T03:54:38.373046+0000","last_peered":"2024-03-26T03:54:38.373046+0000","last_clean":"2024-03-26T03:54:38.373046+0000","last_became_active":"2024-03-26T03:54:29.346977+0000","last_became_peered":"2024-03-26T03:54:29.346977+0000","last_unstale":"2024-03-26T03:54:38.373046+0000","last_undegraded":"2024-03-26T03:54:38.373046+0000","last_fullsized":"2024-03-26T03:54:38.373046+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.245333+0000","last_change":"2024-03-26T03:53:19.352440+0000","last_active":"2024-03-26T03:54:29.245333+0000","last_peered":"2024-03-26T03:54:29.245333+0000","last_clean":"2024-03-26T03:54:29.245333+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:54:29.245333+0000","last_undegraded":"2024-03-26T03:54:29.245333+0000","last_fullsized":"2024-03-26T03:54:29.245333+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:19.352319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.352319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:45.533994+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":211,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784632+0000","last_change":"2024-03-26T03:54:29.359240+0000","last_active":"2024-03-26T03:54:38.784632+0000","last_peered":"2024-03-26T03:54:38.784632+0000","last_clean":"2024-03-26T03:54:38.784632+0000","last_became_active":"2024-03-26T03:54:29.358827+0000","last_became_peered":"2024-03-26T03:54:29.358827+0000","last_unstale":"2024-03-26T03:54:38.784632+0000","last_undegraded":"2024-03-26T03:54:38.784632+0000","last_fullsized":"2024-03-26T03:54:38.784632+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.664206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.664206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.783136+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.249057+0000","last_change":"2024-03-26T03:53:35.307506+0000","last_active":"2024-03-26T03:54:29.249057+0000","last_peered":"2024-03-26T03:54:29.249057+0000","last_clean":"2024-03-26T03:54:29.249057+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:54:29.249057+0000","last_undegraded":"2024-03-26T03:54:29.249057+0000","last_fullsized":"2024-03-26T03:54:29.249057+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:35.307371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:35.307371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:37.812565+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":229,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786287+0000","last_change":"2024-03-26T03:54:29.356025+0000","last_active":"2024-03-26T03:54:38.786287+0000","last_peered":"2024-03-26T03:54:38.786287+0000","last_clean":"2024-03-26T03:54:38.786287+0000","last_became_active":"2024-03-26T03:54:29.355742+0000","last_became_peered":"2024-03-26T03:54:29.355742+0000","last_unstale":"2024-03-26T03:54:38.786287+0000","last_undegraded":"2024-03-26T03:54:38.786287+0000","last_fullsized":"2024-03-26T03:54:38.786287+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.612556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.612556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:25.010157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.276741+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:54:29.276741+0000","last_peered":"2024-03-26T03:54:29.276741+0000","last_clean":"2024-03-26T03:54:29.276741+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:54:29.276741+0000","last_undegraded":"2024-03-26T03:54:29.276741+0000","last_fullsized":"2024-03-26T03:54:29.276741+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":193,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.244527+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:54:29.244527+0000","last_peered":"2024-03-26T03:54:29.244527+0000","last_clean":"2024-03-26T03:54:29.244527+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:54:29.244527+0000","last_undegraded":"2024-03-26T03:54:29.244527+0000","last_fullsized":"2024-03-26T03:54:29.244527+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:33.962429+0000","last_change":"2024-03-26T03:54:33.962451+0000","last_active":"2024-03-26T03:54:33.962429+0000","last_peered":"2024-03-26T03:54:33.962429+0000","last_clean":"2024-03-26T03:54:33.962429+0000","last_became_active":"2024-03-26T03:54:29.428035+0000","last_became_peered":"2024-03-26T03:54:29.428035+0000","last_unstale":"2024-03-26T03:54:33.962429+0000","last_undegraded":"2024-03-26T03:54:33.962429+0000","last_fullsized":"2024-03-26T03:54:33.962429+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.962364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:46.314598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":237,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377473+0000","last_change":"2024-03-26T03:54:29.320669+0000","last_active":"2024-03-26T03:54:38.377473+0000","last_peered":"2024-03-26T03:54:38.377473+0000","last_clean":"2024-03-26T03:54:38.377473+0000","last_became_active":"2024-03-26T03:54:29.320372+0000","last_became_peered":"2024-03-26T03:54:29.320372+0000","last_unstale":"2024-03-26T03:54:38.377473+0000","last_undegraded":"2024-03-26T03:54:38.377473+0000","last_fullsized":"2024-03-26T03:54:38.377473+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:21.839296+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":255,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.247677+0000","last_change":"2024-03-26T03:54:17.818278+0000","last_active":"2024-03-26T03:54:29.247677+0000","last_peered":"2024-03-26T03:54:29.247677+0000","last_clean":"2024-03-26T03:54:29.247677+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:54:29.247677+0000","last_undegraded":"2024-03-26T03:54:29.247677+0000","last_fullsized":"2024-03-26T03:54:29.247677+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.752191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.787526+0000","last_change":"2024-03-26T03:54:38.787090+0000","last_active":"2024-03-26T03:54:38.787526+0000","last_peered":"2024-03-26T03:54:38.787526+0000","last_clean":"2024-03-26T03:54:38.787526+0000","last_became_active":"2024-03-26T03:54:38.786690+0000","last_became_peered":"2024-03-26T03:54:38.786690+0000","last_unstale":"2024-03-26T03:54:38.787526+0000","last_undegraded":"2024-03-26T03:54:38.787526+0000","last_fullsized":"2024-03-26T03:54:38.787526+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:37.421814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":25,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372705+0000","last_change":"2024-03-26T03:54:32.411719+0000","last_active":"2024-03-26T03:54:38.372705+0000","last_peered":"2024-03-26T03:54:38.372705+0000","last_clean":"2024-03-26T03:54:38.372705+0000","last_became_active":"2024-03-26T03:54:29.297355+0000","last_became_peered":"2024-03-26T03:54:29.297355+0000","last_unstale":"2024-03-26T03:54:38.372705+0000","last_undegraded":"2024-03-26T03:54:38.372705+0000","last_fullsized":"2024-03-26T03:54:38.372705+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:40.760867+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":238,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.376014+0000","last_change":"2024-03-26T03:53:37.581465+0000","last_active":"2024-03-26T03:54:38.376014+0000","last_peered":"2024-03-26T03:54:38.376014+0000","last_clean":"2024-03-26T03:54:38.376014+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:54:38.376014+0000","last_undegraded":"2024-03-26T03:54:38.376014+0000","last_fullsized":"2024-03-26T03:54:38.376014+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.527591+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":215,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372246+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:54:38.372246+0000","last_peered":"2024-03-26T03:54:38.372246+0000","last_clean":"2024-03-26T03:54:38.372246+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:54:38.372246+0000","last_undegraded":"2024-03-26T03:54:38.372246+0000","last_fullsized":"2024-03-26T03:54:38.372246+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"154'35","reported_seq":369,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.485688+0000","last_change":"2024-03-26T03:54:29.460582+0000","last_active":"2024-03-26T03:54:29.485688+0000","last_peered":"2024-03-26T03:54:29.485688+0000","last_clean":"2024-03-26T03:54:29.485688+0000","last_became_active":"2024-03-26T03:54:29.371147+0000","last_became_peered":"2024-03-26T03:54:29.371147+0000","last_unstale":"2024-03-26T03:54:29.485688+0000","last_undegraded":"2024-03-26T03:54:29.485688+0000","last_fullsized":"2024-03-26T03:54:29.485688+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:46.129707+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.247749+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:54:29.247749+0000","last_peered":"2024-03-26T03:54:29.247749+0000","last_clean":"2024-03-26T03:54:29.247749+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:54:29.247749+0000","last_undegraded":"2024-03-26T03:54:29.247749+0000","last_fullsized":"2024-03-26T03:54:29.247749+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.155849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.383359+0000","last_change":"2024-03-26T03:54:38.382935+0000","last_active":"2024-03-26T03:54:38.383359+0000","last_peered":"2024-03-26T03:54:38.383359+0000","last_clean":"2024-03-26T03:54:38.383359+0000","last_became_active":"2024-03-26T03:54:38.382581+0000","last_became_peered":"2024-03-26T03:54:38.382581+0000","last_unstale":"2024-03-26T03:54:38.383359+0000","last_undegraded":"2024-03-26T03:54:38.383359+0000","last_fullsized":"2024-03-26T03:54:38.383359+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:51.379097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786434+0000","last_change":"2024-03-26T03:54:29.432774+0000","last_active":"2024-03-26T03:54:38.786434+0000","last_peered":"2024-03-26T03:54:38.786434+0000","last_clean":"2024-03-26T03:54:38.786434+0000","last_became_active":"2024-03-26T03:54:29.432431+0000","last_became_peered":"2024-03-26T03:54:29.432431+0000","last_unstale":"2024-03-26T03:54:38.786434+0000","last_undegraded":"2024-03-26T03:54:38.786434+0000","last_fullsized":"2024-03-26T03:54:38.786434+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782721+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":218,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.259030+0000","last_change":"2024-03-26T03:53:34.835708+0000","last_active":"2024-03-26T03:54:29.259030+0000","last_peered":"2024-03-26T03:54:29.259030+0000","last_clean":"2024-03-26T03:54:29.259030+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:54:29.259030+0000","last_undegraded":"2024-03-26T03:54:29.259030+0000","last_fullsized":"2024-03-26T03:54:29.259030+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.835581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.835581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:03.222485+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":236,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.333438+0000","last_change":"2024-03-26T03:54:29.333438+0000","last_active":"2024-03-26T03:54:29.333438+0000","last_peered":"2024-03-26T03:54:29.333438+0000","last_clean":"2024-03-26T03:54:29.333438+0000","last_became_active":"2024-03-26T03:54:29.319429+0000","last_became_peered":"2024-03-26T03:54:29.319429+0000","last_unstale":"2024-03-26T03:54:29.333438+0000","last_undegraded":"2024-03-26T03:54:29.333438+0000","last_fullsized":"2024-03-26T03:54:29.333438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.962798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.962798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.276435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":239,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.254814+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:54:29.254814+0000","last_peered":"2024-03-26T03:54:29.254814+0000","last_clean":"2024-03-26T03:54:29.254814+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:54:29.254814+0000","last_undegraded":"2024-03-26T03:54:29.254814+0000","last_fullsized":"2024-03-26T03:54:29.254814+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":205,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.311941+0000","last_change":"2024-03-26T03:53:32.388419+0000","last_active":"2024-03-26T03:54:29.311941+0000","last_peered":"2024-03-26T03:54:29.311941+0000","last_clean":"2024-03-26T03:54:29.311941+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:54:29.311941+0000","last_undegraded":"2024-03-26T03:54:29.311941+0000","last_fullsized":"2024-03-26T03:54:29.311941+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.387809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.387809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:39.271413+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.379285+0000","last_change":"2024-03-26T03:54:38.378983+0000","last_active":"2024-03-26T03:54:38.379285+0000","last_peered":"2024-03-26T03:54:38.379285+0000","last_clean":"2024-03-26T03:54:38.379285+0000","last_became_active":"2024-03-26T03:54:38.378668+0000","last_became_peered":"2024-03-26T03:54:38.378668+0000","last_unstale":"2024-03-26T03:54:38.379285+0000","last_undegraded":"2024-03-26T03:54:38.379285+0000","last_fullsized":"2024-03-26T03:54:38.379285+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:57.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":211,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377985+0000","last_change":"2024-03-26T03:54:29.268571+0000","last_active":"2024-03-26T03:54:38.377985+0000","last_peered":"2024-03-26T03:54:38.377985+0000","last_clean":"2024-03-26T03:54:38.377985+0000","last_became_active":"2024-03-26T03:54:29.268254+0000","last_became_peered":"2024-03-26T03:54:29.268254+0000","last_unstale":"2024-03-26T03:54:38.377985+0000","last_undegraded":"2024-03-26T03:54:38.377985+0000","last_fullsized":"2024-03-26T03:54:38.377985+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:51.669410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":473,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784814+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:54:38.784814+0000","last_peered":"2024-03-26T03:54:38.784814+0000","last_clean":"2024-03-26T03:54:38.784814+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:54:38.784814+0000","last_undegraded":"2024-03-26T03:54:38.784814+0000","last_fullsized":"2024-03-26T03:54:38.784814+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782881+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":86,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.376001+0000","last_change":"2024-03-26T03:54:29.287818+0000","last_active":"2024-03-26T03:54:38.376001+0000","last_peered":"2024-03-26T03:54:38.376001+0000","last_clean":"2024-03-26T03:54:38.376001+0000","last_became_active":"2024-03-26T03:54:29.287411+0000","last_became_peered":"2024-03-26T03:54:29.287411+0000","last_unstale":"2024-03-26T03:54:38.376001+0000","last_undegraded":"2024-03-26T03:54:38.376001+0000","last_fullsized":"2024-03-26T03:54:38.376001+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:29.508252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":247,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372859+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:54:38.372859+0000","last_peered":"2024-03-26T03:54:38.372859+0000","last_clean":"2024-03-26T03:54:38.372859+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:54:38.372859+0000","last_undegraded":"2024-03-26T03:54:38.372859+0000","last_fullsized":"2024-03-26T03:54:38.372859+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.399776+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786264+0000","last_change":"2024-03-26T03:53:28.678428+0000","last_active":"2024-03-26T03:54:38.786264+0000","last_peered":"2024-03-26T03:54:38.786264+0000","last_clean":"2024-03-26T03:54:38.786264+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:54:38.786264+0000","last_undegraded":"2024-03-26T03:54:38.786264+0000","last_fullsized":"2024-03-26T03:54:38.786264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.783085+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.371516+0000","last_change":"2024-03-26T03:54:29.371516+0000","last_active":"2024-03-26T03:54:29.371516+0000","last_peered":"2024-03-26T03:54:29.371516+0000","last_clean":"2024-03-26T03:54:29.371516+0000","last_became_active":"2024-03-26T03:54:29.370375+0000","last_became_peered":"2024-03-26T03:54:29.370375+0000","last_unstale":"2024-03-26T03:54:29.371516+0000","last_undegraded":"2024-03-26T03:54:29.371516+0000","last_fullsized":"2024-03-26T03:54:29.371516+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.348117+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.303701+0000","last_change":"2024-03-26T03:54:29.303701+0000","last_active":"2024-03-26T03:54:29.303701+0000","last_peered":"2024-03-26T03:54:29.303701+0000","last_clean":"2024-03-26T03:54:29.303701+0000","last_became_active":"2024-03-26T03:54:29.302729+0000","last_became_peered":"2024-03-26T03:54:29.302729+0000","last_unstale":"2024-03-26T03:54:29.303701+0000","last_undegraded":"2024-03-26T03:54:29.303701+0000","last_fullsized":"2024-03-26T03:54:29.303701+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:33.645447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.645447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.418212+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":576,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784696+0000","last_change":"2024-03-26T03:53:21.716954+0000","last_active":"2024-03-26T03:54:38.784696+0000","last_peered":"2024-03-26T03:54:38.784696+0000","last_clean":"2024-03-26T03:54:38.784696+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:54:38.784696+0000","last_undegraded":"2024-03-26T03:54:38.784696+0000","last_fullsized":"2024-03-26T03:54:38.784696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:53:21.716822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:21.716822+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782989+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":342,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":219,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.268683+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:54:29.268683+0000","last_peered":"2024-03-26T03:54:29.268683+0000","last_clean":"2024-03-26T03:54:29.268683+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:54:29.268683+0000","last_undegraded":"2024-03-26T03:54:29.268683+0000","last_fullsized":"2024-03-26T03:54:29.268683+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":209,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378222+0000","last_change":"2024-03-26T03:54:11.750761+0000","last_active":"2024-03-26T03:54:38.378222+0000","last_peered":"2024-03-26T03:54:38.378222+0000","last_clean":"2024-03-26T03:54:38.378222+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:54:38.378222+0000","last_undegraded":"2024-03-26T03:54:38.378222+0000","last_fullsized":"2024-03-26T03:54:38.378222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.167407+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":198,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.257155+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:54:29.257155+0000","last_peered":"2024-03-26T03:54:29.257155+0000","last_clean":"2024-03-26T03:54:29.257155+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:54:29.257155+0000","last_undegraded":"2024-03-26T03:54:29.257155+0000","last_fullsized":"2024-03-26T03:54:29.257155+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":212,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.256891+0000","last_change":"2024-03-26T03:53:31.824781+0000","last_active":"2024-03-26T03:54:29.256891+0000","last_peered":"2024-03-26T03:54:29.256891+0000","last_clean":"2024-03-26T03:54:29.256891+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:54:29.256891+0000","last_undegraded":"2024-03-26T03:54:29.256891+0000","last_fullsized":"2024-03-26T03:54:29.256891+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:34.207903+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":257,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.247083+0000","last_change":"2024-03-26T03:54:22.831334+0000","last_active":"2024-03-26T03:54:29.247083+0000","last_peered":"2024-03-26T03:54:29.247083+0000","last_clean":"2024-03-26T03:54:29.247083+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:54:29.247083+0000","last_undegraded":"2024-03-26T03:54:29.247083+0000","last_fullsized":"2024-03-26T03:54:29.247083+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:29.785367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":143,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.244081+0000","last_change":"2024-03-26T03:54:18.813803+0000","last_active":"2024-03-26T03:54:29.244081+0000","last_peered":"2024-03-26T03:54:29.244081+0000","last_clean":"2024-03-26T03:54:29.244081+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:54:29.244081+0000","last_undegraded":"2024-03-26T03:54:29.244081+0000","last_fullsized":"2024-03-26T03:54:29.244081+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:22.825611+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":199,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.362049+0000","last_change":"2024-03-26T03:54:29.362049+0000","last_active":"2024-03-26T03:54:29.362049+0000","last_peered":"2024-03-26T03:54:29.362049+0000","last_clean":"2024-03-26T03:54:29.362049+0000","last_became_active":"2024-03-26T03:54:29.344504+0000","last_became_peered":"2024-03-26T03:54:29.344504+0000","last_unstale":"2024-03-26T03:54:29.362049+0000","last_undegraded":"2024-03-26T03:54:29.362049+0000","last_fullsized":"2024-03-26T03:54:29.362049+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:30.464738+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378127+0000","last_change":"2024-03-26T03:54:29.432812+0000","last_active":"2024-03-26T03:54:38.378127+0000","last_peered":"2024-03-26T03:54:38.378127+0000","last_clean":"2024-03-26T03:54:38.378127+0000","last_became_active":"2024-03-26T03:54:29.432437+0000","last_became_peered":"2024-03-26T03:54:29.432437+0000","last_unstale":"2024-03-26T03:54:38.378127+0000","last_undegraded":"2024-03-26T03:54:38.378127+0000","last_fullsized":"2024-03-26T03:54:38.378127+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:23.984390+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":235,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.277248+0000","last_change":"2024-03-26T03:54:15.883651+0000","last_active":"2024-03-26T03:54:29.277248+0000","last_peered":"2024-03-26T03:54:29.277248+0000","last_clean":"2024-03-26T03:54:29.277248+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:54:29.277248+0000","last_undegraded":"2024-03-26T03:54:29.277248+0000","last_fullsized":"2024-03-26T03:54:29.277248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:27.995304+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"154'10","reported_seq":296,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.486228+0000","last_change":"2024-03-26T03:54:29.476464+0000","last_active":"2024-03-26T03:54:29.486228+0000","last_peered":"2024-03-26T03:54:29.486228+0000","last_clean":"2024-03-26T03:54:29.486228+0000","last_became_active":"2024-03-26T03:54:29.360917+0000","last_became_peered":"2024-03-26T03:54:29.360917+0000","last_unstale":"2024-03-26T03:54:29.486228+0000","last_undegraded":"2024-03-26T03:54:29.486228+0000","last_fullsized":"2024-03-26T03:54:29.486228+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:38.830501+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.787244+0000","last_change":"2024-03-26T03:54:38.786182+0000","last_active":"2024-03-26T03:54:38.787244+0000","last_peered":"2024-03-26T03:54:38.787244+0000","last_clean":"2024-03-26T03:54:38.787244+0000","last_became_active":"2024-03-26T03:54:38.785725+0000","last_became_peered":"2024-03-26T03:54:38.785725+0000","last_unstale":"2024-03-26T03:54:38.787244+0000","last_undegraded":"2024-03-26T03:54:38.787244+0000","last_fullsized":"2024-03-26T03:54:38.787244+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:48.243843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":211,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.787101+0000","last_change":"2024-03-26T03:53:34.666684+0000","last_active":"2024-03-26T03:54:38.787101+0000","last_peered":"2024-03-26T03:54:38.787101+0000","last_clean":"2024-03-26T03:54:38.787101+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:54:38.787101+0000","last_undegraded":"2024-03-26T03:54:38.787101+0000","last_fullsized":"2024-03-26T03:54:38.787101+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.666607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.666607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:51.753812+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":214,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.268789+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:54:29.268789+0000","last_peered":"2024-03-26T03:54:29.268789+0000","last_clean":"2024-03-26T03:54:29.268789+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:54:29.268789+0000","last_undegraded":"2024-03-26T03:54:29.268789+0000","last_fullsized":"2024-03-26T03:54:29.268789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:48.854398+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":270,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372746+0000","last_change":"2024-03-26T03:54:16.339624+0000","last_active":"2024-03-26T03:54:38.372746+0000","last_peered":"2024-03-26T03:54:38.372746+0000","last_clean":"2024-03-26T03:54:38.372746+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:54:38.372746+0000","last_undegraded":"2024-03-26T03:54:38.372746+0000","last_fullsized":"2024-03-26T03:54:38.372746+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:36.150917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"154'9","reported_seq":43,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.525902+0000","last_change":"2024-03-26T03:54:29.426592+0000","last_active":"2024-03-26T03:54:29.525902+0000","last_peered":"2024-03-26T03:54:29.525902+0000","last_clean":"2024-03-26T03:54:29.525902+0000","last_became_active":"2024-03-26T03:54:29.356520+0000","last_became_peered":"2024-03-26T03:54:29.356520+0000","last_unstale":"2024-03-26T03:54:29.525902+0000","last_undegraded":"2024-03-26T03:54:29.525902+0000","last_fullsized":"2024-03-26T03:54:29.525902+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:31.483593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.483593+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":203,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372181+0000","last_change":"2024-03-26T03:54:29.432125+0000","last_active":"2024-03-26T03:54:38.372181+0000","last_peered":"2024-03-26T03:54:38.372181+0000","last_clean":"2024-03-26T03:54:38.372181+0000","last_became_active":"2024-03-26T03:54:29.431697+0000","last_became_peered":"2024-03-26T03:54:29.431697+0000","last_unstale":"2024-03-26T03:54:38.372181+0000","last_undegraded":"2024-03-26T03:54:38.372181+0000","last_fullsized":"2024-03-26T03:54:38.372181+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:40.101352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":16,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.786159+0000","last_change":"2024-03-26T03:54:29.531706+0000","last_active":"2024-03-26T03:54:38.786159+0000","last_peered":"2024-03-26T03:54:38.786159+0000","last_clean":"2024-03-26T03:54:38.786159+0000","last_became_active":"2024-03-26T03:54:29.531371+0000","last_became_peered":"2024-03-26T03:54:29.531371+0000","last_unstale":"2024-03-26T03:54:38.786159+0000","last_undegraded":"2024-03-26T03:54:38.786159+0000","last_fullsized":"2024-03-26T03:54:38.786159+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:43.782800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":142,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.784946+0000","last_change":"2024-03-26T03:54:29.536599+0000","last_active":"2024-03-26T03:54:38.784946+0000","last_peered":"2024-03-26T03:54:38.784946+0000","last_clean":"2024-03-26T03:54:38.784946+0000","last_became_active":"2024-03-26T03:54:29.358363+0000","last_became_peered":"2024-03-26T03:54:29.358363+0000","last_unstale":"2024-03-26T03:54:38.784946+0000","last_undegraded":"2024-03-26T03:54:38.784946+0000","last_fullsized":"2024-03-26T03:54:38.784946+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.928019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":207,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.371856+0000","last_change":"2024-03-26T03:54:29.347549+0000","last_active":"2024-03-26T03:54:38.371856+0000","last_peered":"2024-03-26T03:54:38.371856+0000","last_clean":"2024-03-26T03:54:38.371856+0000","last_became_active":"2024-03-26T03:54:29.346424+0000","last_became_peered":"2024-03-26T03:54:29.346424+0000","last_unstale":"2024-03-26T03:54:38.371856+0000","last_undegraded":"2024-03-26T03:54:38.371856+0000","last_fullsized":"2024-03-26T03:54:38.371856+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":205,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377895+0000","last_change":"2024-03-26T03:53:28.635973+0000","last_active":"2024-03-26T03:54:38.377895+0000","last_peered":"2024-03-26T03:54:38.377895+0000","last_clean":"2024-03-26T03:54:38.377895+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:54:38.377895+0000","last_undegraded":"2024-03-26T03:54:38.377895+0000","last_fullsized":"2024-03-26T03:54:38.377895+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:42.817872+0000","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":289,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373291+0000","last_change":"2024-03-26T03:54:33.386428+0000","last_active":"2024-03-26T03:54:38.373291+0000","last_peered":"2024-03-26T03:54:38.373291+0000","last_clean":"2024-03-26T03:54:38.373291+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:54:38.373291+0000","last_undegraded":"2024-03-26T03:54:38.373291+0000","last_fullsized":"2024-03-26T03:54:38.373291+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:01.105650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:31.922358+0000","last_change":"2024-03-26T03:54:31.922380+0000","last_active":"2024-03-26T03:54:31.922358+0000","last_peered":"2024-03-26T03:54:31.922358+0000","last_clean":"2024-03-26T03:54:31.922358+0000","last_became_active":"2024-03-26T03:54:29.377662+0000","last_became_peered":"2024-03-26T03:54:29.377662+0000","last_unstale":"2024-03-26T03:54:31.922358+0000","last_undegraded":"2024-03-26T03:54:31.922358+0000","last_fullsized":"2024-03-26T03:54:31.922358+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:33.087626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":255,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.386220+0000","last_change":"2024-03-26T03:54:29.386220+0000","last_active":"2024-03-26T03:54:29.386220+0000","last_peered":"2024-03-26T03:54:29.386220+0000","last_clean":"2024-03-26T03:54:29.386220+0000","last_became_active":"2024-03-26T03:54:29.299380+0000","last_became_peered":"2024-03-26T03:54:29.299380+0000","last_unstale":"2024-03-26T03:54:29.386220+0000","last_undegraded":"2024-03-26T03:54:29.386220+0000","last_fullsized":"2024-03-26T03:54:29.386220+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.835587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.835587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.048215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378415+0000","last_change":"2024-03-26T03:54:38.377692+0000","last_active":"2024-03-26T03:54:38.378415+0000","last_peered":"2024-03-26T03:54:38.378415+0000","last_clean":"2024-03-26T03:54:38.378415+0000","last_became_active":"2024-03-26T03:54:38.377276+0000","last_became_peered":"2024-03-26T03:54:38.377276+0000","last_unstale":"2024-03-26T03:54:38.378415+0000","last_undegraded":"2024-03-26T03:54:38.378415+0000","last_fullsized":"2024-03-26T03:54:38.378415+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:54.890332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":74,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.377170+0000","last_change":"2024-03-26T03:54:29.320001+0000","last_active":"2024-03-26T03:54:38.377170+0000","last_peered":"2024-03-26T03:54:38.377170+0000","last_clean":"2024-03-26T03:54:38.377170+0000","last_became_active":"2024-03-26T03:54:29.310202+0000","last_became_peered":"2024-03-26T03:54:29.310202+0000","last_unstale":"2024-03-26T03:54:38.377170+0000","last_undegraded":"2024-03-26T03:54:38.377170+0000","last_fullsized":"2024-03-26T03:54:38.377170+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:33.239133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":223,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.378186+0000","last_change":"2024-03-26T03:54:24.803551+0000","last_active":"2024-03-26T03:54:38.378186+0000","last_peered":"2024-03-26T03:54:38.378186+0000","last_clean":"2024-03-26T03:54:38.378186+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:54:38.378186+0000","last_undegraded":"2024-03-26T03:54:38.378186+0000","last_fullsized":"2024-03-26T03:54:38.378186+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.568745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":217,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.371958+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:54:38.371958+0000","last_peered":"2024-03-26T03:54:38.371958+0000","last_clean":"2024-03-26T03:54:38.371958+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:54:38.371958+0000","last_undegraded":"2024-03-26T03:54:38.371958+0000","last_fullsized":"2024-03-26T03:54:38.371958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.372670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":284,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373064+0000","last_change":"2024-03-26T03:54:29.355692+0000","last_active":"2024-03-26T03:54:38.373064+0000","last_peered":"2024-03-26T03:54:38.373064+0000","last_clean":"2024-03-26T03:54:38.373064+0000","last_became_active":"2024-03-26T03:54:29.270339+0000","last_became_peered":"2024-03-26T03:54:29.270339+0000","last_unstale":"2024-03-26T03:54:38.373064+0000","last_undegraded":"2024-03-26T03:54:38.373064+0000","last_fullsized":"2024-03-26T03:54:38.373064+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:44.906103+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-26T03:54:37.359520+0000","last_change":"2024-03-26T03:54:37.359520+0000","last_active":"2024-03-26T03:54:37.359520+0000","last_peered":"2024-03-26T03:54:37.359520+0000","last_clean":"2024-03-26T03:54:37.359520+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-26T03:54:37.359520+0000","last_undegraded":"2024-03-26T03:54:37.359520+0000","last_fullsized":"2024-03-26T03:54:37.359520+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":215,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:31.003513+0000","last_change":"2024-03-26T03:54:31.003536+0000","last_active":"2024-03-26T03:54:31.003513+0000","last_peered":"2024-03-26T03:54:31.003513+0000","last_clean":"2024-03-26T03:54:31.003513+0000","last_became_active":"2024-03-26T03:54:29.296384+0000","last_became_peered":"2024-03-26T03:54:29.296384+0000","last_unstale":"2024-03-26T03:54:31.003513+0000","last_undegraded":"2024-03-26T03:54:31.003513+0000","last_fullsized":"2024-03-26T03:54:31.003513+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.003442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.003442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.045451+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":204,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.262866+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:54:29.262866+0000","last_peered":"2024-03-26T03:54:29.262866+0000","last_clean":"2024-03-26T03:54:29.262866+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:54:29.262866+0000","last_undegraded":"2024-03-26T03:54:29.262866+0000","last_fullsized":"2024-03-26T03:54:29.262866+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:48.412746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":153,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372251+0000","last_change":"2024-03-26T03:54:29.289038+0000","last_active":"2024-03-26T03:54:38.372251+0000","last_peered":"2024-03-26T03:54:38.372251+0000","last_clean":"2024-03-26T03:54:38.372251+0000","last_became_active":"2024-03-26T03:54:29.283289+0000","last_became_peered":"2024-03-26T03:54:29.283289+0000","last_unstale":"2024-03-26T03:54:38.372251+0000","last_undegraded":"2024-03-26T03:54:38.372251+0000","last_fullsized":"2024-03-26T03:54:38.372251+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:36.547229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":241,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372963+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:54:38.372963+0000","last_peered":"2024-03-26T03:54:38.372963+0000","last_clean":"2024-03-26T03:54:38.372963+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:54:38.372963+0000","last_undegraded":"2024-03-26T03:54:38.372963+0000","last_fullsized":"2024-03-26T03:54:38.372963+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:15.733851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.383134+0000","last_change":"2024-03-26T03:54:38.382779+0000","last_active":"2024-03-26T03:54:38.383134+0000","last_peered":"2024-03-26T03:54:38.383134+0000","last_clean":"2024-03-26T03:54:38.383134+0000","last_became_active":"2024-03-26T03:54:38.382404+0000","last_became_peered":"2024-03-26T03:54:38.382404+0000","last_unstale":"2024-03-26T03:54:38.383134+0000","last_undegraded":"2024-03-26T03:54:38.383134+0000","last_fullsized":"2024-03-26T03:54:38.383134+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:03.270619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":205,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.373303+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:54:38.373303+0000","last_peered":"2024-03-26T03:54:38.373303+0000","last_clean":"2024-03-26T03:54:38.373303+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:54:38.373303+0000","last_undegraded":"2024-03-26T03:54:38.373303+0000","last_fullsized":"2024-03-26T03:54:38.373303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:32.239985+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":284,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:29.254753+0000","last_change":"2024-03-26T03:53:17.373928+0000","last_active":"2024-03-26T03:54:29.254753+0000","last_peered":"2024-03-26T03:54:29.254753+0000","last_clean":"2024-03-26T03:54:29.254753+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:54:29.254753+0000","last_undegraded":"2024-03-26T03:54:29.254753+0000","last_fullsized":"2024-03-26T03:54:29.254753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:53:17.373796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.373796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:40.915867+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":154,"state":"active+clean","last_fresh":"2024-03-26T03:54:33.815203+0000","last_change":"2024-03-26T03:54:33.815241+0000","last_active":"2024-03-26T03:54:33.815203+0000","last_peered":"2024-03-26T03:54:33.815203+0000","last_clean":"2024-03-26T03:54:33.815203+0000","last_became_active":"2024-03-26T03:54:29.399842+0000","last_became_peered":"2024-03-26T03:54:29.399842+0000","last_unstale":"2024-03-26T03:54:33.815203+0000","last_undegraded":"2024-03-26T03:54:33.815203+0000","last_fullsized":"2024-03-26T03:54:33.815203+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:37.376467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":256,"reported_epoch":156,"state":"active+clean","last_fresh":"2024-03-26T03:54:38.372866+0000","last_change":"2024-03-26T03:54:29.380642+0000","last_active":"2024-03-26T03:54:38.372866+0000","last_peered":"2024-03-26T03:54:38.372866+0000","last_clean":"2024-03-26T03:54:38.372866+0000","last_became_active":"2024-03-26T03:54:29.296174+0000","last_became_peered":"2024-03-26T03:54:29.296174+0000","last_unstale":"2024-03-26T03:54:38.372866+0000","last_undegraded":"2024-03-26T03:54:38.372866+0000","last_fullsized":"2024-03-26T03:54:38.372866+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":24,"acting":24,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":347,"num_read_kb":342,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":3152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070539,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104768,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640384,"statfs":{"total":95995035648,"available":95887753216,"internally_reserved":0,"allocated":1429504,"data_stored":1003448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574953,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140864,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604288,"statfs":{"total":95995035648,"available":95850790912,"internally_reserved":0,"allocated":1433600,"data_stored":996442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738478,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":149184,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93595968,"statfs":{"total":95995035648,"available":95842271232,"internally_reserved":0,"allocated":1429504,"data_stored":996394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140232,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93604920,"statfs":{"total":95995035648,"available":95851438080,"internally_reserved":0,"allocated":786432,"data_stored":403620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953562,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108344,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636808,"statfs":{"total":95995035648,"available":95884091392,"internally_reserved":0,"allocated":827392,"data_stored":417762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659220,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":98608,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":97790,"kb_avail":93646544,"statfs":{"total":95995035648,"available":95894061056,"internally_reserved":0,"allocated":827392,"data_stored":418240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":100137413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:54:44.024 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:44.026 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:44.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:44.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:44.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:44.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:44.221 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:44.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:44.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:44.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:44.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:44.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:44.556 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:44 smithi006 ceph-mon[37501]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:44.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:44 smithi006 ceph-mon[29446]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:44.556 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:44.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:44.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:44.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:44.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:44.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:44.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:44 smithi086 ceph-mon[32965]: pgmap v242: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:44.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:44.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:44.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:44.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:44.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:44.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:45.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:45.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:45.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:45.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:45.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:45.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:45.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:45.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:45.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:45.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:45.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:45.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:45.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:45.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:45.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:45.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:45.740 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:45.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:54:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:45.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:45.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:45.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:45.996 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:45.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:46.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:46.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:46.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:46.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:46.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:46.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:46.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:46.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:46.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:46.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:46.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:46.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:46 smithi006 ceph-mon[37501]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:54:46.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:46 smithi006 ceph-mon[29446]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:54:46.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:46.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:46.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:46.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:46 smithi086 ceph-mon[32965]: pgmap v243: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T03:54:46.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:46.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:46.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:46.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:47.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:47.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:47.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:47.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:47.283 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:47.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:47.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:54:47.436 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:54:47.457 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:54:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:54:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:54:47.459 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:54:47.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:47.534 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:47.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:47 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:54:47.555 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3441087006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:47.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:47 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:54:47.556 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3441087006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:47.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:47.659 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:47.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.786 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:47.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:47.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:47 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:54:47.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3441087006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:47.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:47.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:47.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:48.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:48.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:48.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:48.160 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:48.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:48.285 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:48.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:48.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:48.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:48.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:48.507 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-26T03:54:48.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:48.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:48.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:48.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:48.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:48.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:48.753 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:54:48.753 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1569510602' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.754 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1569510602' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.755 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.755 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.755 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:48 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:48.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: pgmap v244: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:54:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1569510602' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:54:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:48 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:48.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:48.913 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:48.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:48.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:49.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:49.024 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:54:49.024 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:54:49.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:49.061 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:49.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:49.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:49.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:49.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:49.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:49.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:49.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:49.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:49.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:49.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:49.742 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[29446]: osdmap e158: 8 total, 8 up, 6 in 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[37501]: osdmap e158: 8 total, 8 up, 6 in 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:54:49.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:49 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:49.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:49.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:49 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:54:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:49 smithi086 ceph-mon[32965]: osdmap e158: 8 total, 8 up, 6 in 2024-03-26T03:54:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:49 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:54:49.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:49 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:54:49.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:49 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:54:49.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:49.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:49.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:49.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:50.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:50.048 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:50.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:50.174 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:50.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:50.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:50.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:50.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:50.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:50.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:50.523 DEBUG:teuthology.orchestra.run.smithi006:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-26T03:54:50.523 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T03:54:50.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:54:50.526 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:54:50.526 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:54:50.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:50.581 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:hints = 1 2024-03-26T03:54:50.582 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:54:50.582 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object prefix: benchmark_data_smithi006_103169 2024-03-26T03:54:50.582 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:54:50.582 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:54:50.605 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:50.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:50.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:50.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:50.755 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:50 smithi006 ceph-mon[37501]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:54:50.755 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:50 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:54:50.755 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:50 smithi006 ceph-mon[29446]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:54:50.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:50 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:54:50.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:54:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:50 smithi086 ceph-mon[32965]: pgmap v246: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:54:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:50 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:54:50.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:50.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:50.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:50.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:50.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.014 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:51.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:51.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:51.153 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:51.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:51.280 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:51.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:51.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:51.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:51.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:54:51.546 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:54:51.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:51.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:51 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:54:51.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:51 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:54:51.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:51 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:54:51.578 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:51 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:54:51.582 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 1 16 3084 3068 191.643 191.75 0.00556863 0.00519732 2024-03-26T03:54:51.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:51.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:51.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:51.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:51.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:51.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:51 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:54:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:51 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:54:51.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:51.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:51.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:51.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:52.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:52.099 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:52.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:52.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:52.242 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:52.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:52.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:52.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:52.535 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:54:52.583 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 2 16 6492 6476 202.309 213 0.00164788 0.00491753 2024-03-26T03:54:52.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.644 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:52.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:52.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:52.784 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:54:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:54:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:54:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: osdmap e159: 8 total, 8 up, 6 in 2024-03-26T03:54:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/330665727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:52 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: osdmap e159: 8 total, 8 up, 6 in 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/330665727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: osdmap e159: 8 total, 8 up, 6 in 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/330665727' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:52 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:54:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:52.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:52.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:53.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:53.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:53.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:53.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:53.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:53.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:54:53.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:53.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:53.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:53.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:53.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:53.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:53.583 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 3 16 9582 9566 199.241 193.125 0.00493771 0.00501167 2024-03-26T03:54:53.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:53.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:53.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:53.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:53.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:53.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:53.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:54:53.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:54:53.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[29446]: osdmap e160: 8 total, 8 up, 6 in 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:53.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:53 smithi006 ceph-mon[37501]: osdmap e160: 8 total, 8 up, 6 in 2024-03-26T03:54:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:53 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:54:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:53 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:54:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:53 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:54:53.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:53 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:54:53.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:53 smithi086 ceph-mon[32965]: osdmap e160: 8 total, 8 up, 6 in 2024-03-26T03:54:53.843 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:53.857 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:53.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:53.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:53.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:53.932 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:54:54.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:54.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:54.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:54.204 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:54.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:54.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:54.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:54.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:54.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:54.583 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 4 16 12264 12248 191.335 167.625 0.00754862 0.00521737 2024-03-26T03:54:54.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:54 smithi006 ceph-mon[29446]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:54.609 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:54.610 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:54:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:54:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:54:54.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:54 smithi006 ceph-mon[37501]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:54.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:54.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:54.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:54.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:54.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:54.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:54 smithi086 ceph-mon[32965]: pgmap v250: 148 pgs: 148 active+clean; 582 KiB data, 725 MiB used, 536 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:54:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:54:54.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:54:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:54:54.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:54.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:54.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:54.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:55.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:55.042 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:55.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:55.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:55.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:55.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:55.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:55.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:55.454 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:55.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:55.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:55.583 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 5 16 15351 15335 191.652 192.938 0.004588 0.00521082 2024-03-26T03:54:55.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:55.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:55.621 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:55.621 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:55.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:55.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:55.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:55.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:54:55.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:54:55.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:55.887 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:55.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:55.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:56.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:56.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:56.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:56.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:56.320 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:56.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:56.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:56.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:56.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:56.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:56.583 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 6 16 18086 18070 188.197 170.938 0.00525386 0.00530772 2024-03-26T03:54:56.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.597 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:56.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:56.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:56.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:56.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:56.826 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:56 smithi006 ceph-mon[37501]: pgmap v251: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:54:56.826 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:56 smithi006 ceph-mon[29446]: pgmap v251: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:54:56.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:56 smithi086 ceph-mon[32965]: pgmap v251: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.5 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-26T03:54:56.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:56.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:56.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:57.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:57.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:57.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:57.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:57.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:57.292 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:57.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:57.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:57.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:57.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:57.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 7 16 20632 20616 184.026 159.125 0.00958844 0.00542827 2024-03-26T03:54:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:57.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:57.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:57.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:57.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:57.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:57.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:57.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:54:57.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:57.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:54:58.039 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:54:58.086 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:54:58.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:58.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:58.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:58.133 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:54:58.181 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:54:58.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:58.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:54:58.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:58.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:58.261 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:54:58.307 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:54:58.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:54:58.354 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:54:58.364 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:54:58.365 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:54:58.397 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:54:58.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:58.540 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:58.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 8 16 22535 22519 175.89 118.938 0.00487928 0.00568001 2024-03-26T03:54:58.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:58 smithi006 ceph-mon[37501]: pgmap v252: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:54:58.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1210765464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:58.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:58 smithi006 ceph-mon[29446]: pgmap v252: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:54:58.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1210765464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:58.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:58.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:58.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:54:58.806 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:58.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:54:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:58 smithi086 ceph-mon[32965]: pgmap v252: 148 pgs: 148 active+clean; 732 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-26T03:54:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1210765464' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:54:58.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:58.948 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:58.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:58.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:59.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:54:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:59.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:54:59.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:54:59.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:59.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:54:59.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:59.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:54:59.451 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:54:59.459 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:54:59.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:59.499 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:59.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 9 16 24636 24620 170.937 131.312 0.00901475 0.00584273 2024-03-26T03:54:59.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:54:59.639 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:59.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:54:59.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:54:59.780 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:54:59.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:54:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:54:59 smithi086 ceph-mon[32965]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:59.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:54:59.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:54:59 smithi006 ceph-mon[37501]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:54:59 smithi006 ceph-mon[29446]: from='client.15048 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:54:59.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:54:59.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:54:59.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:00.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.058 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:00.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:00.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:00.193 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:00.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:00.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:00.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:00.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:00.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:00.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:00.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:00.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 10 16 26722 26706 166.879 130.375 0.00545678 0.00598563 2024-03-26T03:55:00.604 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:00.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:00.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:00.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:00 smithi086 ceph-mon[32965]: pgmap v253: 148 pgs: 148 active+clean; 1.1 GiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-26T03:55:00.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:00.880 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:00.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:00 smithi006 ceph-mon[37501]: pgmap v253: 148 pgs: 148 active+clean; 1.1 GiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-26T03:55:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:00 smithi006 ceph-mon[29446]: pgmap v253: 148 pgs: 148 active+clean; 1.1 GiB data, 3.7 GiB used, 533 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.25k op/s 2024-03-26T03:55:00.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:01.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:01.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:01.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:01.147 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:01.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:01.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:01.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:01.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:01.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:01.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:01.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:01.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:01.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 11 16 28860 28844 163.856 133.625 0.00377888 0.00609769 2024-03-26T03:55:01.651 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2024-03-26T03:54:57.289228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":768022952,"num_objects":11940,"num_object_clones":0,"num_object_copies":35820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11940,"num_whiteouts":0,"num_read":1577,"num_read_kb":1391,"num_write":12517,"num_write_kb":751901,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":211,"num_bytes_recovered":2366036,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12549,"ondisk_log_size":12549,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":2635560,"kb_used_data":1920932,"kb_used_omap":9,"kb_used_meta":714550,"kb_avail":559835352,"statfs":{"total":575970213888,"available":573271400448,"internally_reserved":0,"allocated":1967034368,"data_stored":1964373974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9565,"internal_metadata":731699875},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":767426560,"num_objects":11710,"num_object_clones":0,"num_object_copies":35130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11710,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11710,"num_write_kb":749440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002350"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529130+0000","last_change":"2024-03-26T03:53:29.435229+0000","last_active":"2024-03-26T03:54:52.529130+0000","last_peered":"2024-03-26T03:54:52.529130+0000","last_clean":"2024-03-26T03:54:52.529130+0000","last_became_active":"2024-03-26T03:48:50.034930+0000","last_became_peered":"2024-03-26T03:48:50.034930+0000","last_unstale":"2024-03-26T03:54:52.529130+0000","last_undegraded":"2024-03-26T03:54:52.529130+0000","last_fullsized":"2024-03-26T03:54:52.529130+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531779+0000","last_change":"2024-03-26T03:52:43.862548+0000","last_active":"2024-03-26T03:54:52.531779+0000","last_peered":"2024-03-26T03:54:52.531779+0000","last_clean":"2024-03-26T03:54:52.531779+0000","last_became_active":"2024-03-26T03:48:48.025006+0000","last_became_peered":"2024-03-26T03:48:48.025006+0000","last_unstale":"2024-03-26T03:54:52.531779+0000","last_undegraded":"2024-03-26T03:54:52.531779+0000","last_fullsized":"2024-03-26T03:54:52.531779+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531301+0000","last_change":"2024-03-26T03:54:15.751346+0000","last_active":"2024-03-26T03:54:52.531301+0000","last_peered":"2024-03-26T03:54:52.531301+0000","last_clean":"2024-03-26T03:54:52.531301+0000","last_became_active":"2024-03-26T03:48:44.009357+0000","last_became_peered":"2024-03-26T03:48:44.009357+0000","last_unstale":"2024-03-26T03:54:52.531301+0000","last_undegraded":"2024-03-26T03:54:52.531301+0000","last_fullsized":"2024-03-26T03:54:52.531301+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.598037+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":277,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527926+0000","last_change":"2024-03-26T03:54:29.547822+0000","last_active":"2024-03-26T03:54:52.527926+0000","last_peered":"2024-03-26T03:54:52.527926+0000","last_clean":"2024-03-26T03:54:52.527926+0000","last_became_active":"2024-03-26T03:54:29.438463+0000","last_became_peered":"2024-03-26T03:54:29.438463+0000","last_unstale":"2024-03-26T03:54:52.527926+0000","last_undegraded":"2024-03-26T03:54:52.527926+0000","last_fullsized":"2024-03-26T03:54:52.527926+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:52:33.804176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.804176+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":90,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532608+0000","last_change":"2024-03-26T03:54:29.358028+0000","last_active":"2024-03-26T03:54:52.532608+0000","last_peered":"2024-03-26T03:54:52.532608+0000","last_clean":"2024-03-26T03:54:52.532608+0000","last_became_active":"2024-03-26T03:54:29.357724+0000","last_became_peered":"2024-03-26T03:54:29.357724+0000","last_unstale":"2024-03-26T03:54:52.532608+0000","last_undegraded":"2024-03-26T03:54:52.532608+0000","last_fullsized":"2024-03-26T03:54:52.532608+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.442635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.442635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654374+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527687+0000","last_change":"2024-03-26T03:53:13.396637+0000","last_active":"2024-03-26T03:54:52.527687+0000","last_peered":"2024-03-26T03:54:52.527687+0000","last_clean":"2024-03-26T03:54:52.527687+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:54:52.527687+0000","last_undegraded":"2024-03-26T03:54:52.527687+0000","last_fullsized":"2024-03-26T03:54:52.527687+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.396551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.396551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":102,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.537890+0000","last_change":"2024-03-26T03:54:29.982467+0000","last_active":"2024-03-26T03:54:52.537890+0000","last_peered":"2024-03-26T03:54:52.537890+0000","last_clean":"2024-03-26T03:54:52.537890+0000","last_became_active":"2024-03-26T03:54:29.377365+0000","last_became_peered":"2024-03-26T03:54:29.377365+0000","last_unstale":"2024-03-26T03:54:52.537890+0000","last_undegraded":"2024-03-26T03:54:52.537890+0000","last_fullsized":"2024-03-26T03:54:52.537890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.982392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.982392+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.166800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":140,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.539139+0000","last_change":"2024-03-26T03:54:29.395843+0000","last_active":"2024-03-26T03:54:52.539139+0000","last_peered":"2024-03-26T03:54:52.539139+0000","last_clean":"2024-03-26T03:54:52.539139+0000","last_became_active":"2024-03-26T03:54:29.330558+0000","last_became_peered":"2024-03-26T03:54:29.330558+0000","last_unstale":"2024-03-26T03:54:52.539139+0000","last_undegraded":"2024-03-26T03:54:52.539139+0000","last_fullsized":"2024-03-26T03:54:52.539139+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.826868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530321+0000","last_change":"2024-03-26T03:53:22.532263+0000","last_active":"2024-03-26T03:54:52.530321+0000","last_peered":"2024-03-26T03:54:52.530321+0000","last_clean":"2024-03-26T03:54:52.530321+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:54:52.530321+0000","last_undegraded":"2024-03-26T03:54:52.530321+0000","last_fullsized":"2024-03-26T03:54:52.530321+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:22.532151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:22.532151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822058+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531009+0000","last_change":"2024-03-26T03:53:24.585622+0000","last_active":"2024-03-26T03:54:52.531009+0000","last_peered":"2024-03-26T03:54:52.531009+0000","last_clean":"2024-03-26T03:54:52.531009+0000","last_became_active":"2024-03-26T03:48:48.020123+0000","last_became_peered":"2024-03-26T03:48:48.020123+0000","last_unstale":"2024-03-26T03:54:52.531009+0000","last_undegraded":"2024-03-26T03:54:52.531009+0000","last_fullsized":"2024-03-26T03:54:52.531009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.585508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.585508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531599+0000","last_change":"2024-03-26T03:53:18.575050+0000","last_active":"2024-03-26T03:54:52.531599+0000","last_peered":"2024-03-26T03:54:52.531599+0000","last_clean":"2024-03-26T03:54:52.531599+0000","last_became_active":"2024-03-26T03:50:45.593360+0000","last_became_peered":"2024-03-26T03:50:45.593360+0000","last_unstale":"2024-03-26T03:54:52.531599+0000","last_undegraded":"2024-03-26T03:54:52.531599+0000","last_fullsized":"2024-03-26T03:54:52.531599+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":282,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529403+0000","last_change":"2024-03-26T03:54:29.551157+0000","last_active":"2024-03-26T03:54:52.529403+0000","last_peered":"2024-03-26T03:54:52.529403+0000","last_clean":"2024-03-26T03:54:52.529403+0000","last_became_active":"2024-03-26T03:54:29.444189+0000","last_became_peered":"2024-03-26T03:54:29.444189+0000","last_unstale":"2024-03-26T03:54:52.529403+0000","last_undegraded":"2024-03-26T03:54:52.529403+0000","last_fullsized":"2024-03-26T03:54:52.529403+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530586+0000","last_change":"2024-03-26T03:54:29.289257+0000","last_active":"2024-03-26T03:54:52.530586+0000","last_peered":"2024-03-26T03:54:52.530586+0000","last_clean":"2024-03-26T03:54:52.530586+0000","last_became_active":"2024-03-26T03:54:29.288929+0000","last_became_peered":"2024-03-26T03:54:29.288929+0000","last_unstale":"2024-03-26T03:54:52.530586+0000","last_undegraded":"2024-03-26T03:54:52.530586+0000","last_fullsized":"2024-03-26T03:54:52.530586+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.259326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532570+0000","last_change":"2024-03-26T03:54:11.544712+0000","last_active":"2024-03-26T03:54:52.532570+0000","last_peered":"2024-03-26T03:54:52.532570+0000","last_clean":"2024-03-26T03:54:52.532570+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:54:52.532570+0000","last_undegraded":"2024-03-26T03:54:52.532570+0000","last_fullsized":"2024-03-26T03:54:52.532570+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.544637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.544637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:30.660465+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530804+0000","last_change":"2024-03-26T03:54:32.784964+0000","last_active":"2024-03-26T03:54:52.530804+0000","last_peered":"2024-03-26T03:54:52.530804+0000","last_clean":"2024-03-26T03:54:52.530804+0000","last_became_active":"2024-03-26T03:48:44.013488+0000","last_became_peered":"2024-03-26T03:48:44.013488+0000","last_unstale":"2024-03-26T03:54:52.530804+0000","last_undegraded":"2024-03-26T03:54:52.530804+0000","last_fullsized":"2024-03-26T03:54:52.530804+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.924921+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":252,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528815+0000","last_change":"2024-03-26T03:54:29.472775+0000","last_active":"2024-03-26T03:54:52.528815+0000","last_peered":"2024-03-26T03:54:52.528815+0000","last_clean":"2024-03-26T03:54:52.528815+0000","last_became_active":"2024-03-26T03:54:29.360309+0000","last_became_peered":"2024-03-26T03:54:29.360309+0000","last_unstale":"2024-03-26T03:54:52.528815+0000","last_undegraded":"2024-03-26T03:54:52.528815+0000","last_fullsized":"2024-03-26T03:54:52.528815+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.915559+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":218,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528318+0000","last_change":"2024-03-26T03:54:29.346617+0000","last_active":"2024-03-26T03:54:52.528318+0000","last_peered":"2024-03-26T03:54:52.528318+0000","last_clean":"2024-03-26T03:54:52.528318+0000","last_became_active":"2024-03-26T03:54:29.346320+0000","last_became_peered":"2024-03-26T03:54:29.346320+0000","last_unstale":"2024-03-26T03:54:52.528318+0000","last_undegraded":"2024-03-26T03:54:52.528318+0000","last_fullsized":"2024-03-26T03:54:52.528318+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":208,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.535317+0000","last_change":"2024-03-26T03:53:30.866289+0000","last_active":"2024-03-26T03:54:52.535317+0000","last_peered":"2024-03-26T03:54:52.535317+0000","last_clean":"2024-03-26T03:54:52.535317+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:54:52.535317+0000","last_undegraded":"2024-03-26T03:54:52.535317+0000","last_fullsized":"2024-03-26T03:54:52.535317+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.866047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.866047+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":51,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531862+0000","last_change":"2024-03-26T03:53:13.631462+0000","last_active":"2024-03-26T03:54:52.531862+0000","last_peered":"2024-03-26T03:54:52.531862+0000","last_clean":"2024-03-26T03:54:52.531862+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:54:52.531862+0000","last_undegraded":"2024-03-26T03:54:52.531862+0000","last_fullsized":"2024-03-26T03:54:52.531862+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:13.631351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.631351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822171+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":276,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.538201+0000","last_change":"2024-03-26T03:54:29.543230+0000","last_active":"2024-03-26T03:54:52.538201+0000","last_peered":"2024-03-26T03:54:52.538201+0000","last_clean":"2024-03-26T03:54:52.538201+0000","last_became_active":"2024-03-26T03:54:29.346374+0000","last_became_peered":"2024-03-26T03:54:29.346374+0000","last_unstale":"2024-03-26T03:54:52.538201+0000","last_undegraded":"2024-03-26T03:54:52.538201+0000","last_fullsized":"2024-03-26T03:54:52.538201+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.041388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.041388+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:30.279550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":90,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532050+0000","last_change":"2024-03-26T03:54:29.809911+0000","last_active":"2024-03-26T03:54:52.532050+0000","last_peered":"2024-03-26T03:54:52.532050+0000","last_clean":"2024-03-26T03:54:52.532050+0000","last_became_active":"2024-03-26T03:54:29.289421+0000","last_became_peered":"2024-03-26T03:54:29.289421+0000","last_unstale":"2024-03-26T03:54:52.532050+0000","last_undegraded":"2024-03-26T03:54:52.532050+0000","last_fullsized":"2024-03-26T03:54:52.532050+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:48.048135+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528357+0000","last_change":"2024-03-26T03:54:29.360637+0000","last_active":"2024-03-26T03:54:52.528357+0000","last_peered":"2024-03-26T03:54:52.528357+0000","last_clean":"2024-03-26T03:54:52.528357+0000","last_became_active":"2024-03-26T03:54:29.360289+0000","last_became_peered":"2024-03-26T03:54:29.360289+0000","last_unstale":"2024-03-26T03:54:52.528357+0000","last_undegraded":"2024-03-26T03:54:52.528357+0000","last_fullsized":"2024-03-26T03:54:52.528357+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:16.928630+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529507+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:54:52.529507+0000","last_peered":"2024-03-26T03:54:52.529507+0000","last_clean":"2024-03-26T03:54:52.529507+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:54:52.529507+0000","last_undegraded":"2024-03-26T03:54:52.529507+0000","last_fullsized":"2024-03-26T03:54:52.529507+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.802874+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":92,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.535027+0000","last_change":"2024-03-26T03:54:29.404853+0000","last_active":"2024-03-26T03:54:52.535027+0000","last_peered":"2024-03-26T03:54:52.535027+0000","last_clean":"2024-03-26T03:54:52.535027+0000","last_became_active":"2024-03-26T03:54:29.322224+0000","last_became_peered":"2024-03-26T03:54:29.322224+0000","last_unstale":"2024-03-26T03:54:52.535027+0000","last_undegraded":"2024-03-26T03:54:52.535027+0000","last_fullsized":"2024-03-26T03:54:52.535027+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533075+0000","last_change":"2024-03-26T03:54:15.688631+0000","last_active":"2024-03-26T03:54:52.533075+0000","last_peered":"2024-03-26T03:54:52.533075+0000","last_clean":"2024-03-26T03:54:52.533075+0000","last_became_active":"2024-03-26T03:48:50.038500+0000","last_became_peered":"2024-03-26T03:48:50.038500+0000","last_unstale":"2024-03-26T03:54:52.533075+0000","last_undegraded":"2024-03-26T03:54:52.533075+0000","last_fullsized":"2024-03-26T03:54:52.533075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:54:15.688518+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.688518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:42.161568+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":232,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529158+0000","last_change":"2024-03-26T03:52:39.855016+0000","last_active":"2024-03-26T03:54:52.529158+0000","last_peered":"2024-03-26T03:54:52.529158+0000","last_clean":"2024-03-26T03:54:52.529158+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:54:52.529158+0000","last_undegraded":"2024-03-26T03:54:52.529158+0000","last_fullsized":"2024-03-26T03:54:52.529158+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:52:39.854865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:39.854865+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531686+0000","last_change":"2024-03-26T03:54:29.376610+0000","last_active":"2024-03-26T03:54:52.531686+0000","last_peered":"2024-03-26T03:54:52.531686+0000","last_clean":"2024-03-26T03:54:52.531686+0000","last_became_active":"2024-03-26T03:54:29.324000+0000","last_became_peered":"2024-03-26T03:54:29.324000+0000","last_unstale":"2024-03-26T03:54:52.531686+0000","last_undegraded":"2024-03-26T03:54:52.531686+0000","last_fullsized":"2024-03-26T03:54:52.531686+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:13.785265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:13.785265+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":269,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532877+0000","last_change":"2024-03-26T03:54:36.814787+0000","last_active":"2024-03-26T03:54:52.532877+0000","last_peered":"2024-03-26T03:54:52.532877+0000","last_clean":"2024-03-26T03:54:52.532877+0000","last_became_active":"2024-03-26T03:54:29.521665+0000","last_became_peered":"2024-03-26T03:54:29.521665+0000","last_unstale":"2024-03-26T03:54:52.532877+0000","last_undegraded":"2024-03-26T03:54:52.532877+0000","last_fullsized":"2024-03-26T03:54:52.532877+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:58.752253+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":204,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.534916+0000","last_change":"2024-03-26T03:52:42.867889+0000","last_active":"2024-03-26T03:54:52.534916+0000","last_peered":"2024-03-26T03:54:52.534916+0000","last_clean":"2024-03-26T03:54:52.534916+0000","last_became_active":"2024-03-26T03:48:50.034815+0000","last_became_peered":"2024-03-26T03:48:50.034815+0000","last_unstale":"2024-03-26T03:54:52.534916+0000","last_undegraded":"2024-03-26T03:54:52.534916+0000","last_fullsized":"2024-03-26T03:54:52.534916+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":218,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533118+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:54:52.533118+0000","last_peered":"2024-03-26T03:54:52.533118+0000","last_clean":"2024-03-26T03:54:52.533118+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:54:52.533118+0000","last_undegraded":"2024-03-26T03:54:52.533118+0000","last_fullsized":"2024-03-26T03:54:52.533118+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.699157+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":32,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530235+0000","last_change":"2024-03-26T03:54:31.825491+0000","last_active":"2024-03-26T03:54:52.530235+0000","last_peered":"2024-03-26T03:54:52.530235+0000","last_clean":"2024-03-26T03:54:52.530235+0000","last_became_active":"2024-03-26T03:54:29.453692+0000","last_became_peered":"2024-03-26T03:54:29.453692+0000","last_unstale":"2024-03-26T03:54:52.530235+0000","last_undegraded":"2024-03-26T03:54:52.530235+0000","last_fullsized":"2024-03-26T03:54:52.530235+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.059388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":221,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532401+0000","last_change":"2024-03-26T03:52:33.269150+0000","last_active":"2024-03-26T03:54:52.532401+0000","last_peered":"2024-03-26T03:54:52.532401+0000","last_clean":"2024-03-26T03:54:52.532401+0000","last_became_active":"2024-03-26T03:50:45.603208+0000","last_became_peered":"2024-03-26T03:50:45.603208+0000","last_unstale":"2024-03-26T03:54:52.532401+0000","last_undegraded":"2024-03-26T03:54:52.532401+0000","last_fullsized":"2024-03-26T03:54:52.532401+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:33.269037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.269037+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532623+0000","last_change":"2024-03-26T03:54:19.645849+0000","last_active":"2024-03-26T03:54:52.532623+0000","last_peered":"2024-03-26T03:54:52.532623+0000","last_clean":"2024-03-26T03:54:52.532623+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:54:52.532623+0000","last_undegraded":"2024-03-26T03:54:52.532623+0000","last_fullsized":"2024-03-26T03:54:52.532623+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:19.645746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.645746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:45.723244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":155,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527699+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:54:52.527699+0000","last_peered":"2024-03-26T03:54:52.527699+0000","last_clean":"2024-03-26T03:54:52.527699+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:54:52.527699+0000","last_undegraded":"2024-03-26T03:54:52.527699+0000","last_fullsized":"2024-03-26T03:54:52.527699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.501238+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531519+0000","last_change":"2024-03-26T03:54:29.518528+0000","last_active":"2024-03-26T03:54:52.531519+0000","last_peered":"2024-03-26T03:54:52.531519+0000","last_clean":"2024-03-26T03:54:52.531519+0000","last_became_active":"2024-03-26T03:54:29.518254+0000","last_became_peered":"2024-03-26T03:54:29.518254+0000","last_unstale":"2024-03-26T03:54:52.531519+0000","last_undegraded":"2024-03-26T03:54:52.531519+0000","last_fullsized":"2024-03-26T03:54:52.531519+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:23.578640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.578640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.821869+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532153+0000","last_change":"2024-03-26T03:54:29.394685+0000","last_active":"2024-03-26T03:54:52.532153+0000","last_peered":"2024-03-26T03:54:52.532153+0000","last_clean":"2024-03-26T03:54:52.532153+0000","last_became_active":"2024-03-26T03:54:29.345582+0000","last_became_peered":"2024-03-26T03:54:29.345582+0000","last_unstale":"2024-03-26T03:54:52.532153+0000","last_undegraded":"2024-03-26T03:54:52.532153+0000","last_fullsized":"2024-03-26T03:54:52.532153+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":90,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530262+0000","last_change":"2024-03-26T03:54:29.400638+0000","last_active":"2024-03-26T03:54:52.530262+0000","last_peered":"2024-03-26T03:54:52.530262+0000","last_clean":"2024-03-26T03:54:52.530262+0000","last_became_active":"2024-03-26T03:54:29.361440+0000","last_became_peered":"2024-03-26T03:54:29.361440+0000","last_unstale":"2024-03-26T03:54:52.530262+0000","last_undegraded":"2024-03-26T03:54:52.530262+0000","last_fullsized":"2024-03-26T03:54:52.530262+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.352186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:21.094208+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.352186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532163+0000","last_change":"2024-03-26T03:52:48.648107+0000","last_active":"2024-03-26T03:54:52.532163+0000","last_peered":"2024-03-26T03:54:52.532163+0000","last_clean":"2024-03-26T03:54:52.532163+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:54:52.532163+0000","last_undegraded":"2024-03-26T03:54:52.532163+0000","last_fullsized":"2024-03-26T03:54:52.532163+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:48.647994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:48.647994+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":233,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531983+0000","last_change":"2024-03-26T03:54:35.798476+0000","last_active":"2024-03-26T03:54:52.531983+0000","last_peered":"2024-03-26T03:54:52.531983+0000","last_clean":"2024-03-26T03:54:52.531983+0000","last_became_active":"2024-03-26T03:48:44.018826+0000","last_became_peered":"2024-03-26T03:48:44.018826+0000","last_unstale":"2024-03-26T03:54:52.531983+0000","last_undegraded":"2024-03-26T03:54:52.531983+0000","last_fullsized":"2024-03-26T03:54:52.531983+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:57.886093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":125,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532632+0000","last_change":"2024-03-26T03:54:29.548373+0000","last_active":"2024-03-26T03:54:52.532632+0000","last_peered":"2024-03-26T03:54:52.532632+0000","last_clean":"2024-03-26T03:54:52.532632+0000","last_became_active":"2024-03-26T03:54:29.357737+0000","last_became_peered":"2024-03-26T03:54:29.357737+0000","last_unstale":"2024-03-26T03:54:52.532632+0000","last_undegraded":"2024-03-26T03:54:52.532632+0000","last_fullsized":"2024-03-26T03:54:52.532632+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529734+0000","last_change":"2024-03-26T03:54:28.830813+0000","last_active":"2024-03-26T03:54:52.529734+0000","last_peered":"2024-03-26T03:54:52.529734+0000","last_clean":"2024-03-26T03:54:52.529734+0000","last_became_active":"2024-03-26T03:48:50.036736+0000","last_became_peered":"2024-03-26T03:48:50.036736+0000","last_unstale":"2024-03-26T03:54:52.529734+0000","last_undegraded":"2024-03-26T03:54:52.529734+0000","last_fullsized":"2024-03-26T03:54:52.529734+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.572394+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":24,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530365+0000","last_change":"2024-03-26T03:54:29.348291+0000","last_active":"2024-03-26T03:54:52.530365+0000","last_peered":"2024-03-26T03:54:52.530365+0000","last_clean":"2024-03-26T03:54:52.530365+0000","last_became_active":"2024-03-26T03:54:29.344062+0000","last_became_peered":"2024-03-26T03:54:29.344062+0000","last_unstale":"2024-03-26T03:54:52.530365+0000","last_undegraded":"2024-03-26T03:54:52.530365+0000","last_fullsized":"2024-03-26T03:54:52.530365+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.410272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.410272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533174+0000","last_change":"2024-03-26T03:54:29.530213+0000","last_active":"2024-03-26T03:54:52.533174+0000","last_peered":"2024-03-26T03:54:52.533174+0000","last_clean":"2024-03-26T03:54:52.533174+0000","last_became_active":"2024-03-26T03:54:29.529851+0000","last_became_peered":"2024-03-26T03:54:29.529851+0000","last_unstale":"2024-03-26T03:54:52.533174+0000","last_undegraded":"2024-03-26T03:54:52.533174+0000","last_fullsized":"2024-03-26T03:54:52.533174+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654599+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"152'13","reported_seq":256,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533135+0000","last_change":"2024-03-26T03:54:26.861839+0000","last_active":"2024-03-26T03:54:52.533135+0000","last_peered":"2024-03-26T03:54:52.533135+0000","last_clean":"2024-03-26T03:54:52.533135+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:54:52.533135+0000","last_undegraded":"2024-03-26T03:54:52.533135+0000","last_fullsized":"2024-03-26T03:54:52.533135+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:54:26.861674+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:28.880700+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530233+0000","last_change":"2024-03-26T03:54:34.804745+0000","last_active":"2024-03-26T03:54:52.530233+0000","last_peered":"2024-03-26T03:54:52.530233+0000","last_clean":"2024-03-26T03:54:52.530233+0000","last_became_active":"2024-03-26T03:54:29.303365+0000","last_became_peered":"2024-03-26T03:54:29.303365+0000","last_unstale":"2024-03-26T03:54:52.530233+0000","last_undegraded":"2024-03-26T03:54:52.530233+0000","last_fullsized":"2024-03-26T03:54:52.530233+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.804652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:49.787998+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":214,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532910+0000","last_change":"2024-03-26T03:54:29.323517+0000","last_active":"2024-03-26T03:54:52.532910+0000","last_peered":"2024-03-26T03:54:52.532910+0000","last_clean":"2024-03-26T03:54:52.532910+0000","last_became_active":"2024-03-26T03:54:29.319822+0000","last_became_peered":"2024-03-26T03:54:29.319822+0000","last_unstale":"2024-03-26T03:54:52.532910+0000","last_undegraded":"2024-03-26T03:54:52.532910+0000","last_fullsized":"2024-03-26T03:54:52.532910+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528136+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:54:52.528136+0000","last_peered":"2024-03-26T03:54:52.528136+0000","last_clean":"2024-03-26T03:54:52.528136+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:54:52.528136+0000","last_undegraded":"2024-03-26T03:54:52.528136+0000","last_fullsized":"2024-03-26T03:54:52.528136+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.223788+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":250,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528940+0000","last_change":"2024-03-26T03:53:33.392149+0000","last_active":"2024-03-26T03:54:52.528940+0000","last_peered":"2024-03-26T03:54:52.528940+0000","last_clean":"2024-03-26T03:54:52.528940+0000","last_became_active":"2024-03-26T03:50:45.595619+0000","last_became_peered":"2024-03-26T03:50:45.595619+0000","last_unstale":"2024-03-26T03:54:52.528940+0000","last_undegraded":"2024-03-26T03:54:52.528940+0000","last_fullsized":"2024-03-26T03:54:52.528940+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":211,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531309+0000","last_change":"2024-03-26T03:54:29.521980+0000","last_active":"2024-03-26T03:54:52.531309+0000","last_peered":"2024-03-26T03:54:52.531309+0000","last_clean":"2024-03-26T03:54:52.531309+0000","last_became_active":"2024-03-26T03:54:29.520018+0000","last_became_peered":"2024-03-26T03:54:29.520018+0000","last_unstale":"2024-03-26T03:54:52.531309+0000","last_undegraded":"2024-03-26T03:54:52.531309+0000","last_fullsized":"2024-03-26T03:54:52.531309+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:53.774450+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":300,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531737+0000","last_change":"2024-03-26T03:52:38.249583+0000","last_active":"2024-03-26T03:54:52.531737+0000","last_peered":"2024-03-26T03:54:52.531737+0000","last_clean":"2024-03-26T03:54:52.531737+0000","last_became_active":"2024-03-26T03:48:48.022327+0000","last_became_peered":"2024-03-26T03:48:48.022327+0000","last_unstale":"2024-03-26T03:54:52.531737+0000","last_undegraded":"2024-03-26T03:54:52.531737+0000","last_fullsized":"2024-03-26T03:54:52.531737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:52:38.249472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:38.249472+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532705+0000","last_change":"2024-03-26T03:53:20.739662+0000","last_active":"2024-03-26T03:54:52.532705+0000","last_peered":"2024-03-26T03:54:52.532705+0000","last_clean":"2024-03-26T03:54:52.532705+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:54:52.532705+0000","last_undegraded":"2024-03-26T03:54:52.532705+0000","last_fullsized":"2024-03-26T03:54:52.532705+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.739547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.739547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":106,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532235+0000","last_change":"2024-03-26T03:54:32.911388+0000","last_active":"2024-03-26T03:54:52.532235+0000","last_peered":"2024-03-26T03:54:52.532235+0000","last_clean":"2024-03-26T03:54:52.532235+0000","last_became_active":"2024-03-26T03:54:29.390009+0000","last_became_peered":"2024-03-26T03:54:29.390009+0000","last_unstale":"2024-03-26T03:54:52.532235+0000","last_undegraded":"2024-03-26T03:54:52.532235+0000","last_fullsized":"2024-03-26T03:54:52.532235+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:59.591186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529179+0000","last_change":"2024-03-26T03:54:15.344511+0000","last_active":"2024-03-26T03:54:52.529179+0000","last_peered":"2024-03-26T03:54:52.529179+0000","last_clean":"2024-03-26T03:54:52.529179+0000","last_became_active":"2024-03-26T03:48:50.038335+0000","last_became_peered":"2024-03-26T03:48:50.038335+0000","last_unstale":"2024-03-26T03:54:52.529179+0000","last_undegraded":"2024-03-26T03:54:52.529179+0000","last_fullsized":"2024-03-26T03:54:52.529179+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:20.100911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":59,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533065+0000","last_change":"2024-03-26T03:54:29.542214+0000","last_active":"2024-03-26T03:54:52.533065+0000","last_peered":"2024-03-26T03:54:52.533065+0000","last_clean":"2024-03-26T03:54:52.533065+0000","last_became_active":"2024-03-26T03:54:29.389636+0000","last_became_peered":"2024-03-26T03:54:29.389636+0000","last_unstale":"2024-03-26T03:54:52.533065+0000","last_undegraded":"2024-03-26T03:54:52.533065+0000","last_fullsized":"2024-03-26T03:54:52.533065+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:51:34.258218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:51:34.258218+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":243,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.535073+0000","last_change":"2024-03-26T03:54:29.455442+0000","last_active":"2024-03-26T03:54:52.535073+0000","last_peered":"2024-03-26T03:54:52.535073+0000","last_clean":"2024-03-26T03:54:52.535073+0000","last_became_active":"2024-03-26T03:54:29.345136+0000","last_became_peered":"2024-03-26T03:54:29.345136+0000","last_unstale":"2024-03-26T03:54:52.535073+0000","last_undegraded":"2024-03-26T03:54:52.535073+0000","last_fullsized":"2024-03-26T03:54:52.535073+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:53:12.600988+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:12.600988+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822113+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":272,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531938+0000","last_change":"2024-03-26T03:54:29.550761+0000","last_active":"2024-03-26T03:54:52.531938+0000","last_peered":"2024-03-26T03:54:52.531938+0000","last_clean":"2024-03-26T03:54:52.531938+0000","last_became_active":"2024-03-26T03:54:29.522025+0000","last_became_peered":"2024-03-26T03:54:29.522025+0000","last_unstale":"2024-03-26T03:54:52.531938+0000","last_undegraded":"2024-03-26T03:54:52.531938+0000","last_fullsized":"2024-03-26T03:54:52.531938+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.538843+0000","last_change":"2024-03-26T03:54:36.963991+0000","last_active":"2024-03-26T03:54:52.538843+0000","last_peered":"2024-03-26T03:54:52.538843+0000","last_clean":"2024-03-26T03:54:52.538843+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:54:52.538843+0000","last_undegraded":"2024-03-26T03:54:52.538843+0000","last_fullsized":"2024-03-26T03:54:52.538843+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:54:36.963833+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.963833+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:52.907379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":24,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532612+0000","last_change":"2024-03-26T03:54:29.532516+0000","last_active":"2024-03-26T03:54:52.532612+0000","last_peered":"2024-03-26T03:54:52.532612+0000","last_clean":"2024-03-26T03:54:52.532612+0000","last_became_active":"2024-03-26T03:54:29.532138+0000","last_became_peered":"2024-03-26T03:54:29.532138+0000","last_unstale":"2024-03-26T03:54:52.532612+0000","last_undegraded":"2024-03-26T03:54:52.532612+0000","last_fullsized":"2024-03-26T03:54:52.532612+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:23.351964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:23.351964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532476+0000","last_change":"2024-03-26T03:53:29.687145+0000","last_active":"2024-03-26T03:54:52.532476+0000","last_peered":"2024-03-26T03:54:52.532476+0000","last_clean":"2024-03-26T03:54:52.532476+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:54:52.532476+0000","last_undegraded":"2024-03-26T03:54:52.532476+0000","last_fullsized":"2024-03-26T03:54:52.532476+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.687067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.687067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654535+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":273,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532935+0000","last_change":"2024-03-26T03:54:29.542309+0000","last_active":"2024-03-26T03:54:52.532935+0000","last_peered":"2024-03-26T03:54:52.532935+0000","last_clean":"2024-03-26T03:54:52.532935+0000","last_became_active":"2024-03-26T03:54:29.521088+0000","last_became_peered":"2024-03-26T03:54:29.521088+0000","last_unstale":"2024-03-26T03:54:52.532935+0000","last_undegraded":"2024-03-26T03:54:52.532935+0000","last_fullsized":"2024-03-26T03:54:52.532935+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:52:37.649630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.649630+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.821968+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":211,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533098+0000","last_change":"2024-03-26T03:53:17.657919+0000","last_active":"2024-03-26T03:54:52.533098+0000","last_peered":"2024-03-26T03:54:52.533098+0000","last_clean":"2024-03-26T03:54:52.533098+0000","last_became_active":"2024-03-26T03:50:45.595111+0000","last_became_peered":"2024-03-26T03:50:45.595111+0000","last_unstale":"2024-03-26T03:54:52.533098+0000","last_undegraded":"2024-03-26T03:54:52.533098+0000","last_fullsized":"2024-03-26T03:54:52.533098+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531711+0000","last_change":"2024-03-26T03:54:29.288599+0000","last_active":"2024-03-26T03:54:52.531711+0000","last_peered":"2024-03-26T03:54:52.531711+0000","last_clean":"2024-03-26T03:54:52.531711+0000","last_became_active":"2024-03-26T03:54:29.288159+0000","last_became_peered":"2024-03-26T03:54:29.288159+0000","last_unstale":"2024-03-26T03:54:52.531711+0000","last_undegraded":"2024-03-26T03:54:52.531711+0000","last_fullsized":"2024-03-26T03:54:52.531711+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.147231+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530110+0000","last_change":"2024-03-26T03:54:29.454612+0000","last_active":"2024-03-26T03:54:52.530110+0000","last_peered":"2024-03-26T03:54:52.530110+0000","last_clean":"2024-03-26T03:54:52.530110+0000","last_became_active":"2024-03-26T03:54:29.454092+0000","last_became_peered":"2024-03-26T03:54:29.454092+0000","last_unstale":"2024-03-26T03:54:52.530110+0000","last_undegraded":"2024-03-26T03:54:52.530110+0000","last_fullsized":"2024-03-26T03:54:52.530110+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:17.213229+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":75,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533300+0000","last_change":"2024-03-26T03:54:29.485628+0000","last_active":"2024-03-26T03:54:52.533300+0000","last_peered":"2024-03-26T03:54:52.533300+0000","last_clean":"2024-03-26T03:54:52.533300+0000","last_became_active":"2024-03-26T03:54:29.289163+0000","last_became_peered":"2024-03-26T03:54:29.289163+0000","last_unstale":"2024-03-26T03:54:52.533300+0000","last_undegraded":"2024-03-26T03:54:52.533300+0000","last_fullsized":"2024-03-26T03:54:52.533300+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:52:33.466808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:33.466808+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"160'704","reported_seq":724,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.170794+0000","last_change":"2024-03-26T03:54:38.381911+0000","last_active":"2024-03-26T03:54:54.170794+0000","last_peered":"2024-03-26T03:54:54.170794+0000","last_clean":"2024-03-26T03:54:54.170794+0000","last_became_active":"2024-03-26T03:54:38.381579+0000","last_became_peered":"2024-03-26T03:54:38.381579+0000","last_unstale":"2024-03-26T03:54:54.170794+0000","last_undegraded":"2024-03-26T03:54:54.170794+0000","last_fullsized":"2024-03-26T03:54:54.170794+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":704,"log_dups_size":0,"ondisk_log_size":704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:59.910825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":704,"num_object_clones":0,"num_object_copies":2112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":704,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":704,"num_write_kb":45056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531200+0000","last_change":"2024-03-26T03:54:10.731222+0000","last_active":"2024-03-26T03:54:52.531200+0000","last_peered":"2024-03-26T03:54:52.531200+0000","last_clean":"2024-03-26T03:54:52.531200+0000","last_became_active":"2024-03-26T03:50:45.604631+0000","last_became_peered":"2024-03-26T03:50:45.604631+0000","last_unstale":"2024-03-26T03:54:52.531200+0000","last_undegraded":"2024-03-26T03:54:52.531200+0000","last_fullsized":"2024-03-26T03:54:52.531200+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:37.665420+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":237,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533291+0000","last_change":"2024-03-26T03:54:16.696733+0000","last_active":"2024-03-26T03:54:52.533291+0000","last_peered":"2024-03-26T03:54:52.533291+0000","last_clean":"2024-03-26T03:54:52.533291+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:54:52.533291+0000","last_undegraded":"2024-03-26T03:54:52.533291+0000","last_fullsized":"2024-03-26T03:54:52.533291+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:54:16.696602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.696602+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.478810+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":296,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530537+0000","last_change":"2024-03-26T03:54:19.799307+0000","last_active":"2024-03-26T03:54:52.530537+0000","last_peered":"2024-03-26T03:54:52.530537+0000","last_clean":"2024-03-26T03:54:52.530537+0000","last_became_active":"2024-03-26T03:50:45.587375+0000","last_became_peered":"2024-03-26T03:50:45.587375+0000","last_unstale":"2024-03-26T03:54:52.530537+0000","last_undegraded":"2024-03-26T03:54:52.530537+0000","last_fullsized":"2024-03-26T03:54:52.530537+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:35.017235+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":310,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532858+0000","last_change":"2024-03-26T03:53:19.706639+0000","last_active":"2024-03-26T03:54:52.532858+0000","last_peered":"2024-03-26T03:54:52.532858+0000","last_clean":"2024-03-26T03:54:52.532858+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:54:52.532858+0000","last_undegraded":"2024-03-26T03:54:52.532858+0000","last_fullsized":"2024-03-26T03:54:52.532858+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:19.706557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.706557+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"160'726","reported_seq":746,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.194419+0000","last_change":"2024-03-26T03:54:38.786252+0000","last_active":"2024-03-26T03:54:54.194419+0000","last_peered":"2024-03-26T03:54:54.194419+0000","last_clean":"2024-03-26T03:54:54.194419+0000","last_became_active":"2024-03-26T03:54:38.785773+0000","last_became_peered":"2024-03-26T03:54:38.785773+0000","last_unstale":"2024-03-26T03:54:54.194419+0000","last_undegraded":"2024-03-26T03:54:54.194419+0000","last_fullsized":"2024-03-26T03:54:54.194419+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":726,"log_dups_size":0,"ondisk_log_size":726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:45.231198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47579136,"num_objects":726,"num_object_clones":0,"num_object_copies":2178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":726,"num_write_kb":46464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":33,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532722+0000","last_change":"2024-03-26T03:54:30.757009+0000","last_active":"2024-03-26T03:54:52.532722+0000","last_peered":"2024-03-26T03:54:52.532722+0000","last_clean":"2024-03-26T03:54:52.532722+0000","last_became_active":"2024-03-26T03:54:29.304565+0000","last_became_peered":"2024-03-26T03:54:29.304565+0000","last_unstale":"2024-03-26T03:54:52.532722+0000","last_undegraded":"2024-03-26T03:54:52.532722+0000","last_fullsized":"2024-03-26T03:54:52.532722+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:38.498968+0000","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533336+0000","last_change":"2024-03-26T03:54:18.653671+0000","last_active":"2024-03-26T03:54:52.533336+0000","last_peered":"2024-03-26T03:54:52.533336+0000","last_clean":"2024-03-26T03:54:52.533336+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:54:52.533336+0000","last_undegraded":"2024-03-26T03:54:52.533336+0000","last_fullsized":"2024-03-26T03:54:52.533336+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.653555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.653555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:20.217041+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":282,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528594+0000","last_change":"2024-03-26T03:54:29.541247+0000","last_active":"2024-03-26T03:54:52.528594+0000","last_peered":"2024-03-26T03:54:52.528594+0000","last_clean":"2024-03-26T03:54:52.528594+0000","last_became_active":"2024-03-26T03:54:29.346977+0000","last_became_peered":"2024-03-26T03:54:29.346977+0000","last_unstale":"2024-03-26T03:54:52.528594+0000","last_undegraded":"2024-03-26T03:54:52.528594+0000","last_fullsized":"2024-03-26T03:54:52.528594+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:52:30.696931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:30.696931+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.537220+0000","last_change":"2024-03-26T03:53:19.352440+0000","last_active":"2024-03-26T03:54:52.537220+0000","last_peered":"2024-03-26T03:54:52.537220+0000","last_clean":"2024-03-26T03:54:52.537220+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:54:52.537220+0000","last_undegraded":"2024-03-26T03:54:52.537220+0000","last_fullsized":"2024-03-26T03:54:52.537220+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:19.352319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.352319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.837632+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"160'722","reported_seq":742,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.712880+0000","last_change":"2024-03-26T03:54:38.785847+0000","last_active":"2024-03-26T03:54:54.712880+0000","last_peered":"2024-03-26T03:54:54.712880+0000","last_clean":"2024-03-26T03:54:54.712880+0000","last_became_active":"2024-03-26T03:54:38.785420+0000","last_became_peered":"2024-03-26T03:54:38.785420+0000","last_unstale":"2024-03-26T03:54:54.712880+0000","last_undegraded":"2024-03-26T03:54:54.712880+0000","last_fullsized":"2024-03-26T03:54:54.712880+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":722,"log_dups_size":0,"ondisk_log_size":722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.043402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47316992,"num_objects":722,"num_object_clones":0,"num_object_copies":2166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":722,"num_write_kb":46208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531913+0000","last_change":"2024-03-26T03:54:29.359240+0000","last_active":"2024-03-26T03:54:52.531913+0000","last_peered":"2024-03-26T03:54:52.531913+0000","last_clean":"2024-03-26T03:54:52.531913+0000","last_became_active":"2024-03-26T03:54:29.358827+0000","last_became_peered":"2024-03-26T03:54:29.358827+0000","last_unstale":"2024-03-26T03:54:52.531913+0000","last_undegraded":"2024-03-26T03:54:52.531913+0000","last_fullsized":"2024-03-26T03:54:52.531913+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.664206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.664206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.537428+0000","last_change":"2024-03-26T03:53:35.307506+0000","last_active":"2024-03-26T03:54:52.537428+0000","last_peered":"2024-03-26T03:54:52.537428+0000","last_clean":"2024-03-26T03:54:52.537428+0000","last_became_active":"2024-03-26T03:48:48.018010+0000","last_became_peered":"2024-03-26T03:48:48.018010+0000","last_unstale":"2024-03-26T03:54:52.537428+0000","last_undegraded":"2024-03-26T03:54:52.537428+0000","last_fullsized":"2024-03-26T03:54:52.537428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:35.307371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:35.307371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.837599+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":237,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533298+0000","last_change":"2024-03-26T03:54:29.356025+0000","last_active":"2024-03-26T03:54:52.533298+0000","last_peered":"2024-03-26T03:54:52.533298+0000","last_clean":"2024-03-26T03:54:52.533298+0000","last_became_active":"2024-03-26T03:54:29.355742+0000","last_became_peered":"2024-03-26T03:54:29.355742+0000","last_unstale":"2024-03-26T03:54:52.533298+0000","last_undegraded":"2024-03-26T03:54:52.533298+0000","last_fullsized":"2024-03-26T03:54:52.533298+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.612556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.612556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:25.010157+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532771+0000","last_change":"2024-03-26T03:52:23.818501+0000","last_active":"2024-03-26T03:54:52.532771+0000","last_peered":"2024-03-26T03:54:52.532771+0000","last_clean":"2024-03-26T03:54:52.532771+0000","last_became_active":"2024-03-26T03:50:45.594555+0000","last_became_peered":"2024-03-26T03:50:45.594555+0000","last_unstale":"2024-03-26T03:54:52.532771+0000","last_undegraded":"2024-03-26T03:54:52.532771+0000","last_fullsized":"2024-03-26T03:54:52.532771+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:52:23.818354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.818354+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"160'767","reported_seq":787,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.519943+0000","last_change":"2024-03-26T03:54:38.382909+0000","last_active":"2024-03-26T03:54:54.519943+0000","last_peered":"2024-03-26T03:54:54.519943+0000","last_clean":"2024-03-26T03:54:54.519943+0000","last_became_active":"2024-03-26T03:54:38.382277+0000","last_became_peered":"2024-03-26T03:54:38.382277+0000","last_unstale":"2024-03-26T03:54:54.519943+0000","last_undegraded":"2024-03-26T03:54:54.519943+0000","last_fullsized":"2024-03-26T03:54:54.519943+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:52.587691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":49088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":205,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530571+0000","last_change":"2024-03-26T03:52:40.608799+0000","last_active":"2024-03-26T03:54:52.530571+0000","last_peered":"2024-03-26T03:54:52.530571+0000","last_clean":"2024-03-26T03:54:52.530571+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:54:52.530571+0000","last_undegraded":"2024-03-26T03:54:52.530571+0000","last_fullsized":"2024-03-26T03:54:52.530571+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.608679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.608679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.821934+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.539199+0000","last_change":"2024-03-26T03:54:33.962451+0000","last_active":"2024-03-26T03:54:52.539199+0000","last_peered":"2024-03-26T03:54:52.539199+0000","last_clean":"2024-03-26T03:54:52.539199+0000","last_became_active":"2024-03-26T03:54:29.428035+0000","last_became_peered":"2024-03-26T03:54:29.428035+0000","last_unstale":"2024-03-26T03:54:52.539199+0000","last_undegraded":"2024-03-26T03:54:52.539199+0000","last_fullsized":"2024-03-26T03:54:52.539199+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.962364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:46.314598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":245,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530851+0000","last_change":"2024-03-26T03:54:29.320669+0000","last_active":"2024-03-26T03:54:52.530851+0000","last_peered":"2024-03-26T03:54:52.530851+0000","last_clean":"2024-03-26T03:54:52.530851+0000","last_became_active":"2024-03-26T03:54:29.320372+0000","last_became_peered":"2024-03-26T03:54:29.320372+0000","last_unstale":"2024-03-26T03:54:52.530851+0000","last_undegraded":"2024-03-26T03:54:52.530851+0000","last_fullsized":"2024-03-26T03:54:52.530851+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:21.839296+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":267,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532152+0000","last_change":"2024-03-26T03:54:17.818278+0000","last_active":"2024-03-26T03:54:52.532152+0000","last_peered":"2024-03-26T03:54:52.532152+0000","last_clean":"2024-03-26T03:54:52.532152+0000","last_became_active":"2024-03-26T03:50:24.529600+0000","last_became_peered":"2024-03-26T03:50:24.529600+0000","last_unstale":"2024-03-26T03:54:52.532152+0000","last_undegraded":"2024-03-26T03:54:52.532152+0000","last_fullsized":"2024-03-26T03:54:52.532152+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.752191+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"160'689","reported_seq":709,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.173681+0000","last_change":"2024-03-26T03:54:38.787090+0000","last_active":"2024-03-26T03:54:54.173681+0000","last_peered":"2024-03-26T03:54:54.173681+0000","last_clean":"2024-03-26T03:54:54.173681+0000","last_became_active":"2024-03-26T03:54:38.786690+0000","last_became_peered":"2024-03-26T03:54:38.786690+0000","last_unstale":"2024-03-26T03:54:54.173681+0000","last_undegraded":"2024-03-26T03:54:54.173681+0000","last_fullsized":"2024-03-26T03:54:54.173681+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:57.561844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45154304,"num_objects":689,"num_object_clones":0,"num_object_copies":2067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":689,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":689,"num_write_kb":44096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":33,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527965+0000","last_change":"2024-03-26T03:54:32.411719+0000","last_active":"2024-03-26T03:54:52.527965+0000","last_peered":"2024-03-26T03:54:52.527965+0000","last_clean":"2024-03-26T03:54:52.527965+0000","last_became_active":"2024-03-26T03:54:29.297355+0000","last_became_peered":"2024-03-26T03:54:29.297355+0000","last_unstale":"2024-03-26T03:54:52.527965+0000","last_undegraded":"2024-03-26T03:54:52.527965+0000","last_fullsized":"2024-03-26T03:54:52.527965+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:40.760867+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":246,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530880+0000","last_change":"2024-03-26T03:53:37.581465+0000","last_active":"2024-03-26T03:54:52.530880+0000","last_peered":"2024-03-26T03:54:52.530880+0000","last_clean":"2024-03-26T03:54:52.530880+0000","last_became_active":"2024-03-26T03:50:45.595364+0000","last_became_peered":"2024-03-26T03:50:45.595364+0000","last_unstale":"2024-03-26T03:54:52.530880+0000","last_undegraded":"2024-03-26T03:54:52.530880+0000","last_fullsized":"2024-03-26T03:54:52.530880+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":223,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528709+0000","last_change":"2024-03-26T03:52:52.621874+0000","last_active":"2024-03-26T03:54:52.528709+0000","last_peered":"2024-03-26T03:54:52.528709+0000","last_clean":"2024-03-26T03:54:52.528709+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:54:52.528709+0000","last_undegraded":"2024-03-26T03:54:52.528709+0000","last_fullsized":"2024-03-26T03:54:52.528709+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:52.621745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:52:52.621745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"154'35","reported_seq":385,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532177+0000","last_change":"2024-03-26T03:54:29.460582+0000","last_active":"2024-03-26T03:54:52.532177+0000","last_peered":"2024-03-26T03:54:52.532177+0000","last_clean":"2024-03-26T03:54:52.532177+0000","last_became_active":"2024-03-26T03:54:29.371147+0000","last_became_peered":"2024-03-26T03:54:29.371147+0000","last_unstale":"2024-03-26T03:54:52.532177+0000","last_undegraded":"2024-03-26T03:54:52.532177+0000","last_fullsized":"2024-03-26T03:54:52.532177+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:46.129707+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530033+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:54:52.530033+0000","last_peered":"2024-03-26T03:54:52.530033+0000","last_clean":"2024-03-26T03:54:52.530033+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:54:52.530033+0000","last_undegraded":"2024-03-26T03:54:52.530033+0000","last_fullsized":"2024-03-26T03:54:52.530033+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.155849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"160'702","reported_seq":722,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.197722+0000","last_change":"2024-03-26T03:54:38.382935+0000","last_active":"2024-03-26T03:54:54.197722+0000","last_peered":"2024-03-26T03:54:54.197722+0000","last_clean":"2024-03-26T03:54:54.197722+0000","last_became_active":"2024-03-26T03:54:38.382581+0000","last_became_peered":"2024-03-26T03:54:38.382581+0000","last_unstale":"2024-03-26T03:54:54.197722+0000","last_undegraded":"2024-03-26T03:54:54.197722+0000","last_fullsized":"2024-03-26T03:54:54.197722+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":702,"log_dups_size":0,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:07.192417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46006272,"num_objects":702,"num_object_clones":0,"num_object_copies":2106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":702,"num_write_kb":44928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533413+0000","last_change":"2024-03-26T03:54:29.432774+0000","last_active":"2024-03-26T03:54:52.533413+0000","last_peered":"2024-03-26T03:54:52.533413+0000","last_clean":"2024-03-26T03:54:52.533413+0000","last_became_active":"2024-03-26T03:54:29.432431+0000","last_became_peered":"2024-03-26T03:54:29.432431+0000","last_unstale":"2024-03-26T03:54:52.533413+0000","last_undegraded":"2024-03-26T03:54:52.533413+0000","last_fullsized":"2024-03-26T03:54:52.533413+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:44.692790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:44.692790+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654306+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532050+0000","last_change":"2024-03-26T03:53:34.835708+0000","last_active":"2024-03-26T03:54:52.532050+0000","last_peered":"2024-03-26T03:54:52.532050+0000","last_clean":"2024-03-26T03:54:52.532050+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:54:52.532050+0000","last_undegraded":"2024-03-26T03:54:52.532050+0000","last_fullsized":"2024-03-26T03:54:52.532050+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.835581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.835581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:03.222485+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":248,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.539041+0000","last_change":"2024-03-26T03:54:29.333438+0000","last_active":"2024-03-26T03:54:52.539041+0000","last_peered":"2024-03-26T03:54:52.539041+0000","last_clean":"2024-03-26T03:54:52.539041+0000","last_became_active":"2024-03-26T03:54:29.319429+0000","last_became_peered":"2024-03-26T03:54:29.319429+0000","last_unstale":"2024-03-26T03:54:52.539041+0000","last_undegraded":"2024-03-26T03:54:52.539041+0000","last_fullsized":"2024-03-26T03:54:52.539041+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.962798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.962798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.276435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":251,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532373+0000","last_change":"2024-03-26T03:52:32.926568+0000","last_active":"2024-03-26T03:54:52.532373+0000","last_peered":"2024-03-26T03:54:52.532373+0000","last_clean":"2024-03-26T03:54:52.532373+0000","last_became_active":"2024-03-26T03:48:46.018875+0000","last_became_peered":"2024-03-26T03:48:46.018875+0000","last_unstale":"2024-03-26T03:54:52.532373+0000","last_undegraded":"2024-03-26T03:54:52.532373+0000","last_fullsized":"2024-03-26T03:54:52.532373+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.538028+0000","last_change":"2024-03-26T03:53:32.388419+0000","last_active":"2024-03-26T03:54:52.538028+0000","last_peered":"2024-03-26T03:54:52.538028+0000","last_clean":"2024-03-26T03:54:52.538028+0000","last_became_active":"2024-03-26T03:48:48.020337+0000","last_became_peered":"2024-03-26T03:48:48.020337+0000","last_unstale":"2024-03-26T03:54:52.538028+0000","last_undegraded":"2024-03-26T03:54:52.538028+0000","last_fullsized":"2024-03-26T03:54:52.538028+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.387809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.387809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:52.837617+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"160'667","reported_seq":687,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.183703+0000","last_change":"2024-03-26T03:54:38.378983+0000","last_active":"2024-03-26T03:54:54.183703+0000","last_peered":"2024-03-26T03:54:54.183703+0000","last_clean":"2024-03-26T03:54:54.183703+0000","last_became_active":"2024-03-26T03:54:38.378668+0000","last_became_peered":"2024-03-26T03:54:38.378668+0000","last_unstale":"2024-03-26T03:54:54.183703+0000","last_undegraded":"2024-03-26T03:54:54.183703+0000","last_fullsized":"2024-03-26T03:54:54.183703+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":667,"log_dups_size":0,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.160215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43712512,"num_objects":667,"num_object_clones":0,"num_object_copies":2001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":667,"num_write_kb":42688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532212+0000","last_change":"2024-03-26T03:54:29.268571+0000","last_active":"2024-03-26T03:54:52.532212+0000","last_peered":"2024-03-26T03:54:52.532212+0000","last_clean":"2024-03-26T03:54:52.532212+0000","last_became_active":"2024-03-26T03:54:29.268254+0000","last_became_peered":"2024-03-26T03:54:29.268254+0000","last_unstale":"2024-03-26T03:54:52.532212+0000","last_undegraded":"2024-03-26T03:54:52.532212+0000","last_fullsized":"2024-03-26T03:54:52.532212+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":481,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532014+0000","last_change":"2024-03-26T03:52:49.680807+0000","last_active":"2024-03-26T03:54:52.532014+0000","last_peered":"2024-03-26T03:54:52.532014+0000","last_clean":"2024-03-26T03:54:52.532014+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:54:52.532014+0000","last_undegraded":"2024-03-26T03:54:52.532014+0000","last_fullsized":"2024-03-26T03:54:52.532014+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:52:49.680677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:52:49.680677+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654447+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530776+0000","last_change":"2024-03-26T03:54:29.287818+0000","last_active":"2024-03-26T03:54:52.530776+0000","last_peered":"2024-03-26T03:54:52.530776+0000","last_clean":"2024-03-26T03:54:52.530776+0000","last_became_active":"2024-03-26T03:54:29.287411+0000","last_became_peered":"2024-03-26T03:54:29.287411+0000","last_unstale":"2024-03-26T03:54:52.530776+0000","last_undegraded":"2024-03-26T03:54:52.530776+0000","last_fullsized":"2024-03-26T03:54:52.530776+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:29.508252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":258,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529446+0000","last_change":"2024-03-26T03:52:23.761990+0000","last_active":"2024-03-26T03:54:52.529446+0000","last_peered":"2024-03-26T03:54:52.529446+0000","last_clean":"2024-03-26T03:54:52.529446+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:54:52.529446+0000","last_undegraded":"2024-03-26T03:54:52.529446+0000","last_fullsized":"2024-03-26T03:54:52.529446+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:52:23.761888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:23.761888+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533131+0000","last_change":"2024-03-26T03:53:28.678428+0000","last_active":"2024-03-26T03:54:52.533131+0000","last_peered":"2024-03-26T03:54:52.533131+0000","last_clean":"2024-03-26T03:54:52.533131+0000","last_became_active":"2024-03-26T03:48:48.028717+0000","last_became_peered":"2024-03-26T03:48:48.028717+0000","last_unstale":"2024-03-26T03:54:52.533131+0000","last_undegraded":"2024-03-26T03:54:52.533131+0000","last_fullsized":"2024-03-26T03:54:52.533131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654551+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"160'769","reported_seq":789,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.525760+0000","last_change":"2024-03-26T03:54:38.381614+0000","last_active":"2024-03-26T03:54:54.525760+0000","last_peered":"2024-03-26T03:54:54.525760+0000","last_clean":"2024-03-26T03:54:54.525760+0000","last_became_active":"2024-03-26T03:54:38.381253+0000","last_became_peered":"2024-03-26T03:54:38.381253+0000","last_unstale":"2024-03-26T03:54:54.525760+0000","last_undegraded":"2024-03-26T03:54:54.525760+0000","last_fullsized":"2024-03-26T03:54:54.525760+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":769,"log_dups_size":0,"ondisk_log_size":769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:39.725686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50397184,"num_objects":769,"num_object_clones":0,"num_object_copies":2307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":769,"num_write_kb":49216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531737+0000","last_change":"2024-03-26T03:54:29.371516+0000","last_active":"2024-03-26T03:54:52.531737+0000","last_peered":"2024-03-26T03:54:52.531737+0000","last_clean":"2024-03-26T03:54:52.531737+0000","last_became_active":"2024-03-26T03:54:29.370375+0000","last_became_peered":"2024-03-26T03:54:29.370375+0000","last_unstale":"2024-03-26T03:54:52.531737+0000","last_undegraded":"2024-03-26T03:54:52.531737+0000","last_fullsized":"2024-03-26T03:54:52.531737+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530192+0000","last_change":"2024-03-26T03:54:29.303701+0000","last_active":"2024-03-26T03:54:52.530192+0000","last_peered":"2024-03-26T03:54:52.530192+0000","last_clean":"2024-03-26T03:54:52.530192+0000","last_became_active":"2024-03-26T03:54:29.302729+0000","last_became_peered":"2024-03-26T03:54:29.302729+0000","last_unstale":"2024-03-26T03:54:52.530192+0000","last_undegraded":"2024-03-26T03:54:52.530192+0000","last_fullsized":"2024-03-26T03:54:52.530192+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:33.645447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.645447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.418212+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":600,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532021+0000","last_change":"2024-03-26T03:53:21.716954+0000","last_active":"2024-03-26T03:54:52.532021+0000","last_peered":"2024-03-26T03:54:52.532021+0000","last_clean":"2024-03-26T03:54:52.532021+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:54:52.532021+0000","last_undegraded":"2024-03-26T03:54:52.532021+0000","last_fullsized":"2024-03-26T03:54:52.532021+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:53:21.716822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:21.716822+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654500+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":362,"num_read_kb":357,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532637+0000","last_change":"2024-03-26T03:52:27.644692+0000","last_active":"2024-03-26T03:54:52.532637+0000","last_peered":"2024-03-26T03:54:52.532637+0000","last_clean":"2024-03-26T03:54:52.532637+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:54:52.532637+0000","last_undegraded":"2024-03-26T03:54:52.532637+0000","last_fullsized":"2024-03-26T03:54:52.532637+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:52:27.644541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:27.644541+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.821905+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":217,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532482+0000","last_change":"2024-03-26T03:54:11.750761+0000","last_active":"2024-03-26T03:54:52.532482+0000","last_peered":"2024-03-26T03:54:52.532482+0000","last_clean":"2024-03-26T03:54:52.532482+0000","last_became_active":"2024-03-26T03:48:48.022319+0000","last_became_peered":"2024-03-26T03:48:48.022319+0000","last_unstale":"2024-03-26T03:54:52.532482+0000","last_undegraded":"2024-03-26T03:54:52.532482+0000","last_fullsized":"2024-03-26T03:54:52.532482+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.167407+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"160'796","reported_seq":816,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.506953+0000","last_change":"2024-03-26T03:54:38.384402+0000","last_active":"2024-03-26T03:54:54.506953+0000","last_peered":"2024-03-26T03:54:54.506953+0000","last_clean":"2024-03-26T03:54:54.506953+0000","last_became_active":"2024-03-26T03:54:38.384087+0000","last_became_peered":"2024-03-26T03:54:38.384087+0000","last_unstale":"2024-03-26T03:54:54.506953+0000","last_undegraded":"2024-03-26T03:54:54.506953+0000","last_fullsized":"2024-03-26T03:54:54.506953+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:03.561589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52166656,"num_objects":796,"num_object_clones":0,"num_object_copies":2388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":796,"num_write_kb":50944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":210,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531866+0000","last_change":"2024-03-26T03:52:50.803477+0000","last_active":"2024-03-26T03:54:52.531866+0000","last_peered":"2024-03-26T03:54:52.531866+0000","last_clean":"2024-03-26T03:54:52.531866+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:54:52.531866+0000","last_undegraded":"2024-03-26T03:54:52.531866+0000","last_fullsized":"2024-03-26T03:54:52.531866+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:50.803356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:50.803356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":224,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531958+0000","last_change":"2024-03-26T03:53:31.824781+0000","last_active":"2024-03-26T03:54:52.531958+0000","last_peered":"2024-03-26T03:54:52.531958+0000","last_clean":"2024-03-26T03:54:52.531958+0000","last_became_active":"2024-03-26T03:48:44.013358+0000","last_became_peered":"2024-03-26T03:48:44.013358+0000","last_unstale":"2024-03-26T03:54:52.531958+0000","last_undegraded":"2024-03-26T03:54:52.531958+0000","last_fullsized":"2024-03-26T03:54:52.531958+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":269,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531953+0000","last_change":"2024-03-26T03:54:22.831334+0000","last_active":"2024-03-26T03:54:52.531953+0000","last_peered":"2024-03-26T03:54:52.531953+0000","last_clean":"2024-03-26T03:54:52.531953+0000","last_became_active":"2024-03-26T03:48:46.209559+0000","last_became_peered":"2024-03-26T03:48:46.209559+0000","last_unstale":"2024-03-26T03:54:52.531953+0000","last_undegraded":"2024-03-26T03:54:52.531953+0000","last_fullsized":"2024-03-26T03:54:52.531953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:29.785367+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":155,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.534962+0000","last_change":"2024-03-26T03:54:18.813803+0000","last_active":"2024-03-26T03:54:52.534962+0000","last_peered":"2024-03-26T03:54:52.534962+0000","last_clean":"2024-03-26T03:54:52.534962+0000","last_became_active":"2024-03-26T03:50:45.593137+0000","last_became_peered":"2024-03-26T03:50:45.593137+0000","last_unstale":"2024-03-26T03:54:52.534962+0000","last_undegraded":"2024-03-26T03:54:52.534962+0000","last_fullsized":"2024-03-26T03:54:52.534962+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:22.825611+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"160'778","reported_seq":798,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.702216+0000","last_change":"2024-03-26T03:54:38.379391+0000","last_active":"2024-03-26T03:54:54.702216+0000","last_peered":"2024-03-26T03:54:54.702216+0000","last_clean":"2024-03-26T03:54:54.702216+0000","last_became_active":"2024-03-26T03:54:38.379010+0000","last_became_peered":"2024-03-26T03:54:38.379010+0000","last_unstale":"2024-03-26T03:54:54.702216+0000","last_undegraded":"2024-03-26T03:54:54.702216+0000","last_fullsized":"2024-03-26T03:54:54.702216+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":778,"log_dups_size":0,"ondisk_log_size":778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:54.551348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50987008,"num_objects":778,"num_object_clones":0,"num_object_copies":2334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":778,"num_write_kb":49792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":212,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529890+0000","last_change":"2024-03-26T03:54:29.362049+0000","last_active":"2024-03-26T03:54:52.529890+0000","last_peered":"2024-03-26T03:54:52.529890+0000","last_clean":"2024-03-26T03:54:52.529890+0000","last_became_active":"2024-03-26T03:54:29.344504+0000","last_became_peered":"2024-03-26T03:54:29.344504+0000","last_unstale":"2024-03-26T03:54:52.529890+0000","last_undegraded":"2024-03-26T03:54:52.529890+0000","last_fullsized":"2024-03-26T03:54:52.529890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822084+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531788+0000","last_change":"2024-03-26T03:54:29.432812+0000","last_active":"2024-03-26T03:54:52.531788+0000","last_peered":"2024-03-26T03:54:52.531788+0000","last_clean":"2024-03-26T03:54:52.531788+0000","last_became_active":"2024-03-26T03:54:29.432437+0000","last_became_peered":"2024-03-26T03:54:29.432437+0000","last_unstale":"2024-03-26T03:54:52.531788+0000","last_undegraded":"2024-03-26T03:54:52.531788+0000","last_fullsized":"2024-03-26T03:54:52.531788+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:23.984390+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":247,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533003+0000","last_change":"2024-03-26T03:54:15.883651+0000","last_active":"2024-03-26T03:54:52.533003+0000","last_peered":"2024-03-26T03:54:52.533003+0000","last_clean":"2024-03-26T03:54:52.533003+0000","last_became_active":"2024-03-26T03:48:46.020103+0000","last_became_peered":"2024-03-26T03:48:46.020103+0000","last_unstale":"2024-03-26T03:54:52.533003+0000","last_undegraded":"2024-03-26T03:54:52.533003+0000","last_fullsized":"2024-03-26T03:54:52.533003+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:27.995304+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"154'10","reported_seq":312,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532890+0000","last_change":"2024-03-26T03:54:29.476464+0000","last_active":"2024-03-26T03:54:52.532890+0000","last_peered":"2024-03-26T03:54:52.532890+0000","last_clean":"2024-03-26T03:54:52.532890+0000","last_became_active":"2024-03-26T03:54:29.360917+0000","last_became_peered":"2024-03-26T03:54:29.360917+0000","last_unstale":"2024-03-26T03:54:52.532890+0000","last_undegraded":"2024-03-26T03:54:52.532890+0000","last_fullsized":"2024-03-26T03:54:52.532890+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822142+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"160'686","reported_seq":706,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.202610+0000","last_change":"2024-03-26T03:54:38.786182+0000","last_active":"2024-03-26T03:54:54.202610+0000","last_peered":"2024-03-26T03:54:54.202610+0000","last_clean":"2024-03-26T03:54:54.202610+0000","last_became_active":"2024-03-26T03:54:38.785725+0000","last_became_peered":"2024-03-26T03:54:38.785725+0000","last_unstale":"2024-03-26T03:54:54.202610+0000","last_undegraded":"2024-03-26T03:54:54.202610+0000","last_fullsized":"2024-03-26T03:54:54.202610+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:51.416780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44957696,"num_objects":686,"num_object_clones":0,"num_object_copies":2058,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":686,"num_write_kb":43904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":219,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532973+0000","last_change":"2024-03-26T03:53:34.666684+0000","last_active":"2024-03-26T03:54:52.532973+0000","last_peered":"2024-03-26T03:54:52.532973+0000","last_clean":"2024-03-26T03:54:52.532973+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:54:52.532973+0000","last_undegraded":"2024-03-26T03:54:52.532973+0000","last_fullsized":"2024-03-26T03:54:52.532973+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.666607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.666607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":226,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532735+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:54:52.532735+0000","last_peered":"2024-03-26T03:54:52.532735+0000","last_clean":"2024-03-26T03:54:52.532735+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:54:52.532735+0000","last_undegraded":"2024-03-26T03:54:52.532735+0000","last_fullsized":"2024-03-26T03:54:52.532735+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:56.822204+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":278,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528233+0000","last_change":"2024-03-26T03:54:16.339624+0000","last_active":"2024-03-26T03:54:52.528233+0000","last_peered":"2024-03-26T03:54:52.528233+0000","last_clean":"2024-03-26T03:54:52.528233+0000","last_became_active":"2024-03-26T03:50:45.592299+0000","last_became_peered":"2024-03-26T03:50:45.592299+0000","last_unstale":"2024-03-26T03:54:52.528233+0000","last_undegraded":"2024-03-26T03:54:52.528233+0000","last_fullsized":"2024-03-26T03:54:52.528233+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:36.150917+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"154'10","reported_seq":68,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.531684+0000","last_change":"2024-03-26T03:54:34.943516+0000","last_active":"2024-03-26T03:54:52.531684+0000","last_peered":"2024-03-26T03:54:52.531684+0000","last_clean":"2024-03-26T03:54:52.531684+0000","last_became_active":"2024-03-26T03:54:29.356520+0000","last_became_peered":"2024-03-26T03:54:29.356520+0000","last_unstale":"2024-03-26T03:54:52.531684+0000","last_undegraded":"2024-03-26T03:54:52.531684+0000","last_fullsized":"2024-03-26T03:54:52.531684+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:56.388820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"160'767","reported_seq":786,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.720201+0000","last_change":"2024-03-26T03:54:38.786218+0000","last_active":"2024-03-26T03:54:54.720201+0000","last_peered":"2024-03-26T03:54:54.720201+0000","last_clean":"2024-03-26T03:54:54.720201+0000","last_became_active":"2024-03-26T03:54:38.785826+0000","last_became_peered":"2024-03-26T03:54:38.785826+0000","last_unstale":"2024-03-26T03:54:54.720201+0000","last_undegraded":"2024-03-26T03:54:54.720201+0000","last_fullsized":"2024-03-26T03:54:54.720201+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:50.939382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":49088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":211,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529232+0000","last_change":"2024-03-26T03:54:29.432125+0000","last_active":"2024-03-26T03:54:52.529232+0000","last_peered":"2024-03-26T03:54:52.529232+0000","last_clean":"2024-03-26T03:54:52.529232+0000","last_became_active":"2024-03-26T03:54:29.431697+0000","last_became_peered":"2024-03-26T03:54:29.431697+0000","last_unstale":"2024-03-26T03:54:52.529232+0000","last_undegraded":"2024-03-26T03:54:52.529232+0000","last_fullsized":"2024-03-26T03:54:52.529232+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:40.101352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.533220+0000","last_change":"2024-03-26T03:54:29.531706+0000","last_active":"2024-03-26T03:54:52.533220+0000","last_peered":"2024-03-26T03:54:52.533220+0000","last_clean":"2024-03-26T03:54:52.533220+0000","last_became_active":"2024-03-26T03:54:29.531371+0000","last_became_peered":"2024-03-26T03:54:29.531371+0000","last_unstale":"2024-03-26T03:54:52.533220+0000","last_undegraded":"2024-03-26T03:54:52.533220+0000","last_fullsized":"2024-03-26T03:54:52.533220+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:55.654403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":150,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532127+0000","last_change":"2024-03-26T03:54:29.536599+0000","last_active":"2024-03-26T03:54:52.532127+0000","last_peered":"2024-03-26T03:54:52.532127+0000","last_clean":"2024-03-26T03:54:52.532127+0000","last_became_active":"2024-03-26T03:54:29.358363+0000","last_became_peered":"2024-03-26T03:54:29.358363+0000","last_unstale":"2024-03-26T03:54:52.532127+0000","last_undegraded":"2024-03-26T03:54:52.532127+0000","last_fullsized":"2024-03-26T03:54:52.532127+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:43.928019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527814+0000","last_change":"2024-03-26T03:54:29.347549+0000","last_active":"2024-03-26T03:54:52.527814+0000","last_peered":"2024-03-26T03:54:52.527814+0000","last_clean":"2024-03-26T03:54:52.527814+0000","last_became_active":"2024-03-26T03:54:29.346424+0000","last_became_peered":"2024-03-26T03:54:29.346424+0000","last_unstale":"2024-03-26T03:54:52.527814+0000","last_undegraded":"2024-03-26T03:54:52.527814+0000","last_fullsized":"2024-03-26T03:54:52.527814+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:37.885050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:52:37.885050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"160'737","reported_seq":755,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.528257+0000","last_change":"2024-03-26T03:54:38.382486+0000","last_active":"2024-03-26T03:54:54.528257+0000","last_peered":"2024-03-26T03:54:54.528257+0000","last_clean":"2024-03-26T03:54:54.528257+0000","last_became_active":"2024-03-26T03:54:38.382141+0000","last_became_peered":"2024-03-26T03:54:38.382141+0000","last_unstale":"2024-03-26T03:54:54.528257+0000","last_undegraded":"2024-03-26T03:54:54.528257+0000","last_fullsized":"2024-03-26T03:54:54.528257+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:45.342264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48300032,"num_objects":737,"num_object_clones":0,"num_object_copies":2211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":737,"num_write_kb":47168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530880+0000","last_change":"2024-03-26T03:53:28.635973+0000","last_active":"2024-03-26T03:54:52.530880+0000","last_peered":"2024-03-26T03:54:52.530880+0000","last_clean":"2024-03-26T03:54:52.530880+0000","last_became_active":"2024-03-26T03:48:50.034514+0000","last_became_peered":"2024-03-26T03:48:50.034514+0000","last_unstale":"2024-03-26T03:54:52.530880+0000","last_undegraded":"2024-03-26T03:54:52.530880+0000","last_fullsized":"2024-03-26T03:54:52.530880+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":300,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528571+0000","last_change":"2024-03-26T03:54:33.386428+0000","last_active":"2024-03-26T03:54:52.528571+0000","last_peered":"2024-03-26T03:54:52.528571+0000","last_clean":"2024-03-26T03:54:52.528571+0000","last_became_active":"2024-03-26T03:48:48.020586+0000","last_became_peered":"2024-03-26T03:48:48.020586+0000","last_unstale":"2024-03-26T03:54:52.528571+0000","last_undegraded":"2024-03-26T03:54:52.528571+0000","last_fullsized":"2024-03-26T03:54:52.528571+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:01.105650+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":32,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532255+0000","last_change":"2024-03-26T03:54:31.922380+0000","last_active":"2024-03-26T03:54:52.532255+0000","last_peered":"2024-03-26T03:54:52.532255+0000","last_clean":"2024-03-26T03:54:52.532255+0000","last_became_active":"2024-03-26T03:54:29.377662+0000","last_became_peered":"2024-03-26T03:54:29.377662+0000","last_unstale":"2024-03-26T03:54:52.532255+0000","last_undegraded":"2024-03-26T03:54:52.532255+0000","last_fullsized":"2024-03-26T03:54:52.532255+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:33.087626+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":267,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532470+0000","last_change":"2024-03-26T03:54:29.386220+0000","last_active":"2024-03-26T03:54:52.532470+0000","last_peered":"2024-03-26T03:54:52.532470+0000","last_clean":"2024-03-26T03:54:52.532470+0000","last_became_active":"2024-03-26T03:54:29.299380+0000","last_became_peered":"2024-03-26T03:54:29.299380+0000","last_unstale":"2024-03-26T03:54:52.532470+0000","last_undegraded":"2024-03-26T03:54:52.532470+0000","last_fullsized":"2024-03-26T03:54:52.532470+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.835587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.835587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.048215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"160'734","reported_seq":754,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.196633+0000","last_change":"2024-03-26T03:54:38.377692+0000","last_active":"2024-03-26T03:54:54.196633+0000","last_peered":"2024-03-26T03:54:54.196633+0000","last_clean":"2024-03-26T03:54:54.196633+0000","last_became_active":"2024-03-26T03:54:38.377276+0000","last_became_peered":"2024-03-26T03:54:38.377276+0000","last_unstale":"2024-03-26T03:54:54.196633+0000","last_undegraded":"2024-03-26T03:54:54.196633+0000","last_fullsized":"2024-03-26T03:54:54.196633+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":734,"log_dups_size":0,"ondisk_log_size":734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:00.861962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48103424,"num_objects":734,"num_object_clones":0,"num_object_copies":2202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":734,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":734,"num_write_kb":46976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":82,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530711+0000","last_change":"2024-03-26T03:54:29.320001+0000","last_active":"2024-03-26T03:54:52.530711+0000","last_peered":"2024-03-26T03:54:52.530711+0000","last_clean":"2024-03-26T03:54:52.530711+0000","last_became_active":"2024-03-26T03:54:29.310202+0000","last_became_peered":"2024-03-26T03:54:29.310202+0000","last_unstale":"2024-03-26T03:54:52.530711+0000","last_undegraded":"2024-03-26T03:54:52.530711+0000","last_fullsized":"2024-03-26T03:54:52.530711+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:33.239133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":231,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532391+0000","last_change":"2024-03-26T03:54:24.803551+0000","last_active":"2024-03-26T03:54:52.532391+0000","last_peered":"2024-03-26T03:54:52.532391+0000","last_clean":"2024-03-26T03:54:52.532391+0000","last_became_active":"2024-03-26T03:50:45.593201+0000","last_became_peered":"2024-03-26T03:50:45.593201+0000","last_unstale":"2024-03-26T03:54:52.532391+0000","last_undegraded":"2024-03-26T03:54:52.532391+0000","last_fullsized":"2024-03-26T03:54:52.532391+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:26.568745+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":225,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.527914+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:54:52.527914+0000","last_peered":"2024-03-26T03:54:52.527914+0000","last_clean":"2024-03-26T03:54:52.527914+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:54:52.527914+0000","last_undegraded":"2024-03-26T03:54:52.527914+0000","last_fullsized":"2024-03-26T03:54:52.527914+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.372670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":292,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528171+0000","last_change":"2024-03-26T03:54:29.355692+0000","last_active":"2024-03-26T03:54:52.528171+0000","last_peered":"2024-03-26T03:54:52.528171+0000","last_clean":"2024-03-26T03:54:52.528171+0000","last_became_active":"2024-03-26T03:54:29.270339+0000","last_became_peered":"2024-03-26T03:54:29.270339+0000","last_unstale":"2024-03-26T03:54:52.528171+0000","last_undegraded":"2024-03-26T03:54:52.528171+0000","last_fullsized":"2024-03-26T03:54:52.528171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:53.402641+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"160'767","reported_seq":787,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.527601+0000","last_change":"2024-03-26T03:54:38.382794+0000","last_active":"2024-03-26T03:54:54.527601+0000","last_peered":"2024-03-26T03:54:54.527601+0000","last_clean":"2024-03-26T03:54:54.527601+0000","last_became_active":"2024-03-26T03:54:38.382452+0000","last_became_peered":"2024-03-26T03:54:38.382452+0000","last_unstale":"2024-03-26T03:54:54.527601+0000","last_undegraded":"2024-03-26T03:54:54.527601+0000","last_fullsized":"2024-03-26T03:54:54.527601+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:42.718898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50266112,"num_objects":767,"num_object_clones":0,"num_object_copies":2301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":767,"num_write_kb":49088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.541083+0000","last_change":"2024-03-26T03:54:31.003536+0000","last_active":"2024-03-26T03:54:52.541083+0000","last_peered":"2024-03-26T03:54:52.541083+0000","last_clean":"2024-03-26T03:54:52.541083+0000","last_became_active":"2024-03-26T03:54:29.296384+0000","last_became_peered":"2024-03-26T03:54:29.296384+0000","last_unstale":"2024-03-26T03:54:52.541083+0000","last_undegraded":"2024-03-26T03:54:52.541083+0000","last_fullsized":"2024-03-26T03:54:52.541083+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.003442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.003442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:34.045451+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":216,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532356+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:54:52.532356+0000","last_peered":"2024-03-26T03:54:52.532356+0000","last_clean":"2024-03-26T03:54:52.532356+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:54:52.532356+0000","last_undegraded":"2024-03-26T03:54:52.532356+0000","last_fullsized":"2024-03-26T03:54:52.532356+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:48.412746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":161,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.529321+0000","last_change":"2024-03-26T03:54:29.289038+0000","last_active":"2024-03-26T03:54:52.529321+0000","last_peered":"2024-03-26T03:54:52.529321+0000","last_clean":"2024-03-26T03:54:52.529321+0000","last_became_active":"2024-03-26T03:54:29.283289+0000","last_became_peered":"2024-03-26T03:54:29.283289+0000","last_unstale":"2024-03-26T03:54:52.529321+0000","last_undegraded":"2024-03-26T03:54:52.529321+0000","last_fullsized":"2024-03-26T03:54:52.529321+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:36.547229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":249,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.530219+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:54:52.530219+0000","last_peered":"2024-03-26T03:54:52.530219+0000","last_clean":"2024-03-26T03:54:52.530219+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:54:52.530219+0000","last_undegraded":"2024-03-26T03:54:52.530219+0000","last_fullsized":"2024-03-26T03:54:52.530219+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:15.733851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"160'699","reported_seq":719,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:54.193237+0000","last_change":"2024-03-26T03:54:38.382779+0000","last_active":"2024-03-26T03:54:54.193237+0000","last_peered":"2024-03-26T03:54:54.193237+0000","last_clean":"2024-03-26T03:54:54.193237+0000","last_became_active":"2024-03-26T03:54:38.382404+0000","last_became_peered":"2024-03-26T03:54:38.382404+0000","last_unstale":"2024-03-26T03:54:54.193237+0000","last_undegraded":"2024-03-26T03:54:54.193237+0000","last_fullsized":"2024-03-26T03:54:54.193237+0000","mapping_epoch":155,"log_start":"0'0","ondisk_log_start":"0'0","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:41.861016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45809664,"num_objects":699,"num_object_clones":0,"num_object_copies":2097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":699,"num_write_kb":44736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528714+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:54:52.528714+0000","last_peered":"2024-03-26T03:54:52.528714+0000","last_clean":"2024-03-26T03:54:52.528714+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:54:52.528714+0000","last_undegraded":"2024-03-26T03:54:52.528714+0000","last_fullsized":"2024-03-26T03:54:52.528714+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:32.239985+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":300,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.539512+0000","last_change":"2024-03-26T03:53:17.373928+0000","last_active":"2024-03-26T03:54:52.539512+0000","last_peered":"2024-03-26T03:54:52.539512+0000","last_clean":"2024-03-26T03:54:52.539512+0000","last_became_active":"2024-03-26T03:48:48.020559+0000","last_became_peered":"2024-03-26T03:48:48.020559+0000","last_unstale":"2024-03-26T03:54:52.539512+0000","last_undegraded":"2024-03-26T03:54:52.539512+0000","last_fullsized":"2024-03-26T03:54:52.539512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:53:17.373796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.373796+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:54:54.826889+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":32,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.532057+0000","last_change":"2024-03-26T03:54:33.815241+0000","last_active":"2024-03-26T03:54:52.532057+0000","last_peered":"2024-03-26T03:54:52.532057+0000","last_clean":"2024-03-26T03:54:52.532057+0000","last_became_active":"2024-03-26T03:54:29.399842+0000","last_became_peered":"2024-03-26T03:54:29.399842+0000","last_unstale":"2024-03-26T03:54:52.532057+0000","last_undegraded":"2024-03-26T03:54:52.532057+0000","last_fullsized":"2024-03-26T03:54:52.532057+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:55:37.376467+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":264,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-03-26T03:54:52.528309+0000","last_change":"2024-03-26T03:54:29.380642+0000","last_active":"2024-03-26T03:54:52.528309+0000","last_peered":"2024-03-26T03:54:52.528309+0000","last_clean":"2024-03-26T03:54:52.528309+0000","last_became_active":"2024-03-26T03:54:29.296174+0000","last_became_peered":"2024-03-26T03:54:29.296174+0000","last_unstale":"2024-03-26T03:54:52.528309+0000","last_undegraded":"2024-03-26T03:54:52.528309+0000","last_fullsized":"2024-03-26T03:54:52.528309+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:52:31.734108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:31.734108+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":767426560,"num_objects":11710,"num_object_clones":0,"num_object_copies":35130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11710,"num_write_kb":749440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2304507904,"data_stored":2304507904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11710,"ondisk_log_size":11710,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":362,"num_read_kb":357,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":193,"num_bytes_recovered":3152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070542,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291836,"kb_used_data":192624,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93453316,"statfs":{"total":95995035648,"available":95696195584,"internally_reserved":0,"allocated":197246976,"data_stored":196794576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":101579210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":425740,"kb_used_data":290432,"kb_used_omap":1,"kb_used_meta":135294,"kb_avail":93319412,"statfs":{"total":95995035648,"available":95559077888,"internally_reserved":0,"allocated":297402368,"data_stored":296933790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":138541509},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738482,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":480516,"kb_used_data":332732,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93264636,"statfs":{"total":95995035648,"available":95502987264,"internally_reserved":0,"allocated":340717568,"data_stored":340253038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":151321029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411432,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":549332,"kb_used_data":409864,"kb_used_omap":1,"kb_used_meta":139454,"kb_avail":93195820,"statfs":{"total":95995035648,"available":95432519680,"internally_reserved":0,"allocated":419700736,"data_stored":419286504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":142801349},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182498,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953565,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435944,"kb_used_data":328408,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93309208,"statfs":{"total":95995035648,"available":95548628992,"internally_reserved":0,"allocated":336289792,"data_stored":335859226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659223,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":452192,"kb_used_data":366872,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":93292960,"statfs":{"total":95995035648,"available":95531991040,"internally_reserved":0,"allocated":375676928,"data_stored":375246840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":411828224,"data_stored":411828224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":351207424,"data_stored":351207424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":493682688,"data_stored":493682688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":416088064,"data_stored":416088064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":408354816,"data_stored":408354816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":223346688,"data_stored":223346688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:55:01.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-26T03:55:01.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:01.732 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:01.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:01.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:01.880 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:01.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:01.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:02.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:02.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:02.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:02.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:02.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:02.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:02.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:02.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:02.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:02.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 12 16 30799 30783 160.299 121.188 0.0541453 0.00621089 2024-03-26T03:55:02.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:02.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:02.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:02.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:02 smithi086 ceph-mon[32965]: pgmap v254: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-26T03:55:02.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:02 smithi006 ceph-mon[37501]: pgmap v254: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-26T03:55:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:02 smithi006 ceph-mon[29446]: pgmap v254: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-26T03:55:02.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:02.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:02.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:03.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:03.082 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:03.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:03.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:03.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:03.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:03.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:03.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:03.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 13 16 32804 32788 157.607 125.312 0.0100655 0.00633949 2024-03-26T03:55:03.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:03.634 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:03.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:03.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:03.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:03.864 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:03.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:03.886 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:03.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:03.955 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:03.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:04.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:04.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:04.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:04.222 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:04.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:04.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:04.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:04.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:04.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:04.501 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:04.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:04.542 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:04.584 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 14 16 34820 34804 155.349 126 0.0117227 0.00643018 2024-03-26T03:55:04.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:04.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:04.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:04.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:04.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:04 smithi086 ceph-mon[32965]: pgmap v255: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 951 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-26T03:55:04.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:04.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:04.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:04 smithi006 ceph-mon[37501]: pgmap v255: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 951 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-26T03:55:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:04 smithi006 ceph-mon[29446]: pgmap v255: 148 pgs: 148 active+clean; 1.5 GiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 951 B/s rd, 141 MiB/s wr, 2.25k op/s 2024-03-26T03:55:04.919 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:04.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:05.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:05.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:05.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:05.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:05.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:05.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:05.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:05.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:05.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:05.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:05.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:05.585 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 15 16 36982 36966 153.997 135.125 0.00617512 0.00648784 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: osdmap e161: 8 total, 8 up, 6 in 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:55:05.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:55:05.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:05.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:05.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:55:05.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:05.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:05.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:05.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:55:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:55:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: osdmap e161: 8 total, 8 up, 6 in 2024-03-26T03:55:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:55:05.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:05 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:55:05.902 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:05.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:06.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:06.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:55:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:55:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: osdmap e161: 8 total, 8 up, 6 in 2024-03-26T03:55:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:55:06.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:05 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:55:06.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:06.142 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:55:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:55:06.142 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T03:55:06.142 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:55:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:06.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:06.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:06.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:06.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:06.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:06.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:06.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:06.585 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 16 16 38883 38867 151.797 118.812 0.00547083 0.00658232 2024-03-26T03:55:06.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:06.653 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:06.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:06.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:06.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:06 smithi006 ceph-mon[37501]: pgmap v257: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:55:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:06 smithi006 ceph-mon[37501]: osdmap e162: 8 total, 8 up, 6 in 2024-03-26T03:55:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:06 smithi006 ceph-mon[29446]: pgmap v257: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:55:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:06 smithi006 ceph-mon[29446]: osdmap e162: 8 total, 8 up, 6 in 2024-03-26T03:55:06.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:06.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:06.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:06.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:07.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:07.073 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:06 smithi086 ceph-mon[32965]: pgmap v257: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:55:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:06 smithi086 ceph-mon[32965]: osdmap e162: 8 total, 8 up, 6 in 2024-03-26T03:55:07.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:55:07.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:07.238 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:07.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:07.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:07.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:07.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:07.516 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:07.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:07.585 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 17 16 40754 40738 149.746 116.938 0.00611039 0.00667196 2024-03-26T03:55:07.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:07.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:07.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:07.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:07.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:07.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:07.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:08.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:08.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:08.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:08.158 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:08.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:08.231 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:08.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:08.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:08.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.331 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:08.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:08.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:08.585 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 18 16 42513 42497 147.535 109.938 0.00448165 0.00677169 2024-03-26T03:55:08.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:08.625 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:55:08.645 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:08.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:08.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:08.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[37501]: pgmap v259: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2929359928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[29446]: pgmap v259: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2929359928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:08 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:08.925 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:08.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:09.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:09.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:09.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:09.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:08 smithi086 ceph-mon[32965]: pgmap v259: 148 pgs: 148 active+clean; 2.1 GiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 129 MiB/s wr, 2.06k op/s 2024-03-26T03:55:09.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:08 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2929359928' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:09.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:08 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:09.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:09.190 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:09.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:09.328 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:09.430 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:09.472 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:09.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:09.587 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 19 16 44261 44245 145.512 109.25 0.0076508 0.00686645 2024-03-26T03:55:09.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:09.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:09.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:09.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:09.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:09.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[37501]: osdmap e163: 8 total, 8 up, 6 in 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[37501]: osdmap e164: 8 total, 8 up, 6 in 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:09.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:09.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[29446]: osdmap e163: 8 total, 8 up, 6 in 2024-03-26T03:55:09.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:09.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:09 smithi006 ceph-mon[29446]: osdmap e164: 8 total, 8 up, 6 in 2024-03-26T03:55:09.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:10.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:09 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:09 smithi086 ceph-mon[32965]: osdmap e163: 8 total, 8 up, 6 in 2024-03-26T03:55:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:09 smithi086 ceph-mon[32965]: osdmap e164: 8 total, 8 up, 6 in 2024-03-26T03:55:10.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:10.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:10.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:10.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:10.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:10.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:10.432 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:10.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:10.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:10.586 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:55:10.587091+0000 min lat: 0.00117332 max lat: 0.101083 avg lat: 0.00693531 2024-03-26T03:55:10.586 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:55:10.586 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 20 16 46136 46120 144.095 117.188 0.00468194 0.00693531 2024-03-26T03:55:10.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:10.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:10.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:10.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:10.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:10.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:10.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:10 smithi006 ceph-mon[37501]: pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T03:55:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:10 smithi006 ceph-mon[29446]: pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T03:55:10.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.001 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:10 smithi086 ceph-mon[32965]: pgmap v262: 148 pgs: 148 active+clean; 2.4 GiB data, 7.7 GiB used, 529 GiB / 536 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T03:55:11.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:11.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:11.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.295 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:11.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:11.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:11.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:11.586 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 21 16 47813 47797 142.224 104.812 0.0128274 0.00702595 2024-03-26T03:55:11.637 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:55:11.677 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:55:11.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:11.722 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:55:11.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:11.769 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:55:11.781 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T03:55:11.781 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T03:55:11.781 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/658388112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:11.782 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T03:55:11.782 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T03:55:11.782 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/658388112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:11.811 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:55:11.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:11.853 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:55:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:11.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:11.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:11.892 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:55:11.938 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:55:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:55:11.941 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:55:11.942 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:55:11.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:12.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:11 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T03:55:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:11 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T03:55:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/658388112' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:12.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:12.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:12.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:12.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:12.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:12.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:12.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:12.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:12.588 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 22 16 49501 49485 140.546 105.5 0.00503257 0.00711003 2024-03-26T03:55:12.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:12.666 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 1 pgp_num to 1 2024-03-26T03:55:12.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:12.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:12.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[37501]: pgmap v263: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2274222360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[29446]: pgmap v263: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2274222360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:12 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:12.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:12.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:12.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:13.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:55:13.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:12 smithi086 ceph-mon[32965]: pgmap v263: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2274222360' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:12 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-26T03:55:13.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.133 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:13.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:13.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:13.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:13.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:13.412 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:13.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:13.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:13.588 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 23 16 51211 51195 139.082 106.875 0.00522684 0.00718484 2024-03-26T03:55:13.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:13.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:13.830 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:13.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:13.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:13.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:13.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:13.992 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T03:55:13.992 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:13 smithi006 ceph-mon[37501]: osdmap e165: 8 total, 8 up, 6 in 2024-03-26T03:55:13.992 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T03:55:13.992 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:13 smithi006 ceph-mon[29446]: osdmap e165: 8 total, 8 up, 6 in 2024-03-26T03:55:14.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-26T03:55:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:13 smithi086 ceph-mon[32965]: osdmap e165: 8 total, 8 up, 6 in 2024-03-26T03:55:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:14.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:14.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:14.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:14.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:14.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:14.336 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:14.375 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:55:14.375 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:55:14.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.384 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:14.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:14.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:14.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:14.588 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 24 15 52719 52704 137.216 94.3125 0.010466 0.00728304 2024-03-26T03:55:14.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:14.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:14.647 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:14.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:14.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:14.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: pgmap v265: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: 5.14 scrub starts 2024-03-26T03:55:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: 5.14 scrub ok 2024-03-26T03:55:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: osdmap e166: 8 total, 8 up, 6 in 2024-03-26T03:55:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T03:55:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T03:55:14.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:14.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:14.924 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:14.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:14.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:15.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:15.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: pgmap v265: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: 5.14 scrub starts 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: 5.14 scrub ok 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: osdmap e166: 8 total, 8 up, 6 in 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T03:55:15.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: pgmap v265: 148 pgs: 148 active+clean; 2.7 GiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 853 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: 5.14 scrub starts 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: 5.14 scrub ok 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: osdmap e166: 8 total, 8 up, 6 in 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T03:55:15.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:14 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T03:55:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:15.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:15.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:15.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:15.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:15.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:15.357 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:15.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:15.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:15.588 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 25 15 54426 54411 135.995 106.688 0.00606555 0.00734857 2024-03-26T03:55:15.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:15.631 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:15.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:15.776 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:15.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:15.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:15.914 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:15.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:15.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:16.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:16.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:16.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T03:55:16.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T03:55:16.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:16.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T03:55:16.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T03:55:16.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T03:55:16.070 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:15 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T03:55:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T03:55:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T03:55:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T03:55:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T03:55:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T03:55:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T03:55:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T03:55:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:15 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T03:55:16.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:16.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:16.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:16.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:55:16.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:16.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:16.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:16.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:16.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:16.589 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 26 16 55997 55981 134.531 98.125 0.0134543 0.00742796 2024-03-26T03:55:16.616 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:16.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:16.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:16.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:16.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:16.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:16.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:16.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:17.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:17.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:16 smithi086 ceph-mon[32965]: pgmap v267: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 847 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:55:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:16 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T03:55:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:16 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T03:55:17.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[37501]: pgmap v267: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 847 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[29446]: pgmap v267: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 847 B/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T03:55:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:16 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T03:55:17.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:17.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:17.342 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:17.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:17.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:17.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 27 16 57605 57589 133.269 100.5 0.00604451 0.00749819 2024-03-26T03:55:17.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:17.645 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:17.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:17.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T03:55:17.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T03:55:17.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T03:55:17.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T03:55:17.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:17.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:17.828 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:17.832 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:17.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:17.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:17.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:17.977 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T03:55:17.977 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T03:55:17.977 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T03:55:17.977 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:17 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T03:55:18.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:18.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:17 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T03:55:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:17 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T03:55:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:17 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T03:55:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:17 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T03:55:18.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:18.206 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:18.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:55:18.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:55:18.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:18.476 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:18.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:18.590 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 28 16 58901 58885 131.403 81 0.00762747 0.00760269 2024-03-26T03:55:18.608 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:18.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:18.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:18.714 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:55:18.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:18.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:18.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:55:18.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:55:18.782 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-26T03:55:18.782 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T03:55:18.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5 2024-03-26T03:55:18.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:18.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:18.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:18.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:19.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: pgmap v268: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 4.1b scrub starts 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: 4.1b scrub ok 2024-03-26T03:55:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:18 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:19.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:18 smithi086 systemd[1]: Stopping Ceph osd.5 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:55:19.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: pgmap v268: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 4.1b scrub starts 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: 4.1b scrub ok 2024-03-26T03:55:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: pgmap v268: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 103 MiB/s wr, 1.65k op/s 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 4.1b scrub starts 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: 4.1b scrub ok 2024-03-26T03:55:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:18 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.164 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:19.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:19.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:19.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:19.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:19.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:19.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:19.548 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:19.589 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[43916]: 2024-03-26T03:55:19.229+0000 7f5d0da9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T03:55:19.589 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[43916]: 2024-03-26T03:55:19.229+0000 7f5d0da9d700 -1 osd.5 167 *** Got signal Terminated *** 2024-03-26T03:55:19.589 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[43916]: 2024-03-26T03:55:19.229+0000 7f5d0da9d700 -1 osd.5 167 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T03:55:19.590 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 29 16 59980 59964 129.197 67.4375 0.00269714 0.00765519 2024-03-26T03:55:19.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:19.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:19.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:19.798 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: osdmap e167: 8 total, 8 up, 6 in 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: osd.5 marked itself down and dead 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T03:55:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: osdmap e168: 8 total, 7 up, 6 in 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: osdmap e167: 8 total, 8 up, 6 in 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: osd.5 marked itself down and dead 2024-03-26T03:55:19.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T03:55:19.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T03:55:19.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:55:19.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: osdmap e168: 8 total, 7 up, 6 in 2024-03-26T03:55:19.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:55:19.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:19 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:55:19.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:19.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:20.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:20.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:20.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:20.069 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:19 smithi086 podman[86720]: 2024-03-26 03:55:19.824283524 +0000 UTC m=+0.733820521 container died 9af1fb269ca44d33b94ac135383f7ac28351d3597618a981b9ece6b129e2b18f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2906251947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: osdmap e167: 8 total, 8 up, 6 in 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: osd.5 marked itself down and dead 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: osdmap e168: 8 total, 7 up, 6 in 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:55:20.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:19 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:55:20.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:20.175 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:20.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:20.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:20.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:20.337 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:20 smithi086 podman[86720]: 2024-03-26 03:55:20.244712221 +0000 UTC m=+1.154249218 container cleanup 9af1fb269ca44d33b94ac135383f7ac28351d3597618a981b9ece6b129e2b18f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T03:55:20.337 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:20 smithi086 bash[86720]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5 2024-03-26T03:55:20.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:20.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:20.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:20.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:20.590 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 30 16 60324 60308 125.608 21.5 0.0109764 0.00795652 2024-03-26T03:55:20.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:20.708 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:20.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:20.837 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:20 smithi086 podman[86755]: 2024-03-26 03:55:20.471488873 +0000 UTC m=+0.642572630 container remove 9af1fb269ca44d33b94ac135383f7ac28351d3597618a981b9ece6b129e2b18f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True) 2024-03-26T03:55:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: pgmap v271: 148 pgs: 26 stale+active+clean, 122 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T03:55:20.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:20 smithi086 ceph-mon[32965]: osdmap e169: 8 total, 7 up, 6 in 2024-03-26T03:55:20.844 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:20.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:20.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:20.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:20.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:21.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:21.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:21.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: pgmap v271: 148 pgs: 26 stale+active+clean, 122 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[37501]: osdmap e169: 8 total, 7 up, 6 in 2024-03-26T03:55:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: pgmap v271: 148 pgs: 26 stale+active+clean, 122 active+clean; 3.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.7 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T03:55:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:20 smithi006 ceph-mon[29446]: osdmap e169: 8 total, 7 up, 6 in 2024-03-26T03:55:21.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:21.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:21.267 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:20.90815612 +0000 UTC m=+0.019850856 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:55:21.267 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:21.014315065 +0000 UTC m=+0.126009806 container create d62627d704bcb37282cda3b634cb4b706478ba14ab6f882b9518634885a44113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-26T03:55:21.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:21.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:21.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:21.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:21.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:21.587 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:21.26759342 +0000 UTC m=+0.379288162 container init d62627d704bcb37282cda3b634cb4b706478ba14ab6f882b9518634885a44113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T03:55:21.588 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:21.273385378 +0000 UTC m=+0.385080132 container start d62627d704bcb37282cda3b634cb4b706478ba14ab6f882b9518634885a44113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T03:55:21.588 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:21.384533226 +0000 UTC m=+0.496227969 container attach d62627d704bcb37282cda3b634cb4b706478ba14ab6f882b9518634885a44113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T03:55:21.590 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 31 16 61631 61615 124.191 81.6875 0.00656994 0.00804781 2024-03-26T03:55:21.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:21.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:21.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:21.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:21.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:21.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:21.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:21.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:21.906 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T03:55:21.906 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T03:55:21.906 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T03:55:21.906 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T03:55:21.906 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T03:55:21.907 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:21 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T03:55:21.907 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:21 smithi086 podman[86886]: 2024-03-26 03:55:21.65357299 +0000 UTC m=+0.765267737 container died d62627d704bcb37282cda3b634cb4b706478ba14ab6f882b9518634885a44113 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5) 2024-03-26T03:55:22.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:22.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:22.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:22.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:22.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T03:55:22.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T03:55:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:21 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T03:55:22.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:22.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T03:55:22.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:22.301 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T03:55:22.302 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T03:55:22.302 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-03-26T03:55:22.302 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi086.front.sepia.ceph.com 2024-03-26T03:55:22.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.5 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-03-26T03:55:22.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:22.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:22.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:22.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:22.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:22.552 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.5/config 2024-03-26T03:55:22.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:22.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 32 16 62839 62823 122.666 75.5 0.0130114 0.00814687 2024-03-26T03:55:22.593 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:22.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:22.735 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:22.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:22.872 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:22.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:22.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:22.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:22 smithi086 ceph-mon[32965]: pgmap v273: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:22.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:22 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T03:55:22.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:22 smithi086 ceph-mon[32965]: 5.1 scrub starts 2024-03-26T03:55:22.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:22 smithi086 ceph-mon[32965]: 5.1 scrub ok 2024-03-26T03:55:22.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:23.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:23.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:23.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[37501]: pgmap v273: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:23.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T03:55:23.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[37501]: 5.1 scrub starts 2024-03-26T03:55:23.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[37501]: 5.1 scrub ok 2024-03-26T03:55:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[29446]: pgmap v273: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 6 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-26T03:55:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[29446]: 5.1 scrub starts 2024-03-26T03:55:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:22 smithi006 ceph-mon[29446]: 5.1 scrub ok 2024-03-26T03:55:23.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:23.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:23.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:23.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:23.425 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:23.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:23.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:23.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:23.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 33 16 64291 64275 121.699 90.75 0.00457692 0.00821174 2024-03-26T03:55:23.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:23.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:23.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:23.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:23.897 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:23 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:55:23.897 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:23 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:55:23.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:23.949 INFO:teuthology.orchestra.run.smithi086.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:55:23.963 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:23.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:23.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:24.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:24.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:23 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:55:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:23 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:55:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:23 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:55:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:23 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:55:24.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:24.231 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:24.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:24.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:24.363 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:24.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:24.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:24.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:24.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 34 16 65535 65519 120.407 77.75 0.00568747 0.0082996 2024-03-26T03:55:24.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:24.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:24.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:24.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:24.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:24 smithi086 ceph-mon[32965]: pgmap v274: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:24 smithi086 ceph-mon[32965]: osdmap e170: 8 total, 7 up, 6 in 2024-03-26T03:55:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:24.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:24.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:24.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:24.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.034 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:25.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:25.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:25.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[37501]: pgmap v274: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[37501]: osdmap e170: 8 total, 7 up, 6 in 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[29446]: pgmap v274: 148 pgs: 43 peering, 12 stale+active+clean, 93 active+clean; 3.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[29446]: osdmap e170: 8 total, 7 up, 6 in 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:25.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:25.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:25.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:25.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.311 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:25.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:25.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:25.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:25.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:25.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:25.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:25.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 35 16 66667 66651 118.988 70.75 0.00361776 0.00838619 2024-03-26T03:55:25.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:25.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:25.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:8.a 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:8.9 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:8.4 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:8.0 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:6.d 2024-03-26T03:55:25.737 INFO:teuthology.orchestra.run.smithi086.stdout:6.6 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.5 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.c 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.4 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.1e 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:8.3 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.1c 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.1a 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.7 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.11 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:8.8 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.17 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:6.10 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:5.f 2024-03-26T03:55:25.738 INFO:teuthology.orchestra.run.smithi086.stdout:5.e 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:6.1d 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.d 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.b 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.7 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:6.19 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.4 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:8.c 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.3 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:6.a 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:5.2 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:3.a 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:2.2 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:3.d 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:6.1f 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:3.4 2024-03-26T03:55:25.739 INFO:teuthology.orchestra.run.smithi086.stdout:3.1f 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:4.13 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.5 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:8.2 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.10 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:4.1d 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:4.10 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.1d 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:8.d 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:5.1b 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.16 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.7 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.1c 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.c 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:4.8 2024-03-26T03:55:25.740 INFO:teuthology.orchestra.run.smithi086.stdout:3.2 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.1f 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.11 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:6.0 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:5.16 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:5.13 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:5.19 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.15 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.e 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:3.1 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.19 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:5.6 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.2 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.3 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.7 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:6.15 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:3.17 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.a 2024-03-26T03:55:25.741 INFO:teuthology.orchestra.run.smithi086.stdout:4.1 2024-03-26T03:55:25.742 INFO:teuthology.orchestra.run.smithi086.stdout:4.b 2024-03-26T03:55:25.742 INFO:teuthology.orchestra.run.smithi086.stdout:5.12 2024-03-26T03:55:25.742 INFO:teuthology.orchestra.run.smithi086.stdout:5.15 2024-03-26T03:55:25.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:25.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:25.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:25.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:25.976 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:25.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:25.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:26.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:25 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T03:55:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:25 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T03:55:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:25 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T03:55:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:25 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T03:55:26.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:26.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T03:55:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T03:55:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T03:55:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:25 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T03:55:26.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:26.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.254 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:26.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:26.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:26.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:26.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:26.523 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:26.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:26.591 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 36 16 68224 68208 118.386 97.3125 0.0165729 0.008442 2024-03-26T03:55:26.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:26.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:26.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:26.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:26.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: pgmap v276: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 848 B/s rd, 93 MiB/s wr, 1.49k op/s; 36846/196860 objects degraded (18.717%); 11 KiB/s, 0 objects/s recovering 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: Health check failed: Degraded data redundancy: 36846/196860 objects degraded (18.717%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T03:55:26.863 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:26 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T03:55:26.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T03:55:26.943 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T03:55:26.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: pgmap v276: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 848 B/s rd, 93 MiB/s wr, 1.49k op/s; 36846/196860 objects degraded (18.717%); 11 KiB/s, 0 objects/s recovering 2024-03-26T03:55:26.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: Health check failed: Degraded data redundancy: 36846/196860 objects degraded (18.717%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:26.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-03-26T03:55:26.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T03:55:26.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T03:55:26.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: pgmap v276: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 848 B/s rd, 93 MiB/s wr, 1.49k op/s; 36846/196860 objects degraded (18.717%); 11 KiB/s, 0 objects/s recovering 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: Health check failed: Degraded data redundancy: 36846/196860 objects degraded (18.717%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 8 pgs peering) 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T03:55:26.946 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:26 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T03:55:26.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.5 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 5.f --file /var/log/ceph/exp.5.f.5 2024-03-26T03:55:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:26.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:26.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:27.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:27.190 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.5/config 2024-03-26T03:55:27.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:27.250 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:27.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:27.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:27.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:27.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:27.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 37 16 69744 69728 117.754 95 0.0028869 0.00848641 2024-03-26T03:55:27.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:27.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:27.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:27.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:27.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:27.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:27.915 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:55:27.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:27.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:27.968 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:55:28.012 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:55:28.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:28.052 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:55:28.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:28.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:28.096 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:55:28.145 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:55:28.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:28.178 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:55:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:55:28.179 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:55:28.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:28.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:28.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:28.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:28.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:28.460 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:55:28.460 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44446 7.2 7:4ab31f8a:::benchmark_data_smithi006_77260_object44445:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.814077+0000", 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.64137191500001, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14897036499999999, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44446 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.461 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814077+0000", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814077+0000", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814077+0000", 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3e-07 2024-03-26T03:55:28.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814095+0000", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8104000000000001e-05 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814105+0000", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6050000000000004e-06 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.814112+0000", 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0160000000000003e-06 2024-03-26T03:55:28.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.819664+0000", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0055522059999999996 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.819807+0000", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014261800000000001 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.820141+0000", 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033421900000000002 2024-03-26T03:55:28.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822834+0000", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026925349999999998 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823185+0000", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035107999999999998 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962946+0000", 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13976089999999999 2024-03-26T03:55:28.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963005+0000", 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9533000000000001e-05 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963047+0000", 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1819000000000001e-05 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.466 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44449 7.c 7:30a88b6e:::benchmark_data_smithi006_77260_object44448:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.815292+0000", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.64015647400001, 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14722302400000001, 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44449 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815292+0000", 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815292+0000", 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815293+0000", 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815343+0000", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0241999999999998e-05 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815354+0000", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0225e-05 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.815361+0000", 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1049999999999997e-06 2024-03-26T03:55:28.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823142+0000", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0077817210000000001 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823272+0000", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000129352 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823472+0000", 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020031899999999999 2024-03-26T03:55:28.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825256+0000", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017840880000000001 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825859+0000", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060256399999999999 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962402+0000", 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13654318700000001 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962460+0000", 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7640999999999999e-05 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962515+0000", 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.554e-05 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44454 7.2 7:44c02848:::benchmark_data_smithi006_77260_object44453:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.472 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.822881+0000", 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.63256778499999, 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14549235899999999, 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44454 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822881+0000", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822881+0000", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822882+0000", 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4500000000000001e-06 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822910+0000", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7636000000000001e-05 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822920+0000", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0098999999999999e-05 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822928+0000", 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7729999999999999e-06 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823216+0000", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00028831100000000003 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823336+0000", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119676 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.823478+0000", 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014271100000000001 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963094+0000", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13961531199999999 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.967491+0000", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043970090000000003 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968219+0000", 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00072853100000000004 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968287+0000", 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7392999999999997e-05 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968373+0000", 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6458000000000005e-05 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44456 7.2 7:4d7c7941:::benchmark_data_smithi006_77260_object44455:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.478 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.822947+0000", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.632501026, 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16495163400000001, 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44456 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822947+0000", 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822947+0000", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822948+0000", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9100000000000004e-07 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822972+0000", 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4062000000000001e-05 2024-03-26T03:55:28.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822980+0000", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0139999999999998e-06 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.822987+0000", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6139999999999997e-06 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.827170+0000", 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041836240000000004 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.827292+0000", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121872 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.827427+0000", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013446700000000001 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.967527+0000", 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.140100059 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968381+0000", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00085424700000000004 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.987017+0000", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.018636369999999999 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.987702+0000", 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068487599999999995 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.987899+0000", 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019683799999999999 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44461 7.3 7:c4901de4:::benchmark_data_smithi006_77260_object44460:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.824636+0000", 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.63081233599999, 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14398197800000001, 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.484 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44461 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824636+0000", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824636+0000", 2024-03-26T03:55:28.486 INFO:teuthology.orchestra.run.smithi086.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:55:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824637+0000", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9000000000000005e-07 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824660+0000", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2864999999999999e-05 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824669+0000", 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4499999999999993e-06 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.824677+0000", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6399999999999997e-06 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.826105+0000", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014277140000000001 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.826223+0000", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011838200000000001 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.826351+0000", 2024-03-26T03:55:28.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012783799999999999 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962483+0000", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13613229299999999 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962641+0000", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000157613 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968528+0000", 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058872539999999998 2024-03-26T03:55:28.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968578+0000", 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9885000000000001e-05 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968618+0000", 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0054e-05 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.490 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44463 7.2 7:4c6da1e5:::benchmark_data_smithi006_77260_object44462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.825756+0000", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.62969264, 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.162416113, 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44463 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825756+0000", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825756+0000", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825757+0000", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.184e-06 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825780+0000", 2024-03-26T03:55:28.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3025999999999999e-05 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825790+0000", 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6590000000000002e-06 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.825797+0000", 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4039999999999998e-06 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963131+0000", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.137333966 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963252+0000", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121192 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.963384+0000", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013149 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968195+0000", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048111350000000002 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968386+0000", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019075299999999999 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.987940+0000", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019554637 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.988132+0000", 2024-03-26T03:55:28.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019138400000000001 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.988172+0000", 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0283e-05 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44466 7.2 7:4c98fc5b:::benchmark_data_smithi006_77260_object44465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:20.829126+0000", 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.62632266599999, 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.159650985, 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.496 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44466 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.829126+0000", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.829126+0000", 2024-03-26T03:55:28.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.829127+0000", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.342e-06 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.829163+0000", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5898000000000001e-05 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962549+0000", 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.133385543 2024-03-26T03:55:28.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.962559+0000", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0555e-05 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.964891+0000", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023318919999999999 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.965005+0000", 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113961 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.965132+0000", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127026 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.968390+0000", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032580180000000001 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.970876+0000", 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024855760000000002 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.988208+0000", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017332499000000001 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.988738+0000", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052964799999999997 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:20.988777+0000", 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9026999999999998e-05 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.501 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44820 7.c 7:39392e54:::benchmark_data_smithi006_77260_object44819:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.196399+0000", 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.25904908999999, 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15813656100000001, 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44820 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196399+0000", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196399+0000", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196400+0000", 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2399999999999998e-07 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196421+0000", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1166999999999999e-05 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196429+0000", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9389999999999997e-06 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196435+0000", 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2650000000000002e-06 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.200056+0000", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036206530000000002 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.200170+0000", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113479 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.200309+0000", 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013973100000000001 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.202356+0000", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0020471579999999999 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354213+0000", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15185628600000001 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354424+0000", 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021100700000000001 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354484+0000", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0080000000000001e-05 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354536+0000", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2172000000000001e-05 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44821 7.c 7:31b393e0:::benchmark_data_smithi006_77260_object44820:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.196478+0000", 2024-03-26T03:55:28.507 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.25897011999999, 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15823753700000001, 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44821 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196478+0000", 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196478+0000", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196479+0000", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0100000000000004e-07 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196502+0000", 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3268000000000002e-05 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196511+0000", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9760000000000001e-06 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.196518+0000", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5239999999999997e-06 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.202404+0000", 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00588596 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.511 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.202531+0000", 2024-03-26T03:55:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:28.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012752400000000001 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.202667+0000", 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013571300000000001 2024-03-26T03:55:28.512 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.353890+0000", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15122291800000001 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354220+0000", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032994699999999998 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354577+0000", 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000356697 2024-03-26T03:55:28.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354682+0000", 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010568 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354716+0000", 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3628999999999999e-05 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44825 7.3 7:cefda806:::benchmark_data_smithi006_77260_object44824:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.199055+0000", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.25639396400001, 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.156382779, 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44825 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199055+0000", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199055+0000", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199056+0000", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2920000000000001e-06 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199113+0000", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7210999999999998e-05 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199124+0000", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0745e-05 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199131+0000", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4950000000000002e-06 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199281+0000", 2024-03-26T03:55:28.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014923399999999999 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199393+0000", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000112391 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.199528+0000", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013512100000000001 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.201763+0000", 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022352629999999999 2024-03-26T03:55:28.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.202432+0000", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00066873499999999995 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355344+0000", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15291234000000001 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355395+0000", 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0537000000000001e-05 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355437+0000", 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2415e-05 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44830 7.2 7:437ef679:::benchmark_data_smithi006_77260_object44829:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.203744+0000", 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.25170489499999, 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.150929021, 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.520 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44830 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203744+0000", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203744+0000", 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203745+0000", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.248e-06 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203766+0000", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1191000000000001e-05 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203777+0000", 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0852e-05 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203785+0000", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0870000000000007e-06 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203834+0000", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8928999999999999e-05 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.203911+0000", 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7197e-05 2024-03-26T03:55:28.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204014+0000", 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010284400000000001 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352698+0000", 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14868378300000001 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.353912+0000", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012139869999999999 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354589+0000", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00067770000000000005 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354636+0000", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6099e-05 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354673+0000", 2024-03-26T03:55:28.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7104000000000003e-05 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:44821 7.c e108/96 ondisk, result = 0)", 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.204318+0000", 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.251130276, 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14957879299999999, 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204318+0000", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204318+0000", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204319+0000", 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.44e-07 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204325+0000", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5050000000000004e-06 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352366+0000", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14804115600000001 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352385+0000", 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8300000000000001e-05 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.353876+0000", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014915060000000001 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.353884+0000", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6790000000000005e-06 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.353897+0000", 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3103e-05 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:44831 7.7 e108/96 7:e27452a6:::benchmark_data_smithi006_77260_object44830:head v 108'2799, mlcod=108'2797)", 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.208109+0000", 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 247.247339186, 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.146654914, 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208109+0000", 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208109+0000", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208111+0000", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.203e-06 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208141+0000", 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0264e-05 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352640+0000", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.144499448 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352651+0000", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0702e-05 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352703+0000", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1728999999999999e-05 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352722+0000", 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9505e-05 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354730+0000", 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0020073550000000002 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354764+0000", 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4708e-05 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.533 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46478 7.2 7:46050f9c:::benchmark_data_smithi006_77260_object46477:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.783054+0000", 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.67239481600001, 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15183814500000001, 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.534 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46478 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783054+0000", 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783054+0000", 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783055+0000", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.372e-06 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783079+0000", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.357e-05 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783089+0000", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0178e-05 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.783097+0000", 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0120000000000005e-06 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.786515+0000", 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034177270000000002 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.786644+0000", 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012989699999999999 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.786782+0000", 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013731799999999999 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793685+0000", 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006902811 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793698+0000", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3879e-05 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934795+0000", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14109666800000001 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934849+0000", 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3955000000000001e-05 2024-03-26T03:55:28.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934892+0000", 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2758e-05 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46480 7.a 7:593016fe:::benchmark_data_smithi006_77260_object46479:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.785054+0000", 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.67039415299999, 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15024206100000001, 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.540 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46480 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785054+0000", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785054+0000", 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785056+0000", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1370000000000001e-06 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785074+0000", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8711000000000001e-05 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785082+0000", 2024-03-26T03:55:28.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8620000000000003e-06 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785087+0000", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1460000000000004e-06 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.788539+0000", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034516859999999998 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.788660+0000", 2024-03-26T03:55:28.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012125099999999999 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.788780+0000", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000119386 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794186+0000", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.005406297 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934931+0000", 2024-03-26T03:55:28.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.140745589 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935094+0000", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016210399999999999 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935185+0000", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1275999999999998e-05 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935296+0000", 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111616 2024-03-26T03:55:28.545 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46485 7.a 7:54c0d641:::benchmark_data_smithi006_77260_object46484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.785806+0000", 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.66964289500001, 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15469738399999999, 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.546 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46485 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785806+0000", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785806+0000", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785806+0000", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7399999999999998e-07 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785823+0000", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7268000000000001e-05 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785830+0000", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5180000000000001e-06 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785835+0000", 2024-03-26T03:55:28.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4600000000000002e-06 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792030+0000", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061945050000000003 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792236+0000", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020585699999999999 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792383+0000", 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000147145 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934968+0000", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14258508 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935009+0000", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1409999999999998e-05 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940398+0000", 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0053884900000000001 2024-03-26T03:55:28.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940452+0000", 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4057000000000001e-05 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940503+0000", 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1119999999999998e-05 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46489 7.a 7:56a16df7:::benchmark_data_smithi006_77260_object46488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:28.551 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.790513+0000", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.66493532999999, 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15014635800000001, 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46489 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790513+0000", 2024-03-26T03:55:28.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790513+0000", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790515+0000", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5999999999999999e-06 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790545+0000", 2024-03-26T03:55:28.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9969999999999999e-05 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790556+0000", 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1107999999999999e-05 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790564+0000", 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5150000000000006e-06 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794223+0000", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00365862 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794351+0000", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012818500000000001 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794475+0000", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012363700000000001 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935062+0000", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.140587662 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940554+0000", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054916119999999999 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940567+0000", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.255e-05 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940616+0000", 2024-03-26T03:55:28.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9704999999999999e-05 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940660+0000", 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3194e-05 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46491 7.2 7:40c3a5f0:::benchmark_data_smithi006_77260_object46490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.790676+0000", 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.66477288799999, 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.153298926, 2024-03-26T03:55:28.557 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46491 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790676+0000", 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790676+0000", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790676+0000", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.47e-07 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790696+0000", 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9891e-05 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790842+0000", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000145718 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790849+0000", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3180000000000001e-06 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935092+0000", 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14424273600000001 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935220+0000", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127988 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935358+0000", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013821600000000001 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940676+0000", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0053174269999999996 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.940717+0000", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1415999999999998e-05 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943876+0000", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031583399999999999 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943931+0000", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5059000000000003e-05 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943975+0000", 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3970000000000001e-05 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:46478 7.2 e109/95 ondisk, result = 0)", 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.791221+0000", 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.664227995, 2024-03-26T03:55:28.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.144080083, 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791221+0000", 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791221+0000", 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791221+0000", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1799999999999995e-07 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791225+0000", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0300000000000004e-06 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791234+0000", 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1200000000000008e-06 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791241+0000", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8950000000000001e-06 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793563+0000", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023219069999999998 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793675+0000", 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000112413 2024-03-26T03:55:28.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935301+0000", 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14162520000000001 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:46493 7.f e109/95 7:f7346e28:::benchmark_data_smithi006_77260_object46492:head v 109'2898, mlcod=109'2896)", 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.794502+0000", 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 245.66094667900001, 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15165614699999999, 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:55:28.567 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794502+0000", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794502+0000", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794502+0000", 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.39e-07 2024-03-26T03:55:28.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794521+0000", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8981999999999999e-05 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933055+0000", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13853392 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933065+0000", 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0216e-05 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933292+0000", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022661599999999999 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933314+0000", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2116e-05 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946116+0000", 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012801985 2024-03-26T03:55:28.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946158+0000", 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1972999999999999e-05 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:55:28.571 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:55:28.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 38 16 70911 70895 116.574 72.9375 0.0130089 0.00857328 2024-03-26T03:55:28.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:28.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:28.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:28.784 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:28.886 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:28.925 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:28.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:28.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:29.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:29.029 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:28 smithi086 ceph-mon[32965]: pgmap v277: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 36846/196860 objects degraded (18.717%); 8.0 KiB/s, 0 objects/s recovering 2024-03-26T03:55:29.029 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1890150865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.071 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:29.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:28 smithi006 ceph-mon[37501]: pgmap v277: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 36846/196860 objects degraded (18.717%); 8.0 KiB/s, 0 objects/s recovering 2024-03-26T03:55:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1890150865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:28 smithi006 ceph-mon[29446]: pgmap v277: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.13k op/s; 36846/196860 objects degraded (18.717%); 8.0 KiB/s, 0 objects/s recovering 2024-03-26T03:55:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1890150865' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:29.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:Exporting 5.f info 5.f( empty local-lis/les=96/97 n=0 ec=69/69 lis/c=96/96 les/c/f=97/97/0 sis=96) 2024-03-26T03:55:29.184 INFO:teuthology.orchestra.run.smithi086.stderr:Export successful 2024-03-26T03:55:29.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:29.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:55:29.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.364 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:29.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:29.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:29.592 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 39 16 72112 72096 115.51 75.0625 0.0200787 0.00865141 2024-03-26T03:55:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:55:29.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:55:29.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:29.650 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:29.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:29.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:29.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:29 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T03:55:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:29 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T03:55:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:29 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T03:55:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:29 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T03:55:29.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:29.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:30.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:30.039 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:29 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T03:55:30.039 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:29 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T03:55:30.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:30.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:30.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:30.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:30.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.5 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 5.f 2024-03-26T03:55:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.348 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:30.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:30.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:30.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:55:30.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.492 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:30.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:30.579 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.5/config 2024-03-26T03:55:30.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:30.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:55:30.594498+0000 min lat: 0.00117332 max lat: 1.05068 avg lat: 0.00869413 2024-03-26T03:55:30.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:55:30.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 40 16 73606 73590 114.951 93.375 0.00884443 0.00869413 2024-03-26T03:55:30.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:30.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:30.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:30.778 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:30.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:30.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:30.922 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:30.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:30.975 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:30 smithi086 ceph-mon[32965]: 6.13 deep-scrub starts 2024-03-26T03:55:30.975 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:30 smithi086 ceph-mon[32965]: pgmap v278: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1019 B/s rd, 55 MiB/s wr, 877 op/s; 38473/204276 objects degraded (18.834%); 7.1 KiB/s, 0 objects/s recovering 2024-03-26T03:55:30.975 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:30 smithi086 ceph-mon[32965]: 6.13 deep-scrub ok 2024-03-26T03:55:30.976 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:30 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T03:55:30.976 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:30 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T03:55:31.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:31.037 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[37501]: 6.13 deep-scrub starts 2024-03-26T03:55:31.037 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[37501]: pgmap v278: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1019 B/s rd, 55 MiB/s wr, 877 op/s; 38473/204276 objects degraded (18.834%); 7.1 KiB/s, 0 objects/s recovering 2024-03-26T03:55:31.037 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[37501]: 6.13 deep-scrub ok 2024-03-26T03:55:31.037 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T03:55:31.037 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T03:55:31.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[29446]: 6.13 deep-scrub starts 2024-03-26T03:55:31.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[29446]: pgmap v278: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1019 B/s rd, 55 MiB/s wr, 877 op/s; 38473/204276 objects degraded (18.834%); 7.1 KiB/s, 0 objects/s recovering 2024-03-26T03:55:31.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[29446]: 6.13 deep-scrub ok 2024-03-26T03:55:31.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T03:55:31.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:30 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T03:55:31.068 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:31.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:31.207 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:31.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:31.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:31.494 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:31.594 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 41 16 75147 75131 114.496 96.3125 0.00906254 0.00872862 2024-03-26T03:55:31.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:31.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:55:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:31.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:31.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:31.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:31.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:31.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:31.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:31.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:55:31.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:55:31.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T03:55:31.928 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T03:55:31.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:55:31.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:55:31.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T03:55:31.934 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:31 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T03:55:32.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:32.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:32.065 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:31 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:55:32.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:31 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:55:32.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:31 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T03:55:32.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:31 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T03:55:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:32.140 INFO:teuthology.orchestra.run.smithi086.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:55:32.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:32.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:32.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:32.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:32.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:32.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:32.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:32.533 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:32.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:32.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 42 16 76457 76441 113.717 81.875 0.0172699 0.00878837 2024-03-26T03:55:32.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:32.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:32.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:32.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:32.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:32.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:32.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: pgmap v279: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 40496/216639 objects degraded (18.693%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: 4.18 scrub starts 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[29446]: 4.18 scrub ok 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: pgmap v279: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 40496/216639 objects degraded (18.693%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: 4.18 scrub starts 2024-03-26T03:55:32.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:32 smithi006 ceph-mon[37501]: 4.18 scrub ok 2024-03-26T03:55:32.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:33.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:33.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: pgmap v279: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 40496/216639 objects degraded (18.693%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: 4.18 scrub starts 2024-03-26T03:55:33.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:32 smithi086 ceph-mon[32965]: 4.18 scrub ok 2024-03-26T03:55:33.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:33.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:33.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:33.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.428 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:33.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:33.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:33.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:33.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:33.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 43 16 77821 77805 113.055 85.25 0.00558901 0.00884004 2024-03-26T03:55:33.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:33.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:33.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:33.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1186746981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:33.818 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:33 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:33.844 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:55:33.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:33.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:33.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:33.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:34.062 INFO:teuthology.orchestra.run.smithi086.stdout: marking collection for removal 2024-03-26T03:55:34.062 INFO:teuthology.orchestra.run.smithi086.stdout:setting '_remove' omap key 2024-03-26T03:55:34.062 INFO:teuthology.orchestra.run.smithi086.stdout:finish_remove_pgs 5.f_head removing 5.f 2024-03-26T03:55:34.065 INFO:teuthology.orchestra.run.smithi086.stdout:Remove successful 2024-03-26T03:55:34.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:34.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1186746981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:34.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:33 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:34.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1186746981' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:34.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:33 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:55:34.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:34.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:34.240 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:34.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:34.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:34.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:34.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:34.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:34.390 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:78897 8.b 8:d6aa26c0:::benchmark_data_smithi006_103169_object78896:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.382879+0000", 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.007190068, 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00721486, 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 78897 2024-03-26T03:55:34.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382879+0000", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382879+0000", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382880+0000", 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1400000000000001e-06 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382954+0000", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3475999999999999e-05 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382968+0000", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4280999999999999e-05 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.382990+0000", 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1552e-05 2024-03-26T03:55:34.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.383049+0000", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9525999999999999e-05 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.383189+0000", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013981200000000001 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.383357+0000", 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000167768 2024-03-26T03:55:34.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.385250+0000", 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0018933960000000001 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388810+0000", 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035594680000000001 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:78898 8.5 8:a345594b:::benchmark_data_smithi006_103169_object78897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:55:34.395 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.384594+0000", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0054753340000000001, 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0055587060000000001, 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 78898 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384594+0000", 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384594+0000", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384595+0000", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-07 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384643+0000", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8362000000000002e-05 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:34.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384653+0000", 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0399e-05 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384659+0000", 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6690000000000003e-06 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384700+0000", 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1318e-05 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384794+0000", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3545e-05 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.384935+0000", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014135800000000001 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386823+0000", 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00188788 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:78900 8.b 8:d7e9a526:::benchmark_data_smithi006_103169_object78899:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.386542+0000", 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0035274490000000002, 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036591039999999998, 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 78900 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386542+0000", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386542+0000", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386543+0000", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3900000000000003e-07 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:34.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386591+0000", 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8708000000000002e-05 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386602+0000", 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0479000000000001e-05 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386609+0000", 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2180000000000002e-06 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386685+0000", 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6050000000000005e-05 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386773+0000", 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7947000000000003e-05 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:34.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.386899+0000", 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012605099999999999 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388943+0000", 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020440749999999998 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:78902 8.5 8:abc1f772:::benchmark_data_smithi006_103169_object78901:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.387328+0000", 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0027417280000000001, 2024-03-26T03:55:34.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029617720000000001, 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 78902 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387328+0000", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:34.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387328+0000", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387329+0000", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9900000000000009e-07 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387372+0000", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3439000000000002e-05 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:34.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387387+0000", 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5135e-05 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387396+0000", 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.037e-06 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387443+0000", 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6897000000000002e-05 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387563+0000", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011984 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.387712+0000", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000149197 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.389687+0000", 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019752210000000001 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:34.408 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:78901 8.9 e172/168 8:9d7b5c55:::benchmark_data_smithi006_103169_object78900:head v 172'5022, mlcod=172'5018)", 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.388327+0000", 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0017417699999999999, 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002042749, 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388327+0000", 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388327+0000", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388328+0000", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5300000000000004e-07 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388401+0000", 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3257999999999996e-05 2024-03-26T03:55:34.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388416+0000", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4248e-05 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388423+0000", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6939999999999995e-06 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388503+0000", 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9165000000000005e-05 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:34.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.388523+0000", 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0477000000000001e-05 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 5 2024-03-26T03:55:34.412 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:55:34.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:34.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:34.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 44 15 79240 79225 112.503 88.75 0.0178208 0.00888395 2024-03-26T03:55:34.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:34.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:34.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:34.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:34.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:34.780 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:34.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:34.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:34.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[37501]: pgmap v280: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:34.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[37501]: osdmap e171: 8 total, 7 up, 6 in 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[37501]: osdmap e172: 8 total, 7 up, 6 in 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[29446]: pgmap v280: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[29446]: osdmap e171: 8 total, 7 up, 6 in 2024-03-26T03:55:34.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:34 smithi006 ceph-mon[29446]: osdmap e172: 8 total, 7 up, 6 in 2024-03-26T03:55:34.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:34.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:34.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:35.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:35.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-mon[32965]: pgmap v280: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.4 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 921 B/s rd, 75 MiB/s wr, 1.20k op/s; 40496/216639 objects degraded (18.693%); 6.4 KiB/s, 0 objects/s recovering 2024-03-26T03:55:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:55:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-mon[32965]: osdmap e171: 8 total, 7 up, 6 in 2024-03-26T03:55:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:34 smithi086 ceph-mon[32965]: osdmap e172: 8 total, 7 up, 6 in 2024-03-26T03:55:35.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:35.181 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:35.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:35.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:35.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:35.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:35.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:35.464 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:35.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:35.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:35.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 45 16 80733 80717 112.075 93.25 0.0265827 0.00891724 2024-03-26T03:55:35.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:55:35.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:35.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:35.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:35.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:35.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:35.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:35.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:36.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:36.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:36.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:36.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:36.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:36.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.5 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.f.5 2024-03-26T03:55:36.291 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:36.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:36.367 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:36.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:36.432 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:55:36.432 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:36.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:36.473 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:55:36.513 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:55:36.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:36.557 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:55:36.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:36.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 46 16 82154 82138 111.569 88.8125 0.0100135 0.00895841 2024-03-26T03:55:36.595 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:55:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.5/config 2024-03-26T03:55:36.637 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:55:36.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:36.679 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:55:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:55:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:55:36.679 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:55:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:55:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:55:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-26T03:55:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:55:36.680 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:55:36.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:36.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:36.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:36.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:36.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:36.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:36.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:37.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:36 smithi006 ceph-mon[29446]: pgmap v283: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:37.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2500605463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:37.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:36 smithi006 ceph-mon[37501]: pgmap v283: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:37.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2500605463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:37.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:36 smithi086 ceph-mon[32965]: pgmap v283: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.71k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2500605463' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:37.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:55:37.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:37.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:37.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:37.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:37.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:37.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:37.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.384 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:37.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:37.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:55:37.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:55:37.533 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:37.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:37.596 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:37.596 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 47 16 83618 83602 111.142 91.5 0.00589356 0.0089924 2024-03-26T03:55:37.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:37.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:37.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:37.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:37.802 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:37.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:37.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:37.940 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:37.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:37.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:38.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:38.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:38.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:38.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:38.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:38.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:38.476 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:38.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:38.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:38.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 48 16 85096 85080 110.751 92.375 0.00651618 0.0090246 2024-03-26T03:55:38.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:38.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:38.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:38.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:38.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:38.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:38.888 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:38.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:38.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:39.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:39.073 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:38 smithi006 ceph-mon[37501]: pgmap v284: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:39.074 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:38 smithi006 ceph-mon[29446]: pgmap v284: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:38 smithi086 ceph-mon[32965]: pgmap v284: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 4.8 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.40k op/s; 44579/237852 objects degraded (18.742%) 2024-03-26T03:55:39.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:39.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:39.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:39.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:39.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:39.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:39.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:39.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:39.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:39.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:55:39.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:39.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 49 16 86272 86256 109.991 73.5 0.0125295 0.00908726 2024-03-26T03:55:39.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:39.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:39.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:39.803 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:39.803 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:39.804 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:39.804 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 44579/237852 objects degraded (18.742%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 44579/237852 objects degraded (18.742%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:39.846 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:39.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:39.969 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:39.998 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:39.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:40.070 INFO:teuthology.orchestra.run.smithi086.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T03:55:40.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:40.116 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:40.116 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 44579/237852 objects degraded (18.742%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:40.140 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:40.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:40.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:40.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:40.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:40.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:40.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:40.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:40.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:40.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:40.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 50 16 87639 87623 109.499 85.4375 0.0102804 0.00912766 2024-03-26T03:55:40.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:40.679 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:40.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:40.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:40.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:40.832 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:55:40.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:40.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:40.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:41.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:40 smithi086 ceph-mon[32965]: pgmap v285: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 46352/245835 objects degraded (18.855%) 2024-03-26T03:55:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1737204255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:40 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:41.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:41.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[37501]: pgmap v285: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 46352/245835 objects degraded (18.855%) 2024-03-26T03:55:41.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1737204255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[29446]: pgmap v285: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.0 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 46352/245835 objects degraded (18.855%) 2024-03-26T03:55:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1737204255' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:40 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:55:41.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:41.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:41.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:41.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:41.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:41.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:41.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:41.541 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:41.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:41.595 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 51 16 89165 89149 109.222 95.375 0.00443515 0.00915133 2024-03-26T03:55:41.643 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:41.683 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:41.783 INFO:teuthology.orchestra.run.smithi086.stdout:get_pg_num_history pg_num_history pg_num_history(e167 pg_nums {1={19=1},2={64=3},3={65=32},4={67=32},5={69=32},6={71=32},7={95=16},8={155=16}} deleted_pools 154,7) 2024-03-26T03:55:41.784 INFO:teuthology.orchestra.run.smithi086.stdout:Importing pgid 5.f 2024-03-26T03:55:41.784 INFO:teuthology.orchestra.run.smithi086.stdout:write_pg epoch 152 info 5.f( empty local-lis/les=96/97 n=0 ec=69/69 lis/c=96/96 les/c/f=97/97/0 sis=96) 2024-03-26T03:55:41.784 INFO:teuthology.orchestra.run.smithi086.stdout:Import successful 2024-03-26T03:55:41.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:41.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:41.944 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:55:41.944 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:55:41.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:41.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:41.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:42.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:41 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:41 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:41 smithi086 ceph-mon[32965]: osdmap e173: 8 total, 7 up, 6 in 2024-03-26T03:55:42.111 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:42.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[37501]: osdmap e173: 8 total, 7 up, 6 in 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:55:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:41 smithi006 ceph-mon[29446]: osdmap e173: 8 total, 7 up, 6 in 2024-03-26T03:55:42.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:42.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:42.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:42.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:42.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:42.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:42.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:42.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:42.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:42.596 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 52 15 90604 90589 108.853 90 0.0056102 0.00918222 2024-03-26T03:55:42.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:55:42.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:42.696 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:42.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.840 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:42.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:42.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:42 smithi006 ceph-mon[37501]: pgmap v287: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 118 MiB/s wr, 1.89k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:42 smithi006 ceph-mon[29446]: pgmap v287: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 118 MiB/s wr, 1.89k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:42.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:42.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:42.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:43.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:42 smithi086 ceph-mon[32965]: pgmap v287: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 118 MiB/s wr, 1.89k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:43.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:43.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:43.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:43.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:43.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:43.275 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:55:43.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/exp.5.f.5 2024-03-26T03:55:43.319 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:55:43.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:43.351 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:43.370 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:55:43.379 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:43.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:55:43.452 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:55:43.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:43.495 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:55:43.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-26T03:55:43.538 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:55:43.539 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:55:43.596 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 53 16 92044 92028 108.496 89.9375 0.0113408 0.00921244 2024-03-26T03:55:43.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:43.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:43.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:43.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:43.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:43.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:43.934 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:43.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:44.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:44.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1753423521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:44.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1753423521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1753423521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:44.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:44.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:55:44.338 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:55:44.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:44.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:44.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:44.483 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:44.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:44.596 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 54 16 93568 93552 108.25 95.25 0.00507219 0.00923341 2024-03-26T03:55:44.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:44.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:44.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:44.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:55:44.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:44.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:44.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:44.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:44.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:44.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:44.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:44.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:44.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:45.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:45.105 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:44 smithi086 ceph-mon[32965]: pgmap v288: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 566 B/s rd, 80 MiB/s wr, 1.27k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:45.106 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 48554/259089 objects degraded (18.740%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:45.106 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:44 smithi086 ceph-mon[32965]: osdmap e174: 8 total, 7 up, 6 in 2024-03-26T03:55:45.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:45.122 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[29446]: pgmap v288: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 566 B/s rd, 80 MiB/s wr, 1.27k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 48554/259089 objects degraded (18.740%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[29446]: osdmap e174: 8 total, 7 up, 6 in 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[37501]: pgmap v288: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.3 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 566 B/s rd, 80 MiB/s wr, 1.27k op/s; 48554/259089 objects degraded (18.740%) 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 48554/259089 objects degraded (18.740%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:45.144 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:44 smithi006 ceph-mon[37501]: osdmap e174: 8 total, 7 up, 6 in 2024-03-26T03:55:45.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:45.161 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:55:45.161 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T03:55:45.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:44465 7.1 7:83afc1a9:::benchmark_data_smithi006_77260_object44464:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 264.33084295399999, 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14001919600000001, 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 44465 2024-03-26T03:55:45.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827919+0000", 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827921+0000", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3820000000000001e-06 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827973+0000", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2698999999999998e-05 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827983+0000", 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8139999999999995e-06 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.827991+0000", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4429999999999997e-06 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828043+0000", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2494999999999998e-05 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828177+0000", 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000133635 2024-03-26T03:55:45.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.828321+0000", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014464 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.829756+0000", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014350140000000001 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.833618+0000", 2024-03-26T03:55:45.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038613060000000001 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967795+0000", 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13417760400000001 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967892+0000", 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6203e-05 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:20.967939+0000", 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6961e-05 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:45952 7.1 7:88aa6d4d:::benchmark_data_smithi006_77260_object45951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 262.96940990500002, 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13436482599999999, 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.168 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 45952 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189352+0000", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189353+0000", 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0470000000000001e-06 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189407+0000", 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3606999999999997e-05 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189417+0000", 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431999999999999e-05 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189424+0000", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8499999999999996e-06 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189497+0000", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2306999999999997e-05 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189606+0000", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000109842 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.189710+0000", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103675 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.191064+0000", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013539629999999999 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.192039+0000", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097487699999999997 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323621+0000", 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13158181599999999 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323675+0000", 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4469000000000003e-05 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.323717+0000", 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1940999999999997e-05 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46143 7.1 7:81fb4aa4:::benchmark_data_smithi006_77260_object46142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 262.72016389300001, 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10240083799999999, 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.174 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46143 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438598+0000", 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438600+0000", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2619999999999999e-06 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438643+0000", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3565e-05 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438654+0000", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0431999999999999e-05 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438661+0000", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6469999999999998e-06 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438717+0000", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5680000000000002e-05 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438859+0000", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014230699999999999 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439003+0000", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014414000000000001 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.440321+0000", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001317412 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.441425+0000", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001103726 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540905+0000", 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099480856000000006 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540948+0000", 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3044000000000003e-05 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.540999+0000", 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0766999999999997e-05 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.179 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46492 7.1 7:829c04db:::benchmark_data_smithi006_77260_object46491:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 262.36768392599998, 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.14271868800000001, 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46492 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.180 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791078+0000", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791079+0000", 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4300000000000002e-07 2024-03-26T03:55:45.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791120+0000", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0858000000000001e-05 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791131+0000", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0658e-05 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.791136+0000", 2024-03-26T03:55:45.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7690000000000002e-06 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792218+0000", 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010815589999999999 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792297+0000", 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9115000000000004e-05 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.792397+0000", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9881000000000001e-05 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.793328+0000", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00093134399999999997 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.795379+0000", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002050765 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933714+0000", 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13833505600000001 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933759+0000", 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5204999999999998e-05 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.933797+0000", 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7734999999999997e-05 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.185 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:70989 7.1 7:8446efc3:::benchmark_data_smithi006_77260_object70988:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 241.15709570199999, 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080342884000000003, 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.186 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 70989 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001667+0000", 2024-03-26T03:55:45.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001668+0000", 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.398e-06 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001716+0000", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8364e-05 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001729+0000", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2765000000000001e-05 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001737+0000", 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2150000000000001e-06 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001793+0000", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5238000000000002e-05 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.001957+0000", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016436399999999999 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.002145+0000", 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000188569 2024-03-26T03:55:45.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.004409+0000", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022632310000000001 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.005835+0000", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001426278 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.081913+0000", 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076077991999999997 2024-03-26T03:55:45.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.081958+0000", 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5354e-05 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:44.082009+0000", 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1116000000000003e-05 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:74573 7.1 7:845b8743:::benchmark_data_smithi006_77260_object74572:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 237.52545448000001, 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10512387099999999, 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 74573 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633308+0000", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633309+0000", 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.313e-06 2024-03-26T03:55:45.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633367+0000", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7587000000000003e-05 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633379+0000", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2622e-05 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633386+0000", 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3690000000000003e-06 2024-03-26T03:55:45.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633439+0000", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3578000000000001e-05 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633572+0000", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013307 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.633741+0000", 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016872 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.635759+0000", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020182310000000001 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.636437+0000", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00067761600000000003 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738338+0000", 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.101901425 2024-03-26T03:55:45.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738381+0000", 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2623000000000002e-05 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:47.738432+0000", 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0717000000000002e-05 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.198 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133794 7.1 7:8afd9b64:::benchmark_data_smithi006_77260_object21440:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.541031+0000", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 175.61773100600001, 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097634211999999998, 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133794 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541031+0000", 2024-03-26T03:55:45.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541031+0000", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541033+0000", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.336e-06 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541038+0000", 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.242e-06 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541049+0000", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0890000000000001e-05 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541055+0000", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5660000000000003e-06 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541102+0000", 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6817000000000003e-05 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541221+0000", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119059 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.541344+0000", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012281799999999999 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.543222+0000", 2024-03-26T03:55:45.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0018775759999999999 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.543455+0000", 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023385100000000001 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.638570+0000", 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.095114746 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.638612+0000", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1959000000000003e-05 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.638665+0000", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3352e-05 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:156476 7.1 7:8a9d0fdf:::benchmark_data_smithi006_77260_object44122:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:12.404663+0000", 2024-03-26T03:55:45.204 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 152.75409958500001, 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.083358781000000007, 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 156476 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404663+0000", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404663+0000", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404664+0000", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1540000000000001e-06 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404669+0000", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9019999999999998e-06 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404678+0000", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9299999999999992e-06 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404683+0000", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6030000000000004e-06 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404734+0000", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0377000000000002e-05 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404838+0000", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010408600000000001 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.404928+0000", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0095999999999999e-05 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.406908+0000", 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019805809999999999 2024-03-26T03:55:45.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.480837+0000", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073928566000000001 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.487912+0000", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0070753600000000002 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.487973+0000", 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0486999999999999e-05 2024-03-26T03:55:45.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:12.488021+0000", 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8639e-05 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:26345 8.b 8:da9e9e34:::benchmark_data_smithi006_103169_object26344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:00.363513+0000", 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.795249179999999, 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082448575999999996, 2024-03-26T03:55:45.210 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 26345 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363513+0000", 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363513+0000", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363514+0000", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0049999999999999e-06 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363605+0000", 2024-03-26T03:55:45.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0919000000000001e-05 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363618+0000", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3349999999999999e-05 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363626+0000", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9729999999999997e-06 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363665+0000", 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8719e-05 2024-03-26T03:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363797+0000", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000131586 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363977+0000", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018076499999999999 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.366505+0000", 2024-03-26T03:55:45.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002527105 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.366513+0000", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2560000000000002e-06 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.445800+0000", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079286923999999995 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.445874+0000", 2024-03-26T03:55:45.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4123999999999997e-05 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.445962+0000", 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7849999999999994e-05 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:26346 8.b 8:deb12efa:::benchmark_data_smithi006_103169_object26345:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:00.363641+0000", 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.795121158000001, 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.083153125999999994, 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.216 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 26346 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363641+0000", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363641+0000", 2024-03-26T03:55:45.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363642+0000", 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7199999999999999e-07 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363670+0000", 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8177000000000001e-05 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363677+0000", 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0310000000000001e-06 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.363683+0000", 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6130000000000003e-06 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.365011+0000", 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013286420000000001 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.365166+0000", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015440699999999999 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.365315+0000", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014958500000000001 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.366670+0000", 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001355092 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.425261+0000", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.058590978000000002 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.446375+0000", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.021113897999999999 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.446679+0000", 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00030361199999999998 2024-03-26T03:55:45.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.446794+0000", 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115519 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:27483 8.b 8:dbdeaa33:::benchmark_data_smithi006_103169_object27482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:00.939733+0000", 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.219028821000002, 2024-03-26T03:55:45.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.085534338000000001, 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 27483 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939733+0000", 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939733+0000", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939734+0000", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999997e-07 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939914+0000", 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000179643 2024-03-26T03:55:45.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939922+0000", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8110000000000003e-06 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939928+0000", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6189999999999996e-06 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.939976+0000", 2024-03-26T03:55:45.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7530000000000001e-05 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.940144+0000", 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016846600000000001 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.940308+0000", 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016384900000000001 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942099+0000", 2024-03-26T03:55:45.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001790352 2024-03-26T03:55:45.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944109+0000", 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020099889999999998 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025150+0000", 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.081041176000000006 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025222+0000", 2024-03-26T03:55:45.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2348000000000006e-05 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025268+0000", 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5645000000000001e-05 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:27486 8.b 8:d656352b:::benchmark_data_smithi006_103169_object27485:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:00.941837+0000", 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.216925357999997, 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.083569881999999998, 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.229 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 27486 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.941837+0000", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.941837+0000", 2024-03-26T03:55:45.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.941838+0000", 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1200000000000002e-07 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942057+0000", 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021892600000000001 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942065+0000", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6580000000000008e-06 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942073+0000", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6310000000000003e-06 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942228+0000", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000155389 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942345+0000", 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116414 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.942472+0000", 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012692000000000001 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944476+0000", 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020040280000000001 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944910+0000", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00043432900000000001 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025313+0000", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080402508999999997 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025365+0000", 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2759000000000001e-05 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.025407+0000", 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1507000000000001e-05 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:27496 8.b 8:d28db805:::benchmark_data_smithi006_103169_object27495:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:00.944669+0000", 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 44.214093480999999, 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084020803000000005, 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.235 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 27496 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944669+0000", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944669+0000", 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.944670+0000", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2999999999999999e-07 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945268+0000", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000598034 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945278+0000", 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0416e-05 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945286+0000", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2269999999999993e-06 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945367+0000", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0807000000000001e-05 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945488+0000", 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121057 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.945637+0000", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000148942 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.946777+0000", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001139648 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:00.947106+0000", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000329363 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.028533+0000", 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.081426703000000003 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.028641+0000", 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108554 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:01.028690+0000", 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8222e-05 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:29935 8.b 8:d6580ba5:::benchmark_data_smithi006_103169_object29934:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:02.086105+0000", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 43.072657341000003, 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.080098143999999996, 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 29935 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086105+0000", 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086105+0000", 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086106+0000", 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4000000000000001e-07 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086132+0000", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6128000000000001e-05 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086140+0000", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1720000000000007e-06 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086146+0000", 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5100000000000004e-06 2024-03-26T03:55:45.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086203+0000", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6977e-05 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086338+0000", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134058 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.086478+0000", 2024-03-26T03:55:45.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014060400000000001 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.088083+0000", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016048620000000001 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.088592+0000", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00050889499999999999 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.166091+0000", 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.077499246999999993 2024-03-26T03:55:45.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.166154+0000", 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3260000000000001e-05 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.166203+0000", 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8690999999999998e-05 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:30785 8.b 8:de9df25f:::benchmark_data_smithi006_103169_object30784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:02.528180+0000", 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 42.630581999, 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.093945925999999999, 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 30785 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528180+0000", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528180+0000", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528181+0000", 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.16e-07 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528251+0000", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0374000000000001e-05 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528260+0000", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9339999999999995e-06 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528268+0000", 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3039999999999999e-06 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528615+0000", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00034712099999999999 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528733+0000", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118594 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.528887+0000", 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000153327 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.530258+0000", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013707770000000001 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.530813+0000", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00055527500000000004 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.622029+0000", 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091216063 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.622086+0000", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6867999999999998e-05 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.622126+0000", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0472999999999997e-05 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:30792 8.b 8:db5b6dae:::benchmark_data_smithi006_103169_object30791:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:02.530921+0000", 2024-03-26T03:55:45.252 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 42.627841179000001, 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.096009653, 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 30792 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.530921+0000", 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.530921+0000", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.530922+0000", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.22e-06 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531043+0000", 2024-03-26T03:55:45.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012029299999999999 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531052+0000", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3680000000000008e-06 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531059+0000", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2409999999999998e-06 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531106+0000", 2024-03-26T03:55:45.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6313000000000002e-05 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531223+0000", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117509 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.531360+0000", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013727100000000001 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.532806+0000", 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001445539 2024-03-26T03:55:45.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.532929+0000", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012332900000000001 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.626816+0000", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.093887100000000001 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.626886+0000", 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9552000000000001e-05 2024-03-26T03:55:45.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:02.626931+0000", 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4917999999999999e-05 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:71161 8.4 8:230d9420:::benchmark_data_smithi006_103169_object71160:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:28.782930+0000", 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 16.375832407000001, 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082751406, 2024-03-26T03:55:45.258 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 71161 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.782930+0000", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.782930+0000", 2024-03-26T03:55:45.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.782931+0000", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1800000000000005e-07 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783011+0000", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0683e-05 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783021+0000", 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3170000000000008e-06 2024-03-26T03:55:45.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783027+0000", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2670000000000003e-06 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783080+0000", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3495000000000002e-05 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783207+0000", 2024-03-26T03:55:45.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000126227 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2", 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.783359+0000", 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015224499999999999 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.786130+0000", 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002770851 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.865503+0000", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079373025999999999 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.865568+0000", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5801000000000002e-05 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:28.865681+0000", 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011277600000000001 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.263 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:77054 8.b 8:ddb9df34:::benchmark_data_smithi006_103169_object77053:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:33.018555+0000", 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.140207156000001, 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.087070723000000003, 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 77054 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018555+0000", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018555+0000", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018556+0000", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6599999999999996e-07 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018610+0000", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3916999999999999e-05 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018617+0000", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.007e-06 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.018623+0000", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3219999999999998e-06 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.020926+0000", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00230299 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.021029+0000", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103101 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.021179+0000", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000149567 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.022846+0000", 2024-03-26T03:55:45.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001667296 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.027936+0000", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050905780000000001 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.105559+0000", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.077622452999999994 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.105598+0000", 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9020000000000002e-05 2024-03-26T03:55:45.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.105626+0000", 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7906e-05 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:77069 8.b 8:d68c8ae5:::benchmark_data_smithi006_103169_object77068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:33.030808+0000", 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.127954206, 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082298660999999995, 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.269 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 77069 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030808+0000", 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030808+0000", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030809+0000", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.02e-06 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030893+0000", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3698999999999999e-05 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030901+0000", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1410000000000005e-06 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030907+0000", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.728e-06 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.030959+0000", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2632e-05 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.031104+0000", 2024-03-26T03:55:45.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145174 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.031273+0000", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000168567 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.033149+0000", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001876462 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.035701+0000", 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025519620000000001 2024-03-26T03:55:45.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.113008+0000", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.077306546000000004 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.113066+0000", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7519000000000003e-05 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.113107+0000", 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1211000000000002e-05 2024-03-26T03:55:45.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:77079 8.b 8:d4606063:::benchmark_data_smithi006_103169_object77078:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:33.037582+0000", 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 12.121180445, 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.085386895000000004, 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:55:45.275 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 77079 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037582+0000", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037582+0000", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037583+0000", 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1999999999999998e-07 2024-03-26T03:55:45.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037641+0000", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7881999999999999e-05 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037649+0000", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5010000000000004e-06 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037655+0000", 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8479999999999997e-06 2024-03-26T03:55:45.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037693+0000", 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8152999999999999e-05 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037776+0000", 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3331999999999994e-05 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:55:45.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.037886+0000", 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010954 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.039497+0000", 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016110880000000001 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.040501+0000", 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010040260000000001 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.122858+0000", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082356931999999994 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.122916+0000", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7901e-05 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:33.122969+0000", 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2772000000000002e-05 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:55:45.281 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:55:45.281 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:55:45.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:45.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:45.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:45.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:45.460 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:45.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:45.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:45.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:45.596 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 55 16 95042 95026 107.957 92.125 0.0089062 0.00925884 2024-03-26T03:55:45.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:45.738 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:45.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:45.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:45.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:45.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:45.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:46.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:46.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:46.102 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:46.112 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:46.112 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:45 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:55:46.112 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:45 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:55:46.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:46.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:55:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:55:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:55:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:45 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:55:46.141 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:46.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:46.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:55:46.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:46.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:46.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:55:46.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.430 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:46.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:46.469 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:46.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:46.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:46.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 56 16 96507 96491 107.663 91.5625 0.00977937 0.00928316 2024-03-26T03:55:46.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:46.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:46.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:46.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:46.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:46.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:46.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:46.970 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:46.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:47.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:47.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:47.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:47.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:46 smithi006 ceph-mon[37501]: pgmap v290: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:46 smithi006 ceph-mon[37501]: osdmap e175: 8 total, 7 up, 6 in 2024-03-26T03:55:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:46 smithi006 ceph-mon[29446]: pgmap v290: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:46 smithi006 ceph-mon[29446]: osdmap e175: 8 total, 7 up, 6 in 2024-03-26T03:55:47.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:47.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:47.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:46 smithi086 ceph-mon[32965]: pgmap v290: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:46 smithi086 ceph-mon[32965]: osdmap e175: 8 total, 7 up, 6 in 2024-03-26T03:55:47.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:47.384 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:47.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:47.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:47.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:47.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:47.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 57 16 97854 97838 107.251 84.1875 0.011417 0.00931957 2024-03-26T03:55:47.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:47.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:47.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:47.785 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:47.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:47.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:47.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:47.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:48.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:48.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:48.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:48.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:48.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:48.226 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:55:48.226 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:55:48.226 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-03-26T03:55:48.226 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:55:48.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:48.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:48.333 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-26T03:55:48.334 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:55:48.334 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-26T03:55:48.334 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-03-26T03:55:48.334 DEBUG:teuthology.orchestra.run.smithi086:osd.5> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T03:55:48.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl start ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5 2024-03-26T03:55:48.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:48.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:48.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.586 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:48.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:48.587 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:48 smithi086 systemd[1]: Starting Ceph osd.5 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T03:55:48.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 58 16 99007 98991 106.644 72.0625 0.00283538 0.00936897 2024-03-26T03:55:48.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:48.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:48.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:48.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:48.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:48.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:48.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:48.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:48.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:49.047 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:48 smithi086 ceph-mon[32965]: pgmap v292: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 787 B/s rd, 70 MiB/s wr, 1.12k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:49.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:49.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:55:49.115 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:49.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:48 smithi006 ceph-mon[37501]: pgmap v292: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 787 B/s rd, 70 MiB/s wr, 1.12k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:48 smithi006 ceph-mon[29446]: pgmap v292: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 787 B/s rd, 70 MiB/s wr, 1.12k op/s; 52615/280992 objects degraded (18.725%) 2024-03-26T03:55:49.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:49.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.268 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:49.337 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:49 smithi086 podman[90980]: 2024-03-26 03:55:49.139633852 +0000 UTC m=+0.547424050 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:55:49.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:49 smithi086 podman[90980]: 2024-03-26 03:55:49.26290207 +0000 UTC m=+0.670692267 container create 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T03:55:49.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:49.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:49.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:49.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 59 16 100101 100085 105.996 68.375 0.00371747 0.00942809 2024-03-26T03:55:49.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:49.654 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:49.672 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:49.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:49.782 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:49 smithi086 podman[90980]: 2024-03-26 03:55:49.499179565 +0000 UTC m=+0.906969762 container init 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T03:55:49.782 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:49 smithi086 podman[90980]: 2024-03-26 03:55:49.503906395 +0000 UTC m=+0.911696601 container start 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-26T03:55:49.782 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:49 smithi086 podman[90980]: 2024-03-26 03:55:49.590938568 +0000 UTC m=+0.998728774 container attach 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-26T03:55:49.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:49.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:49.950 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:49.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:50.045 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:49 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:55:50.046 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 52615/280992 objects degraded (18.725%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:50.046 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:49 smithi086 ceph-mon[32965]: osdmap e176: 8 total, 7 up, 6 in 2024-03-26T03:55:50.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 52615/280992 objects degraded (18.725%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[37501]: osdmap e176: 8 total, 7 up, 6 in 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 52615/280992 objects degraded (18.725%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:49 smithi006 ceph-mon[29446]: osdmap e176: 8 total, 7 up, 6 in 2024-03-26T03:55:50.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:50.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.227 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:50.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:50.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T03:55:50.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 bash[90980]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-26T03:55:50.339 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate[91128]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-26T03:55:50.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:50.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:50.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:50.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:50.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:55:50.597872+0000 min lat: 0.00117332 max lat: 1.05068 avg lat: 0.00948294 2024-03-26T03:55:50.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:55:50.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 60 16 101224 101208 105.399 70.1875 0.00623241 0.00948294 2024-03-26T03:55:50.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:50.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.635 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:50.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:50.691 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:50 smithi086 podman[90980]: 2024-03-26 03:55:50.358991707 +0000 UTC m=+1.766781953 container died 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T03:55:50.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:50.774 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:50.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:50.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:50.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:51.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:51.038 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:50 smithi086 ceph-mon[32965]: pgmap v294: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 54278/288546 objects degraded (18.811%) 2024-03-26T03:55:51.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:50 smithi006 ceph-mon[37501]: pgmap v294: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 54278/288546 objects degraded (18.811%) 2024-03-26T03:55:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:50 smithi006 ceph-mon[29446]: pgmap v294: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 5.9 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 54278/288546 objects degraded (18.811%) 2024-03-26T03:55:51.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:51.171 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:51.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:51.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:51.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:51.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:51.443 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:51.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:51.587 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:51 smithi086 podman[91383]: 2024-03-26 03:55:51.193099899 +0000 UTC m=+1.057654298 container remove 943addf6b3e253ee0cfb3766a7966de477fc13dcf69432540386018a11cd00d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-activate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T03:55:51.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 61 16 102387 102371 104.862 72.6875 0.00290367 0.00952222 2024-03-26T03:55:51.604 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:51.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:51.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.750 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:51.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:51.895 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:51.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:55:51.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2590621396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:51.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:51.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:51 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:55:51.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:51 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:55:51.987 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:51 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:51.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:52.028 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:52.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:52.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:52.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2590621396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2590621396' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:55:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:51 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:55:52.174 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:52.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:52.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:52.320 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 podman[91543]: 2024-03-26 03:55:51.995880132 +0000 UTC m=+0.253528091 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T03:55:52.320 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 podman[91543]: 2024-03-26 03:55:52.108443893 +0000 UTC m=+0.366091851 container create a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5) 2024-03-26T03:55:52.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:52.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:52.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:52.492 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:55:52.603 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:55:52.603 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T03:55:52.603 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T03:55:52.603 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:55:52.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:52.605 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 62 16 103770 103754 104.565 86.4375 0.00679753 0.00955908 2024-03-26T03:55:52.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:52.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:52.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:55:52.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:52.778 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:52.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:52.869 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 podman[91543]: 2024-03-26 03:55:52.596854832 +0000 UTC m=+0.854502789 container init a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:55:52.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 podman[91543]: 2024-03-26 03:55:52.602153993 +0000 UTC m=+0.859801942 container start a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T03:55:52.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-osd[91621]: -- 172.21.15.86:0/3035744934 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x56147b379ba0 con 0x56147b34b000 2024-03-26T03:55:52.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-osd[91621]: -- 172.21.15.86:0/3035744934 <== mon.2 v2:172.21.15.6:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56147b381680 con 0x56147b34b000 2024-03-26T03:55:52.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 bash[91543]: a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f 2024-03-26T03:55:52.870 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:52 smithi086 systemd[1]: Started Ceph osd.5 for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T03:55:52.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:52.919 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:52.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:53.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:53.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:53.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:53.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[37501]: pgmap v295: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 727 B/s rd, 100 MiB/s wr, 1.59k op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:53.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[37501]: osdmap e177: 8 total, 7 up, 6 in 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[29446]: pgmap v295: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 727 B/s rd, 100 MiB/s wr, 1.59k op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:52 smithi006 ceph-mon[29446]: osdmap e177: 8 total, 7 up, 6 in 2024-03-26T03:55:53.156 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-mon[32965]: pgmap v295: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 727 B/s rd, 100 MiB/s wr, 1.59k op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:53.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:55:53.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:55:53.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:52 smithi086 ceph-mon[32965]: osdmap e177: 8 total, 7 up, 6 in 2024-03-26T03:55:53.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:53.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:53.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:53.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:53.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:53.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:53.460 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T03:55:53.154+0000 7f07802e97c0 -1 Falling back to public interface 2024-03-26T03:55:53.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:53.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:53.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:55:53.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:53.598 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 63 16 105318 105302 104.441 96.75 0.008254 0.00957008 2024-03-26T03:55:53.608 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:53.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:53.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:53.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:53.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:53.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:53.916 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:53.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:53.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T03:55:53.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T03:55:53.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[29446]: 3.12 scrub starts 2024-03-26T03:55:53.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[29446]: 3.12 scrub ok 2024-03-26T03:55:53.969 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:53.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T03:55:53.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T03:55:53.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[37501]: 3.12 scrub starts 2024-03-26T03:55:53.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[37501]: 3.12 scrub ok 2024-03-26T03:55:53.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:54.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:54.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T03:55:54.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T03:55:54.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-mon[32965]: 3.12 scrub starts 2024-03-26T03:55:54.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-mon[32965]: 3.12 scrub ok 2024-03-26T03:55:54.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:55:54.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:54.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:54.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:54.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:54.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:54.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:54.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:54.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:54.483 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:55:54.535 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:55:54.536 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:55:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:55:54.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:54.576 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:55:54.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:54.597 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 64 16 106737 106721 104.195 88.6875 0.00421226 0.00959265 2024-03-26T03:55:54.618 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:55:54.662 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:55:54.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:55:54.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:54.745 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:55:54.746 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:55:54.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:54.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:55:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:55:54.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:54.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:54.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:55.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:55.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:54 smithi086 ceph-mon[32965]: pgmap v297: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 687 B/s rd, 55 MiB/s wr, 879 op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:54 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 56311/300630 objects degraded (18.731%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:54 smithi086 ceph-mon[32965]: osdmap e178: 8 total, 7 up, 6 in 2024-03-26T03:55:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1859947990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[37501]: pgmap v297: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 687 B/s rd, 55 MiB/s wr, 879 op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 56311/300630 objects degraded (18.731%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:55.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[37501]: osdmap e178: 8 total, 7 up, 6 in 2024-03-26T03:55:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1859947990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[29446]: pgmap v297: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.1 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 687 B/s rd, 55 MiB/s wr, 879 op/s; 56311/300630 objects degraded (18.731%) 2024-03-26T03:55:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 56311/300630 objects degraded (18.731%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:55:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[29446]: osdmap e178: 8 total, 7 up, 6 in 2024-03-26T03:55:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1859947990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:55:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:55.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:55:55.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.295 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:55.378 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:55.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:no valid command found; 10 closest matches: 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:0 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:1 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:2 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:abort 2024-03-26T03:55:55.431 INFO:teuthology.orchestra.run.smithi086.stderr:assert 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:config diff 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:config diff get 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:config get 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:config help [] 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:config set ... 2024-03-26T03:55:55.432 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: invalid command 2024-03-26T03:55:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:55.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:55.543 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T03:55:55.544 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T03:55:55.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.591 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:55.598 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 65 16 108155 108139 103.955 88.625 0.00524849 0.00961468 2024-03-26T03:55:55.622 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:55:55.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:55.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:55.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:55.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:55.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:55.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:55.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:56.063 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:56.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:56.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:56.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:55:56.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:56.263 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:56.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:56.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:56.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:56.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:56.598 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 66 16 109447 109431 103.603 80.75 0.00888102 0.00964764 2024-03-26T03:55:56.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:56.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:56.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:56.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:56.810 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:56.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:56.935 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:56.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:56.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[29446]: pgmap v299: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 848 B/s rd, 111 MiB/s wr, 1.77k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:56.945 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[37501]: pgmap v299: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 848 B/s rd, 111 MiB/s wr, 1.77k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:57.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:57.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:57.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:57.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-mon[32965]: pgmap v299: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 848 B/s rd, 111 MiB/s wr, 1.77k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:57.066 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:55:57.067 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:55:57.067 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-mon[32965]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.067 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.067 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T03:55:56.794+0000 7f07802e97c0 -1 osd.5 167 log_to_monitors true 2024-03-26T03:55:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-26T03:55:57.093 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-26T03:55:57.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:57.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:57.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:57.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:57.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:55:57.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:55:57.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[37501]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:55:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:55:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[29446]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:56 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T03:55:57.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:57.458 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:57.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:57.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:55:57.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:57.599 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 67 15 110355 110340 102.903 56.8125 0.00775795 0.0097132 2024-03-26T03:55:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:57.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:57.711 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T03:55:57.383+0000 7f076d739700 -1 osd.5 167 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T03:55:57.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.780 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:57.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:57.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:57.921 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:57.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:58.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: osdmap e179: 8 total, 7 up, 6 in 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: pgmap v301: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:58.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:57 smithi086 ceph-mon[32965]: from='osd.5 ' entity='osd.5' 2024-03-26T03:55:58.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:58.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:58.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:55:58.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:58.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:55:58.192 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:55:58.192 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:55:58.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:58.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: osdmap e179: 8 total, 7 up, 6 in 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: pgmap v301: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[37501]: from='osd.5 ' entity='osd.5' 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: from='osd.5 v2:172.21.15.86:6804/909197223' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: osdmap e179: 8 total, 7 up, 6 in 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]: dispatch 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: pgmap v301: 148 pgs: 30 active+undersized+degraded, 42 active+undersized, 76 active+clean; 6.5 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 70 MiB/s wr, 1.11k op/s; 60056/320673 objects degraded (18.728%) 2024-03-26T03:55:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:57 smithi006 ceph-mon[29446]: from='osd.5 ' entity='osd.5' 2024-03-26T03:55:58.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:58.447 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:58.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:58.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:58.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:58.599 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 68 16 111141 111125 102.111 49.0625 0.00229796 0.00976184 2024-03-26T03:55:58.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:55:58.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:58.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:55:58.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:58.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:55:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:58.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:58.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:55:58.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:58.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:55:59.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:59.065 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:55:59.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T03:55:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T03:55:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T03:55:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T03:55:59.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:55:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:55:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: osd.5 v2:172.21.15.86:6804/909197223 boot 2024-03-26T03:55:59.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:58 smithi086 ceph-mon[32965]: osdmap e180: 8 total, 8 up, 6 in 2024-03-26T03:55:59.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:59.207 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: osd.5 v2:172.21.15.86:6804/909197223 boot 2024-03-26T03:55:59.262 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[37501]: osdmap e180: 8 total, 8 up, 6 in 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: osd.5 v2:172.21.15.86:6804/909197223 boot 2024-03-26T03:55:59.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:58 smithi006 ceph-mon[29446]: osdmap e180: 8 total, 8 up, 6 in 2024-03-26T03:55:59.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:55:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:55:59.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:55:59.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:55:59.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:55:59.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:59.591 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:59.599 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 69 16 111142 111126 100.632 0.0625 0.173012 0.00976331 2024-03-26T03:55:59.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:59.719 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:55:59.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:55:59.846 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:55:59.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:55:59.972 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:55:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:55:59.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:00.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:00.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:00.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:00.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:00.227 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:00.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T03:56:00.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T03:56:00.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 60056/320673 objects degraded (18.728%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:00.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[37501]: osdmap e181: 8 total, 8 up, 6 in 2024-03-26T03:56:00.326 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[37501]: pgmap v304: 148 pgs: 17 peering, 23 active+undersized+degraded, 32 active+undersized, 76 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1016 B/s rd, 64 MiB/s wr, 1.02k op/s; 53453/323397 objects degraded (16.529%) 2024-03-26T03:56:00.327 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T03:56:00.327 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T03:56:00.327 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 60056/320673 objects degraded (18.728%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:00.327 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[29446]: osdmap e181: 8 total, 8 up, 6 in 2024-03-26T03:56:00.327 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:55:59 smithi006 ceph-mon[29446]: pgmap v304: 148 pgs: 17 peering, 23 active+undersized+degraded, 32 active+undersized, 76 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1016 B/s rd, 64 MiB/s wr, 1.02k op/s; 53453/323397 objects degraded (16.529%) 2024-03-26T03:56:00.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:59 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T03:56:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:59 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T03:56:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:59 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 60056/320673 objects degraded (18.728%), 30 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:59 smithi086 ceph-mon[32965]: osdmap e181: 8 total, 8 up, 6 in 2024-03-26T03:56:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:55:59 smithi086 ceph-mon[32965]: pgmap v304: 148 pgs: 17 peering, 23 active+undersized+degraded, 32 active+undersized, 76 active+clean; 6.6 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1016 B/s rd, 64 MiB/s wr, 1.02k op/s; 53453/323397 objects degraded (16.529%) 2024-03-26T03:56:00.357 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:00.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:00.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:00.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:00.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:00.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 70 16 111157 111141 99.2083 0.9375 0.00392564 0.00993876 2024-03-26T03:56:00.638 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:00.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:00.669 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:00.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:00.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:00.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:00.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:00.923 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:00.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:00.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:01.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:01.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:01.113 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:01.114 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:01.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:01.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.236 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:01.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:01.384 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:01.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:01.527 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:01.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:01 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T03:56:01.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:01 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T03:56:01.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:01 smithi086 ceph-mon[32965]: osdmap e182: 8 total, 8 up, 6 in 2024-03-26T03:56:01.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:01 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T03:56:01.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:01 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T03:56:01.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 71 16 111974 111958 98.5302 51.0625 0.0226859 0.0101439 2024-03-26T03:56:01.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:01.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T03:56:01.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T03:56:01.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[29446]: osdmap e182: 8 total, 8 up, 6 in 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[37501]: osdmap e182: 8 total, 8 up, 6 in 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T03:56:01.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:01 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T03:56:01.680 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:01.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:01.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.824 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:01.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:01.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:01.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:01.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:02.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:02.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:56:02.118 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:02.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:02.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:02.310 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:56:02.310 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:02 smithi006 ceph-mon[37501]: pgmap v306: 148 pgs: 8 remapped+peering, 17 peering, 5 active+undersized+degraded, 16 active+undersized, 102 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 22/334092 objects degraded (0.007%) 2024-03-26T03:56:02.310 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:02 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:02.310 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:02 smithi006 ceph-mon[29446]: pgmap v306: 148 pgs: 8 remapped+peering, 17 peering, 5 active+undersized+degraded, 16 active+undersized, 102 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 22/334092 objects degraded (0.007%) 2024-03-26T03:56:02.311 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:02 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:02.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:02.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:02.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:02.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:02 smithi086 ceph-mon[32965]: pgmap v306: 148 pgs: 8 remapped+peering, 17 peering, 5 active+undersized+degraded, 16 active+undersized, 102 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 22/334092 objects degraded (0.007%) 2024-03-26T03:56:02.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:02 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:02.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 72 16 112830 112814 97.9048 53.5 0.0171767 0.010209 2024-03-26T03:56:02.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:02.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:56:02.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:02.801 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:02.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:02.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:02.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:02.956 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:56:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:02.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:03.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:56:03.039 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:56:03.079 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:56:03.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:03.118 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:56:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.142 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:03.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:03.156 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:56:03.197 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:56:03.242 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:56:03.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:56:03.249 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:56:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:03.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:03.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:03.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:03.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:03.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.562 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:03.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:03 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:03 smithi086 ceph-mon[32965]: osdmap e183: 8 total, 8 up, 6 in 2024-03-26T03:56:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/965453397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:03.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 73 16 113745 113729 97.3471 57.1875 0.0137821 0.0102678 2024-03-26T03:56:03.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[37501]: osdmap e183: 8 total, 8 up, 6 in 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/965453397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/549671228' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[29446]: osdmap e183: 8 total, 8 up, 6 in 2024-03-26T03:56:03.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/965453397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:03.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:03.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:03.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:03.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:03.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:03.993 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:03.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:04.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:04.107 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:04.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:04.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:04.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:04.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:04.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:04.409 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:04.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:04.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:04.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:04.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:04.581 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:04 smithi086 ceph-mon[32965]: pgmap v308: 148 pgs: 8 remapped+peering, 17 peering, 123 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 57 MiB/s wr, 909 op/s 2024-03-26T03:56:04.581 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:04 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:04.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 74 16 114779 114763 96.9049 64.625 0.0291918 0.0103151 2024-03-26T03:56:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:04 smithi006 ceph-mon[37501]: pgmap v308: 148 pgs: 8 remapped+peering, 17 peering, 123 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 57 MiB/s wr, 909 op/s 2024-03-26T03:56:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:04 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:04.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:04 smithi006 ceph-mon[29446]: pgmap v308: 148 pgs: 8 remapped+peering, 17 peering, 123 active+clean; 6.8 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 57 MiB/s wr, 909 op/s 2024-03-26T03:56:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:04 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:04.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:04.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:04.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:04.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:04.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:04.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115142 8.0 e184/181 8:0c49243f:::benchmark_data_smithi006_103169_object115141:head v 184'7209, mlcod=167'3666)", 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.959994+0000", 2024-03-26T03:56:04.965 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0040510169999999996, 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0040831729999999998, 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959994+0000", 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959994+0000", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959994+0000", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1100000000000002e-07 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960030+0000", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5642999999999997e-05 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960035+0000", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2129999999999999e-06 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960040+0000", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5059999999999997e-06 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.962234+0000", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021943700000000002 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.962257+0000", 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2714999999999999e-05 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPush(8.2 184/181 [PushOp(8:4767fec4:::benchmark_data_smithi006_103169_object63399:head, version: 169'3907, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:4767fec4:::benchmark_data_smithi006_103169_object63399:head@169'3907, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.960056+0000", 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0039889840000000001, 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041175329999999996, 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960056+0000", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960056+0000", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960056+0000", 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2099999999999998e-07 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960085+0000", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8765e-05 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960098+0000", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.329e-05 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960110+0000", 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1934e-05 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.971 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115143 8.9 e184/181)", 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.961567+0000", 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.002477801, 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026551130000000002, 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961567+0000", 2024-03-26T03:56:04.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961567+0000", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961568+0000", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.23e-07 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961607+0000", 2024-03-26T03:56:04.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9044000000000001e-05 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961618+0000", 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1294999999999999e-05 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961625+0000", 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7170000000000002e-06 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115146 8.8 e184/181 8:1dde4fcc:::benchmark_data_smithi006_103169_object115145:head v 184'7059, mlcod=167'3656)", 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.961643+0000", 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0024016839999999999, 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026308400000000002, 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961643+0000", 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961643+0000", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961644+0000", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2300000000000002e-07 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961678+0000", 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4255000000000003e-05 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961683+0000", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3539999999999999e-06 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.961688+0000", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6789999999999996e-06 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.962425+0000", 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00073746399999999996 2024-03-26T03:56:04.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.962452+0000", 2024-03-26T03:56:04.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6706000000000001e-05 2024-03-26T03:56:04.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115125 8.8 e184/181 8:111e3786:::benchmark_data_smithi006_103169_object115124:head v 184'7057, mlcod=167'3656)", 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.952007+0000", 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.012037847000000001, 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012331679999999999, 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952007+0000", 2024-03-26T03:56:04.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952007+0000", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952008+0000", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1299999999999999e-07 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952042+0000", 2024-03-26T03:56:04.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4038000000000002e-05 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952054+0000", 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2065000000000001e-05 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952061+0000", 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3139999999999998e-06 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952119+0000", 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7951000000000001e-05 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.952147+0000", 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7906999999999999e-05 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.982 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115133 8.8 e184/181 8:1a7d2e94:::benchmark_data_smithi006_103169_object115132:head v 184'7058, mlcod=167'3656)", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.955138+0000", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0089072289999999991, 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0092628290000000002, 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955138+0000", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955138+0000", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955139+0000", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.02e-06 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955176+0000", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6968999999999998e-05 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955188+0000", 2024-03-26T03:56:04.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2412000000000001e-05 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.955198+0000", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5759999999999995e-06 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957228+0000", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020306120000000002 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957253+0000", 2024-03-26T03:56:04.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4819000000000001e-05 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115140 8.9 e184/181 8:9837037c:::benchmark_data_smithi006_103169_object115139:head v 184'7343, mlcod=167'3813)", 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.957523+0000", 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0065220469999999996, 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069397929999999997, 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957523+0000", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957523+0000", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957524+0000", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6099999999999999e-07 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957560+0000", 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6477000000000003e-05 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957570+0000", 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8660000000000001e-06 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957578+0000", 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5050000000000001e-06 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957722+0000", 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014396499999999999 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.957743+0000", 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1531000000000001e-05 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115138 8.d e184/181 8:b2e2e92b:::benchmark_data_smithi006_103169_object115137:head v 184'7195, mlcod=167'3768)", 2024-03-26T03:56:04.989 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.959788+0000", 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0042572570000000004, 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047382120000000003, 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959788+0000", 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959788+0000", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959789+0000", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3500000000000005e-07 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959831+0000", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2574000000000003e-05 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959847+0000", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5549e-05 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959854+0000", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2509999999999997e-06 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960228+0000", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00037443600000000002 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.960255+0000", 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6378999999999999e-05 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15039.0:115137 8.a e184/181 8:5def644d:::benchmark_data_smithi006_103169_object115136:head v 184'7118, mlcod=167'3673)", 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:04.959923+0000", 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0041218080000000002, 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046666219999999996, 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:56:04.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959923+0000", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959923+0000", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959924+0000", 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1300000000000001e-07 2024-03-26T03:56:04.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959962+0000", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8503e-05 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959969+0000", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9009999999999997e-06 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.959974+0000", 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8620000000000002e-06 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.963728+0000", 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037544050000000002 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:04.963754+0000", 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5403999999999999e-05 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T03:56:04.996 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 9 2024-03-26T03:56:04.997 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:56:04.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:04.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:04.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:05.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:05.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:05.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:05.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:05.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:05.422 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:05.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:56:05.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:05.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:05.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:05 smithi086 ceph-mon[32965]: osdmap e184: 8 total, 8 up, 6 in 2024-03-26T03:56:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:05 smithi086 ceph-mon[32965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded) 2024-03-26T03:56:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:05 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:56:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:05.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 75 16 115811 115795 96.4728 64.5 0.0174579 0.0103608 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[37501]: osdmap e184: 8 total, 8 up, 6 in 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[37501]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded) 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[29446]: osdmap e184: 8 total, 8 up, 6 in 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[29446]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/334092 objects degraded (0.007%), 5 pgs degraded) 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:56:05.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:05.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:05.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:05.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:05.811 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:05.865 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:05.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:05.897 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:05.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:06.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:06.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:56:06.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:06.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:06.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.291 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:06.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:06.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:06.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:06.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:06.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:06.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:06.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:06.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:06 smithi086 ceph-mon[32965]: pgmap v310: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 28382/345114 objects degraded (8.224%); 4.8 MiB/s, 77 objects/s recovering 2024-03-26T03:56:06.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 76 16 116775 116759 95.9962 60.25 0.0173259 0.0104123 2024-03-26T03:56:06.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:06 smithi006 ceph-mon[37501]: pgmap v310: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 28382/345114 objects degraded (8.224%); 4.8 MiB/s, 77 objects/s recovering 2024-03-26T03:56:06.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:06 smithi006 ceph-mon[29446]: pgmap v310: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 853 B/s rd, 75 MiB/s wr, 1.21k op/s; 28382/345114 objects degraded (8.224%); 4.8 MiB/s, 77 objects/s recovering 2024-03-26T03:56:06.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:06.708 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:06.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:06.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:06.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:06.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:06.980 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:06.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:06.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:07.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:07.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:07.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:07.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:07.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:07.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:07.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:07 smithi006 ceph-mon[29446]: Health check failed: Degraded data redundancy: 28382/345114 objects degraded (8.224%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:07.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:07 smithi006 ceph-mon[37501]: Health check failed: Degraded data redundancy: 28382/345114 objects degraded (8.224%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:07.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:07.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:07.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.585 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:07.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:07.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:07 smithi086 ceph-mon[32965]: Health check failed: Degraded data redundancy: 28382/345114 objects degraded (8.224%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:07.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 77 16 117770 117754 95.5571 62.1875 0.0203174 0.0104605 2024-03-26T03:56:07.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:07.732 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:07.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:07.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:07.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:07.926 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:07.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:08.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:08.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:08.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:08.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:08.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:08.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:08.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:08.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:08.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:08.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:08.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:08.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:08.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 78 16 118605 118589 95.0011 52.1875 0.00702905 0.0105182 2024-03-26T03:56:08.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[37501]: pgmap v311: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 727 B/s rd, 33 MiB/s wr, 522 op/s; 28382/345114 objects degraded (8.224%); 4.1 MiB/s, 66 objects/s recovering 2024-03-26T03:56:08.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:08.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[29446]: pgmap v311: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 727 B/s rd, 33 MiB/s wr, 522 op/s; 28382/345114 objects degraded (8.224%); 4.1 MiB/s, 66 objects/s recovering 2024-03-26T03:56:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:08.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:08 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:08.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:08.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:08.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:08 smithi086 ceph-mon[32965]: pgmap v311: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 727 B/s rd, 33 MiB/s wr, 522 op/s; 28382/345114 objects degraded (8.224%); 4.1 MiB/s, 66 objects/s recovering 2024-03-26T03:56:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:08 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:08 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:08.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:08.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:08.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:08.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:08.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:09.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:09.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:09.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:09.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:09.253 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:09.301 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:56:09.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:09.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:09.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:09.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:09.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 79 16 119469 119453 94.4821 54 0.0140877 0.0105796 2024-03-26T03:56:09.629 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:09.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: osdmap e185: 8 total, 8 up, 6 in 2024-03-26T03:56:09.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2299203650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:09.645 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[29446]: osdmap e186: 8 total, 8 up, 6 in 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: osdmap e185: 8 total, 8 up, 6 in 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2299203650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:56:09.646 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:09 smithi006 ceph-mon[37501]: osdmap e186: 8 total, 8 up, 6 in 2024-03-26T03:56:09.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.666 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:09.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:09.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:09.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: osdmap e185: 8 total, 8 up, 6 in 2024-03-26T03:56:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2299203650' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:56:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:56:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:09 smithi086 ceph-mon[32965]: osdmap e186: 8 total, 8 up, 6 in 2024-03-26T03:56:09.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:09.937 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:09.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:09.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:10.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:10.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:10.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:10.217 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:10.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:10.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:10.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:10.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:10.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:56:10.601265+0000 min lat: 0.00117332 max lat: 2.24849 avg lat: 0.0106205 2024-03-26T03:56:10.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:56:10.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 80 16 120512 120496 94.1159 65.1875 0.00625724 0.0106205 2024-03-26T03:56:10.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:10.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:10.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:10.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:10 smithi086 ceph-mon[32965]: pgmap v314: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 28132/347742 objects degraded (8.090%); 7.4 MiB/s, 119 objects/s recovering 2024-03-26T03:56:10.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:10.780 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:10.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:10 smithi006 ceph-mon[29446]: pgmap v314: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 28132/347742 objects degraded (8.090%); 7.4 MiB/s, 119 objects/s recovering 2024-03-26T03:56:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:10 smithi006 ceph-mon[37501]: pgmap v314: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.7 KiB/s rd, 47 MiB/s wr, 759 op/s; 28132/347742 objects degraded (8.090%); 7.4 MiB/s, 119 objects/s recovering 2024-03-26T03:56:10.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:10.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:11.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:11.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:11.077 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:56:11.077 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:56:11.077 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T03:56:11.077 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T03:56:11.077 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:56:11.164 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:56:11.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:11.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:11.209 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:56:11.257 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:56:11.296 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:56:11.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:11.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:56:11.347 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:11.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:56:11.422 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:56:11.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:11.468 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:56:11.468 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/948259330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:11.468 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/948259330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:56:11.474 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:56:11.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:11.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:11.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 81 16 121499 121483 93.7156 61.6875 0.0154062 0.010666 2024-03-26T03:56:11.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:11.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:11.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:11.750 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/948259330' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:11.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:11.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:56:11.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:11.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:11.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:11.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:12.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:12.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:12.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:12.245 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:12.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:12.323 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:12.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:12.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:12.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:12.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:12.600 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 82 16 122544 122528 93.3692 65.3125 0.00871091 0.010706 2024-03-26T03:56:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:12.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:12 smithi086 ceph-mon[32965]: pgmap v315: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 727 B/s rd, 69 MiB/s wr, 1.10k op/s; 27767/359166 objects degraded (7.731%); 7.9 MiB/s, 126 objects/s recovering 2024-03-26T03:56:12.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:12 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 27767/359166 objects degraded (7.731%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:12.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:12 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:12.652 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:12 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:12.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:12.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:12.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:12.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:12.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[37501]: pgmap v315: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 727 B/s rd, 69 MiB/s wr, 1.10k op/s; 27767/359166 objects degraded (7.731%); 7.9 MiB/s, 126 objects/s recovering 2024-03-26T03:56:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 27767/359166 objects degraded (7.731%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[29446]: pgmap v315: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 727 B/s rd, 69 MiB/s wr, 1.10k op/s; 27767/359166 objects degraded (7.731%); 7.9 MiB/s, 126 objects/s recovering 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 27767/359166 objects degraded (7.731%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:12 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:12.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:13.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:13.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:13.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:13.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:13.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:56:13.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:13.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:13.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:13.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:13.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:13.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:13.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:13.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:13.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 83 16 123581 123565 93.0251 64.8125 0.0121125 0.0107449 2024-03-26T03:56:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:13 smithi006 ceph-mon[37501]: osdmap e187: 8 total, 8 up, 6 in 2024-03-26T03:56:13.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:13 smithi006 ceph-mon[29446]: osdmap e187: 8 total, 8 up, 6 in 2024-03-26T03:56:13.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:13.704 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:13.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:13.828 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:13 smithi086 ceph-mon[32965]: osdmap e187: 8 total, 8 up, 6 in 2024-03-26T03:56:13.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:13.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:13.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:13.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:13.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:14.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:14.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:14.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:56:14.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:14.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:14.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:14.329 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:14.330 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:14.341 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:14.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:14.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:14.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:14.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:14.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:14.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 84 16 124558 124542 92.6446 61.0625 0.0246697 0.0107887 2024-03-26T03:56:14.626 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:14.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:14.759 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:14 smithi086 ceph-mon[32965]: pgmap v317: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s; 27767/359166 objects degraded (7.731%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:56:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:14 smithi086 ceph-mon[32965]: osdmap e188: 8 total, 8 up, 6 in 2024-03-26T03:56:14.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:14.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:14.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:14 smithi006 ceph-mon[37501]: pgmap v317: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s; 27767/359166 objects degraded (7.731%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:56:14.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:14 smithi006 ceph-mon[37501]: osdmap e188: 8 total, 8 up, 6 in 2024-03-26T03:56:14.876 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:14 smithi006 ceph-mon[29446]: pgmap v317: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.3 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 49 MiB/s wr, 780 op/s; 27767/359166 objects degraded (7.731%); 6.4 MiB/s, 102 objects/s recovering 2024-03-26T03:56:14.876 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:14 smithi006 ceph-mon[29446]: osdmap e188: 8 total, 8 up, 6 in 2024-03-26T03:56:14.900 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:14.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:14.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:14.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:56:14.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:56:14.925 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-03-26T03:56:14.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:56:15.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:15.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:15.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:15.176 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:15.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:15.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:15.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:15.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:15.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:15.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 85 16 125432 125416 92.1973 54.625 0.0176407 0.0108417 2024-03-26T03:56:15.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:15.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:15.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:15 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:15 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:15.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:15.853 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:15.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:15.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:15.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:15.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:15.994 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:15.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:15.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:16.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:16.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:16.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:16.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:16.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:16.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:16.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:16.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.525 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:16.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:16.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 86 16 126210 126194 91.6907 48.625 0.0122892 0.0109012 2024-03-26T03:56:16.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:16.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:16 smithi006 ceph-mon[37501]: pgmap v319: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 847 B/s rd, 93 MiB/s wr, 1.48k op/s; 27127/374562 objects degraded (7.242%); 10 MiB/s, 166 objects/s recovering 2024-03-26T03:56:16.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:16 smithi006 ceph-mon[29446]: pgmap v319: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 847 B/s rd, 93 MiB/s wr, 1.48k op/s; 27127/374562 objects degraded (7.242%); 10 MiB/s, 166 objects/s recovering 2024-03-26T03:56:16.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:16.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.797 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:16.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:16 smithi086 ceph-mon[32965]: pgmap v319: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 847 B/s rd, 93 MiB/s wr, 1.48k op/s; 27127/374562 objects degraded (7.242%); 10 MiB/s, 166 objects/s recovering 2024-03-26T03:56:16.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:16.936 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:16.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:17.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:17.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:56:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:17.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:17.159 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:17.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:17.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:17.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:17.313 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:17.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:17.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:17.396 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:17.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:17.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:17.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:17.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:17.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 87 16 126996 126980 91.2014 49.125 0.0126275 0.0109545 2024-03-26T03:56:17.619 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:17.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:17.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:17.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:17.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:17.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:17.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:17.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:17.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:18.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:18.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:18.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:18.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:18.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:18.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:18.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:18.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:18.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:18.456 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:18.466 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:18.557 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:18.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 88 16 127973 127957 90.8589 61.0625 0.0292984 0.0110013 2024-03-26T03:56:18.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[37501]: pgmap v320: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s; 27127/374562 objects degraded (7.242%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:18.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:18.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:18.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[37501]: osdmap e189: 8 total, 8 up, 6 in 2024-03-26T03:56:18.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[29446]: pgmap v320: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s; 27127/374562 objects degraded (7.242%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:18.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:18.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:18.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:18 smithi006 ceph-mon[29446]: osdmap e189: 8 total, 8 up, 6 in 2024-03-26T03:56:18.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:18.711 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:18 smithi086 ceph-mon[32965]: pgmap v320: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s; 27127/374562 objects degraded (7.242%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:18.711 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:18 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:18.711 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:18 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:18.711 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:18 smithi086 ceph-mon[32965]: osdmap e189: 8 total, 8 up, 6 in 2024-03-26T03:56:18.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:18.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:18.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:18.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:18.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:19.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:19.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:19.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:19.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:19.305 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:19.307 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:19.307 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:19.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:19.452 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:19.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:19 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 27127/374562 objects degraded (7.242%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:19 smithi086 ceph-mon[32965]: osdmap e190: 8 total, 8 up, 6 in 2024-03-26T03:56:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:19.592 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:19.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:19.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 89 16 128930 128914 90.5101 59.8125 0.00949212 0.0110441 2024-03-26T03:56:19.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:19.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:19.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 27127/374562 objects degraded (7.242%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[37501]: osdmap e190: 8 total, 8 up, 6 in 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 27127/374562 objects degraded (7.242%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[29446]: osdmap e190: 8 total, 8 up, 6 in 2024-03-26T03:56:19.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:19.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:19.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:19.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:19.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:20.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:20.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:20.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:56:20.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:20.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:20.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:20.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:20.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:56:20.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:20.399 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:20.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:20.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[37501]: pgmap v323: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 991 op/s; 26791/376986 objects degraded (7.107%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[37501]: osdmap e191: 8 total, 8 up, 6 in 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[29446]: pgmap v323: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 991 op/s; 26791/376986 objects degraded (7.107%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:20.565 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:20.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:20.566 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:20 smithi006 ceph-mon[29446]: osdmap e191: 8 total, 8 up, 6 in 2024-03-26T03:56:20.595 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:20.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total time run: 90.0121 2024-03-26T03:56:20.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total writes made: 129928 2024-03-26T03:56:20.601 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Write size: 65536 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object size: 65536 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Bandwidth (MB/sec): 90.2157 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Bandwidth: 38.003 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max bandwidth (MB/sec): 213 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min bandwidth (MB/sec): 0.0625 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average IOPS: 1443 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev IOPS: 608.048 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max IOPS: 3408 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min IOPS: 1 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average Latency(s): 0.0110804 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Latency(s): 0.028249 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max latency(s): 2.24849 2024-03-26T03:56:20.602 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min latency(s): 0.00117332 2024-03-26T03:56:20.610 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T03:56:20.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.743 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:20.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:20.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:20 smithi086 ceph-mon[32965]: pgmap v323: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 62 MiB/s wr, 991 op/s; 26791/376986 objects degraded (7.107%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:20.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:20 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:20.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2371625566' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:20.816 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:20 smithi086 ceph-mon[32965]: osdmap e191: 8 total, 8 up, 6 in 2024-03-26T03:56:20.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:20.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:20.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:20.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:21.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:21.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:21.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:21.293 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:21.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:21.430 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:21.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:21.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:21.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:21.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:21.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:21.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:21.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:21.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:21.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:21.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:21.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:22.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:22.116 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:22.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.245 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:22.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:22.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:22.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:22.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:22.527 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:22.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:22.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:22.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:22.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:22.790 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:22.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:22 smithi086 ceph-mon[32965]: pgmap v325: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s; 26390/387675 objects degraded (6.807%); 7.7 MiB/s, 122 objects/s recovering 2024-03-26T03:56:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:22.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[37501]: pgmap v325: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s; 26390/387675 objects degraded (6.807%); 7.7 MiB/s, 122 objects/s recovering 2024-03-26T03:56:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[29446]: pgmap v325: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 46 MiB/s wr, 729 op/s; 26390/387675 objects degraded (6.807%); 7.7 MiB/s, 122 objects/s recovering 2024-03-26T03:56:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:22.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:22.916 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:22.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:23.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:23.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:23.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:23.196 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:23.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:23.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:23.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:23.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:23.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:23.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:23.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:23.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:23.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:23.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:23 smithi006 ceph-mon[37501]: osdmap e192: 8 total, 8 up, 6 in 2024-03-26T03:56:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:23 smithi006 ceph-mon[29446]: osdmap e192: 8 total, 8 up, 6 in 2024-03-26T03:56:23.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:23.735 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:23 smithi086 ceph-mon[32965]: osdmap e192: 8 total, 8 up, 6 in 2024-03-26T03:56:23.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:23.819 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:23.819 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:23.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:23.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:23.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:23.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:24.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:24.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:24.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:24.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:24.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:24.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:24.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:24.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:24.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:24.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:24.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:24.579 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:24.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:56:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:24.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:24.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:24.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:24.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:24.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:24.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:24.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:25.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:25.019 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:56:25.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:25.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[37501]: pgmap v327: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 872 B/s rd, 47 MiB/s wr, 745 op/s; 26390/387675 objects degraded (6.807%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 26390/387675 objects degraded (6.807%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[29446]: pgmap v327: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 872 B/s rd, 47 MiB/s wr, 745 op/s; 26390/387675 objects degraded (6.807%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 26390/387675 objects degraded (6.807%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:25.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:25.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:25.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:24 smithi086 ceph-mon[32965]: pgmap v327: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 872 B/s rd, 47 MiB/s wr, 745 op/s; 26390/387675 objects degraded (6.807%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T03:56:25.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:25.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:25.157 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 26390/387675 objects degraded (6.807%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:25.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:25.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:25.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:25.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:25.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:25.528 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:25.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:25.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:25.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:25.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:25.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:25.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:25.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:25.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:25.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:25.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:25.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:26.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:26.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:26.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:26.235 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:56:26.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:26.249 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:26.250 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:56:26.275 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:26.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:26.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:26.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:26.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:26.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:26.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:26.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:26.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:26.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:26.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:26.949 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:26.949 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:26.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:27.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:26 smithi086 ceph-mon[32965]: pgmap v328: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1018 B/s rd, 55 MiB/s wr, 1.51k op/s; 25618/378972 objects degraded (6.760%); 16 MiB/s, 250 objects/s recovering 2024-03-26T03:56:27.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:27.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:26 smithi006 ceph-mon[37501]: pgmap v328: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1018 B/s rd, 55 MiB/s wr, 1.51k op/s; 25618/378972 objects degraded (6.760%); 16 MiB/s, 250 objects/s recovering 2024-03-26T03:56:27.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:26 smithi006 ceph-mon[29446]: pgmap v328: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1018 B/s rd, 55 MiB/s wr, 1.51k op/s; 25618/378972 objects degraded (6.760%); 16 MiB/s, 250 objects/s recovering 2024-03-26T03:56:27.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:27.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:27.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:27.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:27.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:27.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:27.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:27.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:27.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:27.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:56:27.672 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:27.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:27.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:27.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:27.952 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:27.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:27.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:28.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:28.098 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:28.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:27 smithi006 ceph-mon[37501]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:56:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:27 smithi006 ceph-mon[29446]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:56:28.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:28.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:28.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:27 smithi086 ceph-mon[32965]: from='client.24964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:56:28.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:28.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:28.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:28.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:28.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:56:28.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:28.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:28.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:28.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:28.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:28.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:28.950 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:28.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:29.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:29.101 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:29.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:29.159 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:56:29.159 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:28 smithi006 ceph-mon[37501]: pgmap v329: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 35 MiB/s wr, 1.04k op/s; 25618/378972 objects degraded (6.760%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:56:29.159 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:28 smithi006 ceph-mon[29446]: pgmap v329: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 35 MiB/s wr, 1.04k op/s; 25618/378972 objects degraded (6.760%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:56:29.200 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:56:29.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:29.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:29.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:29.254 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:56:29.263 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-03-26T03:56:25.326399+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8264227264,"num_objects":126324,"num_object_clones":0,"num_object_copies":378972,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126324,"num_whiteouts":0,"num_read":1668,"num_read_kb":1482,"num_write":134571,"num_write_kb":8317854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3420,"num_bytes_recovered":214046967,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64934,"ondisk_log_size":64934,"up":444,"acting":435,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":24156568,"kb_used_data":22725604,"kb_used_omap":12,"kb_used_meta":1430515,"kb_avail":538314344,"statfs":{"total":575970213888,"available":551233888256,"internally_reserved":0,"allocated":23271018496,"data_stored":23267359384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12418,"internal_metadata":1464848254},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":96272408,"num_objects":1470,"num_object_clones":0,"num_object_copies":4410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1509,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1470,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":9140,"num_write_kb":339457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1509,"num_bytes_recovered":98893824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.034244"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":279,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472076+0000","last_change":"2024-03-26T03:55:59.300239+0000","last_active":"2024-03-26T03:56:22.472076+0000","last_peered":"2024-03-26T03:56:22.472076+0000","last_clean":"2024-03-26T03:56:22.472076+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T03:56:22.472076+0000","last_undegraded":"2024-03-26T03:56:22.472076+0000","last_fullsized":"2024-03-26T03:56:22.472076+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481480+0000","last_change":"2024-03-26T03:55:59.291583+0000","last_active":"2024-03-26T03:56:22.481480+0000","last_peered":"2024-03-26T03:56:22.481480+0000","last_clean":"2024-03-26T03:56:22.481480+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T03:56:22.481480+0000","last_undegraded":"2024-03-26T03:56:22.481480+0000","last_fullsized":"2024-03-26T03:56:22.481480+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493180+0000","last_change":"2024-03-26T03:55:59.307852+0000","last_active":"2024-03-26T03:56:22.493180+0000","last_peered":"2024-03-26T03:56:22.493180+0000","last_clean":"2024-03-26T03:56:22.493180+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T03:56:22.493180+0000","last_undegraded":"2024-03-26T03:56:22.493180+0000","last_fullsized":"2024-03-26T03:56:22.493180+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":350,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471721+0000","last_change":"2024-03-26T03:55:32.246310+0000","last_active":"2024-03-26T03:56:22.471721+0000","last_peered":"2024-03-26T03:56:22.471721+0000","last_clean":"2024-03-26T03:56:22.471721+0000","last_became_active":"2024-03-26T03:54:29.438463+0000","last_became_peered":"2024-03-26T03:54:29.438463+0000","last_unstale":"2024-03-26T03:56:22.471721+0000","last_undegraded":"2024-03-26T03:56:22.471721+0000","last_fullsized":"2024-03-26T03:56:22.471721+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:52.813656+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":162,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491572+0000","last_change":"2024-03-26T03:55:20.740072+0000","last_active":"2024-03-26T03:56:22.491572+0000","last_peered":"2024-03-26T03:56:22.491572+0000","last_clean":"2024-03-26T03:56:22.491572+0000","last_became_active":"2024-03-26T03:54:29.357724+0000","last_became_peered":"2024-03-26T03:54:29.357724+0000","last_unstale":"2024-03-26T03:56:22.491572+0000","last_undegraded":"2024-03-26T03:56:22.491572+0000","last_fullsized":"2024-03-26T03:56:22.491572+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:39.040158+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471400+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T03:56:22.471400+0000","last_peered":"2024-03-26T03:56:22.471400+0000","last_clean":"2024-03-26T03:56:22.471400+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:56:22.471400+0000","last_undegraded":"2024-03-26T03:56:22.471400+0000","last_fullsized":"2024-03-26T03:56:22.471400+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:14.310169+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":174,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.495299+0000","last_change":"2024-03-26T03:55:56.852293+0000","last_active":"2024-03-26T03:56:22.495299+0000","last_peered":"2024-03-26T03:56:22.495299+0000","last_clean":"2024-03-26T03:56:22.495299+0000","last_became_active":"2024-03-26T03:54:29.377365+0000","last_became_peered":"2024-03-26T03:54:29.377365+0000","last_unstale":"2024-03-26T03:56:22.495299+0000","last_undegraded":"2024-03-26T03:56:22.495299+0000","last_fullsized":"2024-03-26T03:56:22.495299+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:26.353468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":213,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.494538+0000","last_change":"2024-03-26T03:55:59.310364+0000","last_active":"2024-03-26T03:56:22.494538+0000","last_peered":"2024-03-26T03:56:22.494538+0000","last_clean":"2024-03-26T03:56:22.494538+0000","last_became_active":"2024-03-26T03:55:59.309970+0000","last_became_peered":"2024-03-26T03:55:59.309970+0000","last_unstale":"2024-03-26T03:56:22.494538+0000","last_undegraded":"2024-03-26T03:56:22.494538+0000","last_fullsized":"2024-03-26T03:56:22.494538+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.497214+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T03:56:22.497214+0000","last_peered":"2024-03-26T03:56:22.497214+0000","last_clean":"2024-03-26T03:56:22.497214+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:56:22.497214+0000","last_undegraded":"2024-03-26T03:56:22.497214+0000","last_fullsized":"2024-03-26T03:56:22.497214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:59.995323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":270,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321258+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T03:56:20.321258+0000","last_peered":"2024-03-26T03:56:20.321258+0000","last_clean":"2024-03-26T03:56:20.321258+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T03:56:20.321258+0000","last_undegraded":"2024-03-26T03:56:20.321258+0000","last_fullsized":"2024-03-26T03:56:20.321258+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:41.156708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":220,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321020+0000","last_change":"2024-03-26T03:55:59.304086+0000","last_active":"2024-03-26T03:56:20.321020+0000","last_peered":"2024-03-26T03:56:20.321020+0000","last_clean":"2024-03-26T03:56:20.321020+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T03:56:20.321020+0000","last_undegraded":"2024-03-26T03:56:20.321020+0000","last_fullsized":"2024-03-26T03:56:20.321020+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":347,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472729+0000","last_change":"2024-03-26T03:54:29.551157+0000","last_active":"2024-03-26T03:56:22.472729+0000","last_peered":"2024-03-26T03:56:22.472729+0000","last_clean":"2024-03-26T03:56:22.472729+0000","last_became_active":"2024-03-26T03:54:29.444189+0000","last_became_peered":"2024-03-26T03:54:29.444189+0000","last_unstale":"2024-03-26T03:56:22.472729+0000","last_undegraded":"2024-03-26T03:56:22.472729+0000","last_fullsized":"2024-03-26T03:56:22.472729+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":240,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320235+0000","last_change":"2024-03-26T03:55:59.298016+0000","last_active":"2024-03-26T03:56:20.320235+0000","last_peered":"2024-03-26T03:56:20.320235+0000","last_clean":"2024-03-26T03:56:20.320235+0000","last_became_active":"2024-03-26T03:55:59.297675+0000","last_became_peered":"2024-03-26T03:55:59.297675+0000","last_unstale":"2024-03-26T03:56:20.320235+0000","last_undegraded":"2024-03-26T03:56:20.320235+0000","last_fullsized":"2024-03-26T03:56:20.320235+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491248+0000","last_change":"2024-03-26T03:54:11.544712+0000","last_active":"2024-03-26T03:56:22.491248+0000","last_peered":"2024-03-26T03:56:22.491248+0000","last_clean":"2024-03-26T03:56:22.491248+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:56:22.491248+0000","last_undegraded":"2024-03-26T03:56:22.491248+0000","last_fullsized":"2024-03-26T03:56:22.491248+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.544637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.544637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":245,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320473+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T03:56:20.320473+0000","last_peered":"2024-03-26T03:56:20.320473+0000","last_clean":"2024-03-26T03:56:20.320473+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T03:56:20.320473+0000","last_undegraded":"2024-03-26T03:56:20.320473+0000","last_fullsized":"2024-03-26T03:56:20.320473+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":325,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472200+0000","last_change":"2024-03-26T03:55:18.237943+0000","last_active":"2024-03-26T03:56:22.472200+0000","last_peered":"2024-03-26T03:56:22.472200+0000","last_clean":"2024-03-26T03:56:22.472200+0000","last_became_active":"2024-03-26T03:54:29.360309+0000","last_became_peered":"2024-03-26T03:54:29.360309+0000","last_unstale":"2024-03-26T03:56:22.472200+0000","last_undegraded":"2024-03-26T03:56:22.472200+0000","last_fullsized":"2024-03-26T03:56:22.472200+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:42.185581+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472432+0000","last_change":"2024-03-26T03:55:59.295858+0000","last_active":"2024-03-26T03:56:22.472432+0000","last_peered":"2024-03-26T03:56:22.472432+0000","last_clean":"2024-03-26T03:56:22.472432+0000","last_became_active":"2024-03-26T03:55:59.295544+0000","last_became_peered":"2024-03-26T03:55:59.295544+0000","last_unstale":"2024-03-26T03:56:22.472432+0000","last_undegraded":"2024-03-26T03:56:22.472432+0000","last_fullsized":"2024-03-26T03:56:22.472432+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.482753+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T03:56:22.482753+0000","last_peered":"2024-03-26T03:56:22.482753+0000","last_clean":"2024-03-26T03:56:22.482753+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:56:22.482753+0000","last_undegraded":"2024-03-26T03:56:22.482753+0000","last_fullsized":"2024-03-26T03:56:22.482753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:48.944214+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.492234+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T03:56:22.492234+0000","last_peered":"2024-03-26T03:56:22.492234+0000","last_clean":"2024-03-26T03:56:22.492234+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:56:22.492234+0000","last_undegraded":"2024-03-26T03:56:22.492234+0000","last_fullsized":"2024-03-26T03:56:22.492234+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:05.710428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":349,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.501807+0000","last_change":"2024-03-26T03:55:30.771720+0000","last_active":"2024-03-26T03:56:22.501807+0000","last_peered":"2024-03-26T03:56:22.501807+0000","last_clean":"2024-03-26T03:56:22.501807+0000","last_became_active":"2024-03-26T03:54:29.346374+0000","last_became_peered":"2024-03-26T03:54:29.346374+0000","last_unstale":"2024-03-26T03:56:22.501807+0000","last_undegraded":"2024-03-26T03:56:22.501807+0000","last_fullsized":"2024-03-26T03:56:22.501807+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:30.771573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.771573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:51.100616+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":110,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321485+0000","last_change":"2024-03-26T03:55:59.320775+0000","last_active":"2024-03-26T03:56:20.321485+0000","last_peered":"2024-03-26T03:56:20.321485+0000","last_clean":"2024-03-26T03:56:20.321485+0000","last_became_active":"2024-03-26T03:55:59.318416+0000","last_became_peered":"2024-03-26T03:55:59.318416+0000","last_unstale":"2024-03-26T03:56:20.321485+0000","last_undegraded":"2024-03-26T03:56:20.321485+0000","last_fullsized":"2024-03-26T03:56:20.321485+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472048+0000","last_change":"2024-03-26T03:54:29.360637+0000","last_active":"2024-03-26T03:56:22.472048+0000","last_peered":"2024-03-26T03:56:22.472048+0000","last_clean":"2024-03-26T03:56:22.472048+0000","last_became_active":"2024-03-26T03:54:29.360289+0000","last_became_peered":"2024-03-26T03:54:29.360289+0000","last_unstale":"2024-03-26T03:56:22.472048+0000","last_undegraded":"2024-03-26T03:56:22.472048+0000","last_fullsized":"2024-03-26T03:56:22.472048+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472823+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:56:22.472823+0000","last_peered":"2024-03-26T03:56:22.472823+0000","last_clean":"2024-03-26T03:56:22.472823+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:56:22.472823+0000","last_undegraded":"2024-03-26T03:56:22.472823+0000","last_fullsized":"2024-03-26T03:56:22.472823+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":165,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.482247+0000","last_change":"2024-03-26T03:55:59.312779+0000","last_active":"2024-03-26T03:56:22.482247+0000","last_peered":"2024-03-26T03:56:22.482247+0000","last_clean":"2024-03-26T03:56:22.482247+0000","last_became_active":"2024-03-26T03:55:59.312386+0000","last_became_peered":"2024-03-26T03:55:59.312386+0000","last_unstale":"2024-03-26T03:56:22.482247+0000","last_undegraded":"2024-03-26T03:56:22.482247+0000","last_fullsized":"2024-03-26T03:56:22.482247+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.552551+0000","last_change":"2024-03-26T03:55:59.300737+0000","last_active":"2024-03-26T03:56:22.552551+0000","last_peered":"2024-03-26T03:56:22.552551+0000","last_clean":"2024-03-26T03:56:22.552551+0000","last_became_active":"2024-03-26T03:55:59.299335+0000","last_became_peered":"2024-03-26T03:55:59.299335+0000","last_unstale":"2024-03-26T03:56:22.552551+0000","last_undegraded":"2024-03-26T03:56:22.552551+0000","last_fullsized":"2024-03-26T03:56:22.552551+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:54:15.688518+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.688518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":323,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472877+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T03:56:22.472877+0000","last_peered":"2024-03-26T03:56:22.472877+0000","last_clean":"2024-03-26T03:56:22.472877+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:56:22.472877+0000","last_undegraded":"2024-03-26T03:56:22.472877+0000","last_fullsized":"2024-03-26T03:56:22.472877+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:25.265208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481390+0000","last_change":"2024-03-26T03:55:20.749859+0000","last_active":"2024-03-26T03:56:22.481390+0000","last_peered":"2024-03-26T03:56:22.481390+0000","last_clean":"2024-03-26T03:56:22.481390+0000","last_became_active":"2024-03-26T03:54:29.324000+0000","last_became_peered":"2024-03-26T03:54:29.324000+0000","last_unstale":"2024-03-26T03:56:22.481390+0000","last_undegraded":"2024-03-26T03:56:22.481390+0000","last_fullsized":"2024-03-26T03:56:22.481390+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:55:20.749583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.749583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:27.421402+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":333,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493196+0000","last_change":"2024-03-26T03:54:36.814787+0000","last_active":"2024-03-26T03:56:22.493196+0000","last_peered":"2024-03-26T03:56:22.493196+0000","last_clean":"2024-03-26T03:56:22.493196+0000","last_became_active":"2024-03-26T03:54:29.521665+0000","last_became_peered":"2024-03-26T03:54:29.521665+0000","last_unstale":"2024-03-26T03:56:22.493196+0000","last_undegraded":"2024-03-26T03:56:22.493196+0000","last_fullsized":"2024-03-26T03:56:22.493196+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.482171+0000","last_change":"2024-03-26T03:55:59.343699+0000","last_active":"2024-03-26T03:56:22.482171+0000","last_peered":"2024-03-26T03:56:22.482171+0000","last_clean":"2024-03-26T03:56:22.482171+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T03:56:22.482171+0000","last_undegraded":"2024-03-26T03:56:22.482171+0000","last_fullsized":"2024-03-26T03:56:22.482171+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493436+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:56:22.493436+0000","last_peered":"2024-03-26T03:56:22.493436+0000","last_clean":"2024-03-26T03:56:22.493436+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:56:22.493436+0000","last_undegraded":"2024-03-26T03:56:22.493436+0000","last_fullsized":"2024-03-26T03:56:22.493436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":96,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.497138+0000","last_change":"2024-03-26T03:54:31.825491+0000","last_active":"2024-03-26T03:56:22.497138+0000","last_peered":"2024-03-26T03:56:22.497138+0000","last_clean":"2024-03-26T03:56:22.497138+0000","last_became_active":"2024-03-26T03:54:29.453692+0000","last_became_peered":"2024-03-26T03:54:29.453692+0000","last_unstale":"2024-03-26T03:56:22.497138+0000","last_undegraded":"2024-03-26T03:56:22.497138+0000","last_fullsized":"2024-03-26T03:56:22.497138+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":274,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321039+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T03:56:20.321039+0000","last_peered":"2024-03-26T03:56:20.321039+0000","last_clean":"2024-03-26T03:56:20.321039+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T03:56:20.321039+0000","last_undegraded":"2024-03-26T03:56:20.321039+0000","last_fullsized":"2024-03-26T03:56:20.321039+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:30.023866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491335+0000","last_change":"2024-03-26T03:54:19.645849+0000","last_active":"2024-03-26T03:56:22.491335+0000","last_peered":"2024-03-26T03:56:22.491335+0000","last_clean":"2024-03-26T03:56:22.491335+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:56:22.491335+0000","last_undegraded":"2024-03-26T03:56:22.491335+0000","last_fullsized":"2024-03-26T03:56:22.491335+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:19.645746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.645746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":220,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471386+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:56:22.471386+0000","last_peered":"2024-03-26T03:56:22.471386+0000","last_clean":"2024-03-26T03:56:22.471386+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:56:22.471386+0000","last_undegraded":"2024-03-26T03:56:22.471386+0000","last_fullsized":"2024-03-26T03:56:22.471386+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":305,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493372+0000","last_change":"2024-03-26T03:55:59.307951+0000","last_active":"2024-03-26T03:56:22.493372+0000","last_peered":"2024-03-26T03:56:22.493372+0000","last_clean":"2024-03-26T03:56:22.493372+0000","last_became_active":"2024-03-26T03:55:59.307490+0000","last_became_peered":"2024-03-26T03:55:59.307490+0000","last_unstale":"2024-03-26T03:56:22.493372+0000","last_undegraded":"2024-03-26T03:56:22.493372+0000","last_fullsized":"2024-03-26T03:56:22.493372+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":289,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.483517+0000","last_change":"2024-03-26T03:55:59.299317+0000","last_active":"2024-03-26T03:56:22.483517+0000","last_peered":"2024-03-26T03:56:22.483517+0000","last_clean":"2024-03-26T03:56:22.483517+0000","last_became_active":"2024-03-26T03:55:59.298906+0000","last_became_peered":"2024-03-26T03:55:59.298906+0000","last_unstale":"2024-03-26T03:56:22.483517+0000","last_undegraded":"2024-03-26T03:56:22.483517+0000","last_fullsized":"2024-03-26T03:56:22.483517+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":162,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491223+0000","last_change":"2024-03-26T03:55:29.312054+0000","last_active":"2024-03-26T03:56:22.491223+0000","last_peered":"2024-03-26T03:56:22.491223+0000","last_clean":"2024-03-26T03:56:22.491223+0000","last_became_active":"2024-03-26T03:54:29.361440+0000","last_became_peered":"2024-03-26T03:54:29.361440+0000","last_unstale":"2024-03-26T03:56:22.491223+0000","last_undegraded":"2024-03-26T03:56:22.491223+0000","last_fullsized":"2024-03-26T03:56:22.491223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:30.202957+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490887+0000","last_change":"2024-03-26T03:55:25.766015+0000","last_active":"2024-03-26T03:56:22.490887+0000","last_peered":"2024-03-26T03:56:22.490887+0000","last_clean":"2024-03-26T03:56:22.490887+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:56:22.490887+0000","last_undegraded":"2024-03-26T03:56:22.490887+0000","last_fullsized":"2024-03-26T03:56:22.490887+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.765904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.765904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:26.508267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":253,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320464+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T03:56:20.320464+0000","last_peered":"2024-03-26T03:56:20.320464+0000","last_clean":"2024-03-26T03:56:20.320464+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T03:56:20.320464+0000","last_undegraded":"2024-03-26T03:56:20.320464+0000","last_fullsized":"2024-03-26T03:56:20.320464+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":198,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491491+0000","last_change":"2024-03-26T03:55:59.303323+0000","last_active":"2024-03-26T03:56:22.491491+0000","last_peered":"2024-03-26T03:56:22.491491+0000","last_clean":"2024-03-26T03:56:22.491491+0000","last_became_active":"2024-03-26T03:55:59.302965+0000","last_became_peered":"2024-03-26T03:55:59.302965+0000","last_unstale":"2024-03-26T03:56:22.491491+0000","last_undegraded":"2024-03-26T03:56:22.491491+0000","last_fullsized":"2024-03-26T03:56:22.491491+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.496774+0000","last_change":"2024-03-26T03:55:59.324307+0000","last_active":"2024-03-26T03:56:22.496774+0000","last_peered":"2024-03-26T03:56:22.496774+0000","last_clean":"2024-03-26T03:56:22.496774+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T03:56:22.496774+0000","last_undegraded":"2024-03-26T03:56:22.496774+0000","last_fullsized":"2024-03-26T03:56:22.496774+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":105,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491315+0000","last_change":"2024-03-26T03:56:00.569951+0000","last_active":"2024-03-26T03:56:22.491315+0000","last_peered":"2024-03-26T03:56:22.491315+0000","last_clean":"2024-03-26T03:56:22.491315+0000","last_became_active":"2024-03-26T03:55:59.308759+0000","last_became_peered":"2024-03-26T03:55:59.308759+0000","last_unstale":"2024-03-26T03:56:22.491315+0000","last_undegraded":"2024-03-26T03:56:22.491315+0000","last_fullsized":"2024-03-26T03:56:22.491315+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:12.010641+0000","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551071+0000","last_change":"2024-03-26T03:54:29.530213+0000","last_active":"2024-03-26T03:56:22.551071+0000","last_peered":"2024-03-26T03:56:22.551071+0000","last_clean":"2024-03-26T03:56:22.551071+0000","last_became_active":"2024-03-26T03:54:29.529851+0000","last_became_peered":"2024-03-26T03:54:29.529851+0000","last_unstale":"2024-03-26T03:56:22.551071+0000","last_undegraded":"2024-03-26T03:56:22.551071+0000","last_fullsized":"2024-03-26T03:56:22.551071+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"152'13","reported_seq":329,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.486573+0000","last_change":"2024-03-26T03:55:30.839253+0000","last_active":"2024-03-26T03:56:22.486573+0000","last_peered":"2024-03-26T03:56:22.486573+0000","last_clean":"2024-03-26T03:56:22.486573+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:56:22.486573+0000","last_undegraded":"2024-03-26T03:56:22.486573+0000","last_fullsized":"2024-03-26T03:56:22.486573+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:55:30.839058+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.839058+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:38.794343+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.502516+0000","last_change":"2024-03-26T03:55:59.311115+0000","last_active":"2024-03-26T03:56:22.502516+0000","last_peered":"2024-03-26T03:56:22.502516+0000","last_clean":"2024-03-26T03:56:22.502516+0000","last_became_active":"2024-03-26T03:55:59.310758+0000","last_became_peered":"2024-03-26T03:55:59.310758+0000","last_unstale":"2024-03-26T03:56:22.502516+0000","last_undegraded":"2024-03-26T03:56:22.502516+0000","last_fullsized":"2024-03-26T03:56:22.502516+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.804652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.486395+0000","last_change":"2024-03-26T03:55:59.302276+0000","last_active":"2024-03-26T03:56:22.486395+0000","last_peered":"2024-03-26T03:56:22.486395+0000","last_clean":"2024-03-26T03:56:22.486395+0000","last_became_active":"2024-03-26T03:55:59.299889+0000","last_became_peered":"2024-03-26T03:55:59.299889+0000","last_unstale":"2024-03-26T03:56:22.486395+0000","last_undegraded":"2024-03-26T03:56:22.486395+0000","last_fullsized":"2024-03-26T03:56:22.486395+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471875+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:56:22.471875+0000","last_peered":"2024-03-26T03:56:22.471875+0000","last_clean":"2024-03-26T03:56:22.471875+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:56:22.471875+0000","last_undegraded":"2024-03-26T03:56:22.471875+0000","last_fullsized":"2024-03-26T03:56:22.471875+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":324,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472453+0000","last_change":"2024-03-26T03:55:59.300136+0000","last_active":"2024-03-26T03:56:22.472453+0000","last_peered":"2024-03-26T03:56:22.472453+0000","last_clean":"2024-03-26T03:56:22.472453+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T03:56:22.472453+0000","last_undegraded":"2024-03-26T03:56:22.472453+0000","last_fullsized":"2024-03-26T03:56:22.472453+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":276,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.492884+0000","last_change":"2024-03-26T03:54:29.521980+0000","last_active":"2024-03-26T03:56:22.492884+0000","last_peered":"2024-03-26T03:56:22.492884+0000","last_clean":"2024-03-26T03:56:22.492884+0000","last_became_active":"2024-03-26T03:54:29.520018+0000","last_became_peered":"2024-03-26T03:54:29.520018+0000","last_unstale":"2024-03-26T03:56:22.492884+0000","last_undegraded":"2024-03-26T03:56:22.492884+0000","last_fullsized":"2024-03-26T03:56:22.492884+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":361,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320037+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T03:56:20.320037+0000","last_peered":"2024-03-26T03:56:20.320037+0000","last_clean":"2024-03-26T03:56:20.320037+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T03:56:20.320037+0000","last_undegraded":"2024-03-26T03:56:20.320037+0000","last_fullsized":"2024-03-26T03:56:20.320037+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:31.627476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.550564+0000","last_change":"2024-03-26T03:55:57.916905+0000","last_active":"2024-03-26T03:56:22.550564+0000","last_peered":"2024-03-26T03:56:22.550564+0000","last_clean":"2024-03-26T03:56:22.550564+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:56:22.550564+0000","last_undegraded":"2024-03-26T03:56:22.550564+0000","last_fullsized":"2024-03-26T03:56:22.550564+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.916786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.916786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:27.322837+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":170,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481951+0000","last_change":"2024-03-26T03:54:32.911388+0000","last_active":"2024-03-26T03:56:22.481951+0000","last_peered":"2024-03-26T03:56:22.481951+0000","last_clean":"2024-03-26T03:56:22.481951+0000","last_became_active":"2024-03-26T03:54:29.390009+0000","last_became_peered":"2024-03-26T03:54:29.390009+0000","last_unstale":"2024-03-26T03:56:22.481951+0000","last_undegraded":"2024-03-26T03:56:22.481951+0000","last_fullsized":"2024-03-26T03:56:22.481951+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":287,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472938+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T03:56:22.472938+0000","last_peered":"2024-03-26T03:56:22.472938+0000","last_clean":"2024-03-26T03:56:22.472938+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T03:56:22.472938+0000","last_undegraded":"2024-03-26T03:56:22.472938+0000","last_fullsized":"2024-03-26T03:56:22.472938+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":149,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.552493+0000","last_change":"2024-03-26T03:55:13.805008+0000","last_active":"2024-03-26T03:56:22.552493+0000","last_peered":"2024-03-26T03:56:22.552493+0000","last_clean":"2024-03-26T03:56:22.552493+0000","last_became_active":"2024-03-26T03:54:29.389636+0000","last_became_peered":"2024-03-26T03:54:29.389636+0000","last_unstale":"2024-03-26T03:56:22.552493+0000","last_undegraded":"2024-03-26T03:56:22.552493+0000","last_fullsized":"2024-03-26T03:56:22.552493+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:34.068948+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":316,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491349+0000","last_change":"2024-03-26T03:55:53.234770+0000","last_active":"2024-03-26T03:56:22.491349+0000","last_peered":"2024-03-26T03:56:22.491349+0000","last_clean":"2024-03-26T03:56:22.491349+0000","last_became_active":"2024-03-26T03:54:29.345136+0000","last_became_peered":"2024-03-26T03:54:29.345136+0000","last_unstale":"2024-03-26T03:56:22.491349+0000","last_undegraded":"2024-03-26T03:56:22.491349+0000","last_fullsized":"2024-03-26T03:56:22.491349+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:03.979602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":293,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320855+0000","last_change":"2024-03-26T03:55:59.342176+0000","last_active":"2024-03-26T03:56:20.320855+0000","last_peered":"2024-03-26T03:56:20.320855+0000","last_clean":"2024-03-26T03:56:20.320855+0000","last_became_active":"2024-03-26T03:55:59.340772+0000","last_became_peered":"2024-03-26T03:55:59.340772+0000","last_unstale":"2024-03-26T03:56:20.320855+0000","last_undegraded":"2024-03-26T03:56:20.320855+0000","last_fullsized":"2024-03-26T03:56:20.320855+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.502127+0000","last_change":"2024-03-26T03:54:36.963991+0000","last_active":"2024-03-26T03:56:22.502127+0000","last_peered":"2024-03-26T03:56:22.502127+0000","last_clean":"2024-03-26T03:56:22.502127+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:56:22.502127+0000","last_undegraded":"2024-03-26T03:56:22.502127+0000","last_fullsized":"2024-03-26T03:56:22.502127+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:54:36.963833+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.963833+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":96,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491033+0000","last_change":"2024-03-26T03:55:19.712372+0000","last_active":"2024-03-26T03:56:22.491033+0000","last_peered":"2024-03-26T03:56:22.491033+0000","last_clean":"2024-03-26T03:56:22.491033+0000","last_became_active":"2024-03-26T03:54:29.532138+0000","last_became_peered":"2024-03-26T03:54:29.532138+0000","last_unstale":"2024-03-26T03:56:22.491033+0000","last_undegraded":"2024-03-26T03:56:22.491033+0000","last_fullsized":"2024-03-26T03:56:22.491033+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:28.746531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491424+0000","last_change":"2024-03-26T03:53:29.687145+0000","last_active":"2024-03-26T03:56:22.491424+0000","last_peered":"2024-03-26T03:56:22.491424+0000","last_clean":"2024-03-26T03:56:22.491424+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:56:22.491424+0000","last_undegraded":"2024-03-26T03:56:22.491424+0000","last_fullsized":"2024-03-26T03:56:22.491424+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.687067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.687067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":346,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493292+0000","last_change":"2024-03-26T03:55:25.160341+0000","last_active":"2024-03-26T03:56:22.493292+0000","last_peered":"2024-03-26T03:56:22.493292+0000","last_clean":"2024-03-26T03:56:22.493292+0000","last_became_active":"2024-03-26T03:54:29.521088+0000","last_became_peered":"2024-03-26T03:54:29.521088+0000","last_unstale":"2024-03-26T03:56:22.493292+0000","last_undegraded":"2024-03-26T03:56:22.493292+0000","last_fullsized":"2024-03-26T03:56:22.493292+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:31.956440+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.550966+0000","last_change":"2024-03-26T03:55:59.300900+0000","last_active":"2024-03-26T03:56:22.550966+0000","last_peered":"2024-03-26T03:56:22.550966+0000","last_clean":"2024-03-26T03:56:22.550966+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T03:56:22.550966+0000","last_undegraded":"2024-03-26T03:56:22.550966+0000","last_fullsized":"2024-03-26T03:56:22.550966+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":244,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320625+0000","last_change":"2024-03-26T03:55:59.305418+0000","last_active":"2024-03-26T03:56:20.320625+0000","last_peered":"2024-03-26T03:56:20.320625+0000","last_clean":"2024-03-26T03:56:20.320625+0000","last_became_active":"2024-03-26T03:55:59.304158+0000","last_became_peered":"2024-03-26T03:55:59.304158+0000","last_unstale":"2024-03-26T03:56:20.320625+0000","last_undegraded":"2024-03-26T03:56:20.320625+0000","last_fullsized":"2024-03-26T03:56:20.320625+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":235,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.497062+0000","last_change":"2024-03-26T03:55:59.325776+0000","last_active":"2024-03-26T03:56:22.497062+0000","last_peered":"2024-03-26T03:56:22.497062+0000","last_clean":"2024-03-26T03:56:22.497062+0000","last_became_active":"2024-03-26T03:55:59.325344+0000","last_became_peered":"2024-03-26T03:55:59.325344+0000","last_unstale":"2024-03-26T03:56:22.497062+0000","last_undegraded":"2024-03-26T03:56:22.497062+0000","last_fullsized":"2024-03-26T03:56:22.497062+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":147,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551420+0000","last_change":"2024-03-26T03:55:19.697165+0000","last_active":"2024-03-26T03:56:22.551420+0000","last_peered":"2024-03-26T03:56:22.551420+0000","last_clean":"2024-03-26T03:56:22.551420+0000","last_became_active":"2024-03-26T03:54:29.289163+0000","last_became_peered":"2024-03-26T03:54:29.289163+0000","last_unstale":"2024-03-26T03:56:22.551420+0000","last_undegraded":"2024-03-26T03:56:22.551420+0000","last_fullsized":"2024-03-26T03:56:22.551420+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:40.231867+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"192'8377","reported_seq":10456,"reported_epoch":192,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.658612+0000","last_change":"2024-03-26T03:56:00.576156+0000","last_active":"2024-03-26T03:56:24.658612+0000","last_peered":"2024-03-26T03:56:24.658612+0000","last_clean":"2024-03-26T03:55:19.255434+0000","last_became_active":"2024-03-26T03:56:00.557563+0000","last_became_peered":"2024-03-26T03:56:00.557563+0000","last_unstale":"2024-03-26T03:56:24.658612+0000","last_undegraded":"2024-03-26T03:56:00.310379+0000","last_fullsized":"2024-03-26T03:56:00.281761+0000","mapping_epoch":181,"log_start":"169'3800","ondisk_log_start":"169'3800","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4577,"log_dups_size":0,"ondisk_log_size":4577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":518586368,"num_objects":7913,"num_object_clones":0,"num_object_copies":23739,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2212,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7913,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8377,"num_write_kb":521280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":995,"num_bytes_recovered":65208320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[2,4],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":5701}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":237,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321819+0000","last_change":"2024-03-26T03:55:59.325043+0000","last_active":"2024-03-26T03:56:20.321819+0000","last_peered":"2024-03-26T03:56:20.321819+0000","last_clean":"2024-03-26T03:56:20.321819+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T03:56:20.321819+0000","last_undegraded":"2024-03-26T03:56:20.321819+0000","last_fullsized":"2024-03-26T03:56:20.321819+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":302,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551321+0000","last_change":"2024-03-26T03:54:16.696733+0000","last_active":"2024-03-26T03:56:22.551321+0000","last_peered":"2024-03-26T03:56:22.551321+0000","last_clean":"2024-03-26T03:56:22.551321+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:56:22.551321+0000","last_undegraded":"2024-03-26T03:56:22.551321+0000","last_fullsized":"2024-03-26T03:56:22.551321+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:54:16.696602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.696602+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":310,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321136+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T03:56:20.321136+0000","last_peered":"2024-03-26T03:56:20.321136+0000","last_clean":"2024-03-26T03:56:20.321136+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T03:56:20.321136+0000","last_undegraded":"2024-03-26T03:56:20.321136+0000","last_fullsized":"2024-03-26T03:56:20.321136+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":401,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491557+0000","last_change":"2024-03-26T03:55:52.900155+0000","last_active":"2024-03-26T03:56:22.491557+0000","last_peered":"2024-03-26T03:56:22.491557+0000","last_clean":"2024-03-26T03:56:22.491557+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:56:22.491557+0000","last_undegraded":"2024-03-26T03:56:22.491557+0000","last_fullsized":"2024-03-26T03:56:22.491557+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:52.900040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:52.900040+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:06.366979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"192'8311","reported_seq":8395,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.255617+0000","last_change":"2024-03-26T03:54:38.786252+0000","last_active":"2024-03-26T03:56:24.255617+0000","last_peered":"2024-03-26T03:56:24.255617+0000","last_clean":"2024-03-26T03:56:24.255617+0000","last_became_active":"2024-03-26T03:54:38.785773+0000","last_became_peered":"2024-03-26T03:54:38.785773+0000","last_unstale":"2024-03-26T03:56:24.255617+0000","last_undegraded":"2024-03-26T03:56:24.255617+0000","last_fullsized":"2024-03-26T03:56:24.255617+0000","mapping_epoch":155,"log_start":"170'4300","ondisk_log_start":"170'4300","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4011,"log_dups_size":0,"ondisk_log_size":4011,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513998848,"num_objects":7843,"num_object_clones":0,"num_object_copies":23529,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7843,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8311,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":97,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.550561+0000","last_change":"2024-03-26T03:54:30.757009+0000","last_active":"2024-03-26T03:56:22.550561+0000","last_peered":"2024-03-26T03:56:22.550561+0000","last_clean":"2024-03-26T03:56:22.550561+0000","last_became_active":"2024-03-26T03:54:29.304565+0000","last_became_peered":"2024-03-26T03:54:29.304565+0000","last_unstale":"2024-03-26T03:56:22.550561+0000","last_undegraded":"2024-03-26T03:56:22.550561+0000","last_fullsized":"2024-03-26T03:56:22.550561+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551938+0000","last_change":"2024-03-26T03:54:18.653671+0000","last_active":"2024-03-26T03:56:22.551938+0000","last_peered":"2024-03-26T03:56:22.551938+0000","last_clean":"2024-03-26T03:56:22.551938+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:56:22.551938+0000","last_undegraded":"2024-03-26T03:56:22.551938+0000","last_fullsized":"2024-03-26T03:56:22.551938+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.653555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.653555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":355,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471954+0000","last_change":"2024-03-26T03:55:14.285475+0000","last_active":"2024-03-26T03:56:22.471954+0000","last_peered":"2024-03-26T03:56:22.471954+0000","last_clean":"2024-03-26T03:56:22.471954+0000","last_became_active":"2024-03-26T03:54:29.346977+0000","last_became_peered":"2024-03-26T03:54:29.346977+0000","last_unstale":"2024-03-26T03:56:22.471954+0000","last_undegraded":"2024-03-26T03:56:22.471954+0000","last_fullsized":"2024-03-26T03:56:22.471954+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.494269+0000","last_change":"2024-03-26T03:55:19.828552+0000","last_active":"2024-03-26T03:56:22.494269+0000","last_peered":"2024-03-26T03:56:22.494269+0000","last_clean":"2024-03-26T03:56:22.494269+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:56:22.494269+0000","last_undegraded":"2024-03-26T03:56:22.494269+0000","last_fullsized":"2024-03-26T03:56:22.494269+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.828444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.828444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:44.125561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"192'8235","reported_seq":8319,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.840647+0000","last_change":"2024-03-26T03:54:38.785847+0000","last_active":"2024-03-26T03:56:24.840647+0000","last_peered":"2024-03-26T03:56:24.840647+0000","last_clean":"2024-03-26T03:56:24.840647+0000","last_became_active":"2024-03-26T03:54:38.785420+0000","last_became_peered":"2024-03-26T03:54:38.785420+0000","last_unstale":"2024-03-26T03:56:24.840647+0000","last_undegraded":"2024-03-26T03:56:24.840647+0000","last_fullsized":"2024-03-26T03:56:24.840647+0000","mapping_epoch":155,"log_start":"170'4200","ondisk_log_start":"170'4200","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4035,"log_dups_size":0,"ondisk_log_size":4035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509542400,"num_objects":7775,"num_object_clones":0,"num_object_copies":23325,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7775,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8235,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490447+0000","last_change":"2024-03-26T03:54:29.359240+0000","last_active":"2024-03-26T03:56:22.490447+0000","last_peered":"2024-03-26T03:56:22.490447+0000","last_clean":"2024-03-26T03:56:22.490447+0000","last_became_active":"2024-03-26T03:54:29.358827+0000","last_became_peered":"2024-03-26T03:54:29.358827+0000","last_unstale":"2024-03-26T03:56:22.490447+0000","last_undegraded":"2024-03-26T03:56:22.490447+0000","last_fullsized":"2024-03-26T03:56:22.490447+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.664206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.664206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.499551+0000","last_change":"2024-03-26T03:55:59.304842+0000","last_active":"2024-03-26T03:56:22.499551+0000","last_peered":"2024-03-26T03:56:22.499551+0000","last_clean":"2024-03-26T03:56:22.499551+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T03:56:22.499551+0000","last_undegraded":"2024-03-26T03:56:22.499551+0000","last_fullsized":"2024-03-26T03:56:22.499551+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.876322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.876322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":311,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551260+0000","last_change":"2024-03-26T03:55:59.299719+0000","last_active":"2024-03-26T03:56:22.551260+0000","last_peered":"2024-03-26T03:56:22.551260+0000","last_clean":"2024-03-26T03:56:22.551260+0000","last_became_active":"2024-03-26T03:55:59.299160+0000","last_became_peered":"2024-03-26T03:55:59.299160+0000","last_unstale":"2024-03-26T03:56:22.551260+0000","last_undegraded":"2024-03-26T03:56:22.551260+0000","last_fullsized":"2024-03-26T03:56:22.551260+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.612556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.612556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.486252+0000","last_change":"2024-03-26T03:55:59.302041+0000","last_active":"2024-03-26T03:56:22.486252+0000","last_peered":"2024-03-26T03:56:22.486252+0000","last_clean":"2024-03-26T03:56:22.486252+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T03:56:22.486252+0000","last_undegraded":"2024-03-26T03:56:22.486252+0000","last_fullsized":"2024-03-26T03:56:22.486252+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:55:14.773103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773103+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:25.583025+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"192'8292","reported_seq":8376,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.749952+0000","last_change":"2024-03-26T03:54:38.382909+0000","last_active":"2024-03-26T03:56:24.749952+0000","last_peered":"2024-03-26T03:56:24.749952+0000","last_clean":"2024-03-26T03:56:24.749952+0000","last_became_active":"2024-03-26T03:54:38.382277+0000","last_became_peered":"2024-03-26T03:54:38.382277+0000","last_unstale":"2024-03-26T03:56:24.749952+0000","last_undegraded":"2024-03-26T03:56:24.749952+0000","last_fullsized":"2024-03-26T03:56:24.749952+0000","mapping_epoch":155,"log_start":"170'4700","ondisk_log_start":"170'4700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":512491520,"num_objects":7820,"num_object_clones":0,"num_object_copies":23460,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7820,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8292,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.503080+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T03:56:22.503080+0000","last_peered":"2024-03-26T03:56:22.503080+0000","last_clean":"2024-03-26T03:56:22.503080+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:56:22.503080+0000","last_undegraded":"2024-03-26T03:56:22.503080+0000","last_fullsized":"2024-03-26T03:56:22.503080+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:27.647080+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.495269+0000","last_change":"2024-03-26T03:54:33.962451+0000","last_active":"2024-03-26T03:56:22.495269+0000","last_peered":"2024-03-26T03:56:22.495269+0000","last_clean":"2024-03-26T03:56:22.495269+0000","last_became_active":"2024-03-26T03:54:29.428035+0000","last_became_peered":"2024-03-26T03:54:29.428035+0000","last_unstale":"2024-03-26T03:56:22.495269+0000","last_undegraded":"2024-03-26T03:56:22.495269+0000","last_fullsized":"2024-03-26T03:56:22.495269+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.962364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":266,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320450+0000","last_change":"2024-03-26T03:55:59.341967+0000","last_active":"2024-03-26T03:56:20.320450+0000","last_peered":"2024-03-26T03:56:20.320450+0000","last_clean":"2024-03-26T03:56:20.320450+0000","last_became_active":"2024-03-26T03:55:59.341062+0000","last_became_peered":"2024-03-26T03:55:59.341062+0000","last_unstale":"2024-03-26T03:56:20.320450+0000","last_undegraded":"2024-03-26T03:56:20.320450+0000","last_fullsized":"2024-03-26T03:56:20.320450+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":341,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.492503+0000","last_change":"2024-03-26T03:55:59.309207+0000","last_active":"2024-03-26T03:56:22.492503+0000","last_peered":"2024-03-26T03:56:22.492503+0000","last_clean":"2024-03-26T03:56:22.492503+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T03:56:22.492503+0000","last_undegraded":"2024-03-26T03:56:22.492503+0000","last_fullsized":"2024-03-26T03:56:22.492503+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"192'8222","reported_seq":8311,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.844853+0000","last_change":"2024-03-26T03:56:00.577986+0000","last_active":"2024-03-26T03:56:24.844853+0000","last_peered":"2024-03-26T03:56:24.844853+0000","last_clean":"2024-03-26T03:55:19.234210+0000","last_became_active":"2024-03-26T03:56:00.572420+0000","last_became_peered":"2024-03-26T03:56:00.572420+0000","last_unstale":"2024-03-26T03:56:24.844853+0000","last_undegraded":"2024-03-26T03:56:00.323147+0000","last_fullsized":"2024-03-26T03:56:00.288259+0000","mapping_epoch":181,"log_start":"170'4177","ondisk_log_start":"170'4177","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4045,"log_dups_size":0,"ondisk_log_size":4045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506593280,"num_objects":7730,"num_object_clones":0,"num_object_copies":23190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3168,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8222,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":4562}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":106,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472065+0000","last_change":"2024-03-26T03:55:59.289528+0000","last_active":"2024-03-26T03:56:22.472065+0000","last_peered":"2024-03-26T03:56:22.472065+0000","last_clean":"2024-03-26T03:56:22.472065+0000","last_became_active":"2024-03-26T03:55:59.288910+0000","last_became_peered":"2024-03-26T03:55:59.288910+0000","last_unstale":"2024-03-26T03:56:22.472065+0000","last_undegraded":"2024-03-26T03:56:22.472065+0000","last_fullsized":"2024-03-26T03:56:22.472065+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":240,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321702+0000","last_change":"2024-03-26T03:55:59.316898+0000","last_active":"2024-03-26T03:56:20.321702+0000","last_peered":"2024-03-26T03:56:20.321702+0000","last_clean":"2024-03-26T03:56:20.321702+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T03:56:20.321702+0000","last_undegraded":"2024-03-26T03:56:20.321702+0000","last_fullsized":"2024-03-26T03:56:20.321702+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":296,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471834+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T03:56:22.471834+0000","last_peered":"2024-03-26T03:56:22.471834+0000","last_clean":"2024-03-26T03:56:22.471834+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:56:22.471834+0000","last_undegraded":"2024-03-26T03:56:22.471834+0000","last_fullsized":"2024-03-26T03:56:22.471834+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:43.076943+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"154'35","reported_seq":468,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481906+0000","last_change":"2024-03-26T03:54:29.460582+0000","last_active":"2024-03-26T03:56:22.481906+0000","last_peered":"2024-03-26T03:56:22.481906+0000","last_clean":"2024-03-26T03:56:22.481906+0000","last_became_active":"2024-03-26T03:54:29.371147+0000","last_became_peered":"2024-03-26T03:54:29.371147+0000","last_unstale":"2024-03-26T03:56:22.481906+0000","last_undegraded":"2024-03-26T03:56:22.481906+0000","last_fullsized":"2024-03-26T03:56:22.481906+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":282,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.500373+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:56:22.500373+0000","last_peered":"2024-03-26T03:56:22.500373+0000","last_clean":"2024-03-26T03:56:22.500373+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:56:22.500373+0000","last_undegraded":"2024-03-26T03:56:22.500373+0000","last_fullsized":"2024-03-26T03:56:22.500373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"192'8523","reported_seq":8607,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.292067+0000","last_change":"2024-03-26T03:54:38.382935+0000","last_active":"2024-03-26T03:56:24.292067+0000","last_peered":"2024-03-26T03:56:24.292067+0000","last_clean":"2024-03-26T03:56:24.292067+0000","last_became_active":"2024-03-26T03:54:38.382581+0000","last_became_peered":"2024-03-26T03:54:38.382581+0000","last_unstale":"2024-03-26T03:56:24.292067+0000","last_undegraded":"2024-03-26T03:56:24.292067+0000","last_fullsized":"2024-03-26T03:56:24.292067+0000","mapping_epoch":155,"log_start":"170'4500","ondisk_log_start":"170'4500","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4023,"log_dups_size":0,"ondisk_log_size":4023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":530776064,"num_objects":8099,"num_object_clones":0,"num_object_copies":24297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8523,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":299,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551399+0000","last_change":"2024-03-26T03:55:59.310549+0000","last_active":"2024-03-26T03:56:22.551399+0000","last_peered":"2024-03-26T03:56:22.551399+0000","last_clean":"2024-03-26T03:56:22.551399+0000","last_became_active":"2024-03-26T03:55:59.310313+0000","last_became_peered":"2024-03-26T03:55:59.310313+0000","last_unstale":"2024-03-26T03:56:22.551399+0000","last_undegraded":"2024-03-26T03:56:22.551399+0000","last_fullsized":"2024-03-26T03:56:22.551399+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.773109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:40.709048+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":303,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.483427+0000","last_change":"2024-03-26T03:55:29.863335+0000","last_active":"2024-03-26T03:56:22.483427+0000","last_peered":"2024-03-26T03:56:22.483427+0000","last_clean":"2024-03-26T03:56:22.483427+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:56:22.483427+0000","last_undegraded":"2024-03-26T03:56:22.483427+0000","last_fullsized":"2024-03-26T03:56:22.483427+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.863201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.863201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:47.102177+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.494451+0000","last_change":"2024-03-26T03:54:29.333438+0000","last_active":"2024-03-26T03:56:22.494451+0000","last_peered":"2024-03-26T03:56:22.494451+0000","last_clean":"2024-03-26T03:56:22.494451+0000","last_became_active":"2024-03-26T03:54:29.319429+0000","last_became_peered":"2024-03-26T03:54:29.319429+0000","last_unstale":"2024-03-26T03:56:22.494451+0000","last_undegraded":"2024-03-26T03:56:22.494451+0000","last_fullsized":"2024-03-26T03:56:22.494451+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.962798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.962798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":325,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.487600+0000","last_change":"2024-03-26T03:55:59.300086+0000","last_active":"2024-03-26T03:56:22.487600+0000","last_peered":"2024-03-26T03:56:22.487600+0000","last_clean":"2024-03-26T03:56:22.487600+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T03:56:22.487600+0000","last_undegraded":"2024-03-26T03:56:22.487600+0000","last_fullsized":"2024-03-26T03:56:22.487600+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.498774+0000","last_change":"2024-03-26T03:55:59.310466+0000","last_active":"2024-03-26T03:56:22.498774+0000","last_peered":"2024-03-26T03:56:22.498774+0000","last_clean":"2024-03-26T03:56:22.498774+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T03:56:22.498774+0000","last_undegraded":"2024-03-26T03:56:22.498774+0000","last_fullsized":"2024-03-26T03:56:22.498774+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.387809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.387809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"192'8512","reported_seq":8607,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.842034+0000","last_change":"2024-03-26T03:56:00.607744+0000","last_active":"2024-03-26T03:56:24.842034+0000","last_peered":"2024-03-26T03:56:24.842034+0000","last_clean":"2024-03-26T03:55:19.247499+0000","last_became_active":"2024-03-26T03:56:00.597387+0000","last_became_peered":"2024-03-26T03:56:00.597387+0000","last_unstale":"2024-03-26T03:56:24.842034+0000","last_undegraded":"2024-03-26T03:56:00.318184+0000","last_fullsized":"2024-03-26T03:56:00.286579+0000","mapping_epoch":181,"log_start":"170'4500","ondisk_log_start":"170'4500","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4012,"log_dups_size":0,"ondisk_log_size":4012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":524419072,"num_objects":8002,"num_object_clones":0,"num_object_copies":24006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8512,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4734}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":240,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321240+0000","last_change":"2024-03-26T03:55:59.296154+0000","last_active":"2024-03-26T03:56:20.321240+0000","last_peered":"2024-03-26T03:56:20.321240+0000","last_clean":"2024-03-26T03:56:20.321240+0000","last_became_active":"2024-03-26T03:55:59.294575+0000","last_became_peered":"2024-03-26T03:55:59.294575+0000","last_unstale":"2024-03-26T03:56:20.321240+0000","last_undegraded":"2024-03-26T03:56:20.321240+0000","last_fullsized":"2024-03-26T03:56:20.321240+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":554,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490621+0000","last_change":"2024-03-26T03:55:17.691688+0000","last_active":"2024-03-26T03:56:22.490621+0000","last_peered":"2024-03-26T03:56:22.490621+0000","last_clean":"2024-03-26T03:56:22.490621+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:56:22.490621+0000","last_undegraded":"2024-03-26T03:56:22.490621+0000","last_fullsized":"2024-03-26T03:56:22.490621+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:55:17.691502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.691502+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":115,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321323+0000","last_change":"2024-03-26T03:55:59.298495+0000","last_active":"2024-03-26T03:56:20.321323+0000","last_peered":"2024-03-26T03:56:20.321323+0000","last_clean":"2024-03-26T03:56:20.321323+0000","last_became_active":"2024-03-26T03:55:59.298187+0000","last_became_peered":"2024-03-26T03:55:59.298187+0000","last_unstale":"2024-03-26T03:56:20.321323+0000","last_undegraded":"2024-03-26T03:56:20.321323+0000","last_fullsized":"2024-03-26T03:56:20.321323+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":349,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472227+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T03:56:22.472227+0000","last_peered":"2024-03-26T03:56:22.472227+0000","last_clean":"2024-03-26T03:56:22.472227+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:56:22.472227+0000","last_undegraded":"2024-03-26T03:56:22.472227+0000","last_fullsized":"2024-03-26T03:56:22.472227+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:45.959788+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.552652+0000","last_change":"2024-03-26T03:55:59.299269+0000","last_active":"2024-03-26T03:56:22.552652+0000","last_peered":"2024-03-26T03:56:22.552652+0000","last_clean":"2024-03-26T03:56:22.552652+0000","last_became_active":"2024-03-26T03:55:59.298950+0000","last_became_peered":"2024-03-26T03:55:59.298950+0000","last_unstale":"2024-03-26T03:56:22.552652+0000","last_undegraded":"2024-03-26T03:56:22.552652+0000","last_fullsized":"2024-03-26T03:56:22.552652+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"192'8303","reported_seq":8394,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.761380+0000","last_change":"2024-03-26T03:56:00.604282+0000","last_active":"2024-03-26T03:56:24.761380+0000","last_peered":"2024-03-26T03:56:24.761380+0000","last_clean":"2024-03-26T03:55:19.256252+0000","last_became_active":"2024-03-26T03:56:00.597388+0000","last_became_peered":"2024-03-26T03:56:00.597388+0000","last_unstale":"2024-03-26T03:56:24.761380+0000","last_undegraded":"2024-03-26T03:56:00.387509+0000","last_fullsized":"2024-03-26T03:56:00.283703+0000","mapping_epoch":181,"log_start":"170'4700","ondisk_log_start":"170'4700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509280256,"num_objects":7771,"num_object_clones":0,"num_object_copies":23313,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7771,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8303,"num_write_kb":514368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":4563}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481567+0000","last_change":"2024-03-26T03:55:59.300224+0000","last_active":"2024-03-26T03:56:22.481567+0000","last_peered":"2024-03-26T03:56:22.481567+0000","last_clean":"2024-03-26T03:56:22.481567+0000","last_became_active":"2024-03-26T03:55:59.299592+0000","last_became_peered":"2024-03-26T03:55:59.299592+0000","last_unstale":"2024-03-26T03:56:22.481567+0000","last_undegraded":"2024-03-26T03:56:22.481567+0000","last_fullsized":"2024-03-26T03:56:22.481567+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":313,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.502413+0000","last_change":"2024-03-26T03:55:59.309172+0000","last_active":"2024-03-26T03:56:22.502413+0000","last_peered":"2024-03-26T03:56:22.502413+0000","last_clean":"2024-03-26T03:56:22.502413+0000","last_became_active":"2024-03-26T03:55:59.308797+0000","last_became_peered":"2024-03-26T03:55:59.308797+0000","last_unstale":"2024-03-26T03:56:22.502413+0000","last_undegraded":"2024-03-26T03:56:22.502413+0000","last_fullsized":"2024-03-26T03:56:22.502413+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:31.280453+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":762,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490575+0000","last_change":"2024-03-26T03:55:58.942740+0000","last_active":"2024-03-26T03:56:22.490575+0000","last_peered":"2024-03-26T03:56:22.490575+0000","last_clean":"2024-03-26T03:56:22.490575+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:56:22.490575+0000","last_undegraded":"2024-03-26T03:56:22.490575+0000","last_fullsized":"2024-03-26T03:56:22.490575+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:55:58.942635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.942635+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:03.755920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":452,"num_read_kb":447,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":304,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493011+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T03:56:22.493011+0000","last_peered":"2024-03-26T03:56:22.493011+0000","last_clean":"2024-03-26T03:56:22.493011+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:56:22.493011+0000","last_undegraded":"2024-03-26T03:56:22.493011+0000","last_fullsized":"2024-03-26T03:56:22.493011+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:23.638440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":227,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321331+0000","last_change":"2024-03-26T03:55:59.311162+0000","last_active":"2024-03-26T03:56:20.321331+0000","last_peered":"2024-03-26T03:56:20.321331+0000","last_clean":"2024-03-26T03:56:20.321331+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T03:56:20.321331+0000","last_undegraded":"2024-03-26T03:56:20.321331+0000","last_fullsized":"2024-03-26T03:56:20.321331+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"192'8441","reported_seq":8525,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.646703+0000","last_change":"2024-03-26T03:54:38.384402+0000","last_active":"2024-03-26T03:56:24.646703+0000","last_peered":"2024-03-26T03:56:24.646703+0000","last_clean":"2024-03-26T03:56:24.646703+0000","last_became_active":"2024-03-26T03:54:38.384087+0000","last_became_peered":"2024-03-26T03:54:38.384087+0000","last_unstale":"2024-03-26T03:56:24.646703+0000","last_undegraded":"2024-03-26T03:56:24.646703+0000","last_fullsized":"2024-03-26T03:56:24.646703+0000","mapping_epoch":155,"log_start":"167'3800","ondisk_log_start":"167'3800","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4641,"log_dups_size":0,"ondisk_log_size":4641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520421376,"num_objects":7941,"num_object_clones":0,"num_object_copies":23823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8441,"num_write_kb":524224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":283,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481562+0000","last_change":"2024-03-26T03:55:15.751265+0000","last_active":"2024-03-26T03:56:22.481562+0000","last_peered":"2024-03-26T03:56:22.481562+0000","last_clean":"2024-03-26T03:56:22.481562+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:56:22.481562+0000","last_undegraded":"2024-03-26T03:56:22.481562+0000","last_fullsized":"2024-03-26T03:56:22.481562+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:15.751136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:15.751136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:33.339989+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":298,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.481742+0000","last_change":"2024-03-26T03:55:59.301777+0000","last_active":"2024-03-26T03:56:22.481742+0000","last_peered":"2024-03-26T03:56:22.481742+0000","last_clean":"2024-03-26T03:56:22.481742+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T03:56:22.481742+0000","last_undegraded":"2024-03-26T03:56:22.481742+0000","last_fullsized":"2024-03-26T03:56:22.481742+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":343,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.492325+0000","last_change":"2024-03-26T03:55:59.309535+0000","last_active":"2024-03-26T03:56:22.492325+0000","last_peered":"2024-03-26T03:56:22.492325+0000","last_clean":"2024-03-26T03:56:22.492325+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T03:56:22.492325+0000","last_undegraded":"2024-03-26T03:56:22.492325+0000","last_fullsized":"2024-03-26T03:56:22.492325+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":229,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491250+0000","last_change":"2024-03-26T03:55:59.308367+0000","last_active":"2024-03-26T03:56:22.491250+0000","last_peered":"2024-03-26T03:56:22.491250+0000","last_clean":"2024-03-26T03:56:22.491250+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T03:56:22.491250+0000","last_undegraded":"2024-03-26T03:56:22.491250+0000","last_fullsized":"2024-03-26T03:56:22.491250+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"192'8333","reported_seq":8417,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.823222+0000","last_change":"2024-03-26T03:54:38.379391+0000","last_active":"2024-03-26T03:56:24.823222+0000","last_peered":"2024-03-26T03:56:24.823222+0000","last_clean":"2024-03-26T03:56:24.823222+0000","last_became_active":"2024-03-26T03:54:38.379010+0000","last_became_peered":"2024-03-26T03:54:38.379010+0000","last_unstale":"2024-03-26T03:56:24.823222+0000","last_undegraded":"2024-03-26T03:56:24.823222+0000","last_fullsized":"2024-03-26T03:56:24.823222+0000","mapping_epoch":155,"log_start":"170'4300","ondisk_log_start":"170'4300","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4033,"log_dups_size":0,"ondisk_log_size":4033,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514785280,"num_objects":7855,"num_object_clones":0,"num_object_copies":23565,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7855,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8333,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":285,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.496860+0000","last_change":"2024-03-26T03:55:59.324661+0000","last_active":"2024-03-26T03:56:22.496860+0000","last_peered":"2024-03-26T03:56:22.496860+0000","last_clean":"2024-03-26T03:56:22.496860+0000","last_became_active":"2024-03-26T03:55:59.324158+0000","last_became_peered":"2024-03-26T03:55:59.324158+0000","last_unstale":"2024-03-26T03:56:22.496860+0000","last_undegraded":"2024-03-26T03:56:22.496860+0000","last_fullsized":"2024-03-26T03:56:22.496860+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":252,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321310+0000","last_change":"2024-03-26T03:55:59.304498+0000","last_active":"2024-03-26T03:56:20.321310+0000","last_peered":"2024-03-26T03:56:20.321310+0000","last_clean":"2024-03-26T03:56:20.321310+0000","last_became_active":"2024-03-26T03:55:59.298094+0000","last_became_peered":"2024-03-26T03:55:59.298094+0000","last_unstale":"2024-03-26T03:56:20.321310+0000","last_undegraded":"2024-03-26T03:56:20.321310+0000","last_fullsized":"2024-03-26T03:56:20.321310+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":321,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.486479+0000","last_change":"2024-03-26T03:55:59.299304+0000","last_active":"2024-03-26T03:56:22.486479+0000","last_peered":"2024-03-26T03:56:22.486479+0000","last_clean":"2024-03-26T03:56:22.486479+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T03:56:22.486479+0000","last_undegraded":"2024-03-26T03:56:22.486479+0000","last_fullsized":"2024-03-26T03:56:22.486479+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"181'12","reported_seq":406,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.491546+0000","last_change":"2024-03-26T03:55:59.310069+0000","last_active":"2024-03-26T03:56:22.491546+0000","last_peered":"2024-03-26T03:56:22.491546+0000","last_clean":"2024-03-26T03:56:22.491546+0000","last_became_active":"2024-03-26T03:55:59.309116+0000","last_became_peered":"2024-03-26T03:55:59.309116+0000","last_unstale":"2024-03-26T03:56:22.491546+0000","last_undegraded":"2024-03-26T03:56:22.491546+0000","last_fullsized":"2024-03-26T03:56:22.491546+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"192'8306","reported_seq":8397,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.582567+0000","last_change":"2024-03-26T03:56:00.577247+0000","last_active":"2024-03-26T03:56:24.582567+0000","last_peered":"2024-03-26T03:56:24.582567+0000","last_clean":"2024-03-26T03:55:19.255545+0000","last_became_active":"2024-03-26T03:56:00.572469+0000","last_became_peered":"2024-03-26T03:56:00.572469+0000","last_unstale":"2024-03-26T03:56:24.582567+0000","last_undegraded":"2024-03-26T03:56:00.311081+0000","last_fullsized":"2024-03-26T03:56:00.282491+0000","mapping_epoch":181,"log_start":"166'3700","ondisk_log_start":"166'3700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515178520,"num_objects":7862,"num_object_clones":0,"num_object_copies":23586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3174,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7862,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8306,"num_write_kb":517313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":4688}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":284,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.550881+0000","last_change":"2024-03-26T03:53:34.666684+0000","last_active":"2024-03-26T03:56:22.550881+0000","last_peered":"2024-03-26T03:56:22.550881+0000","last_clean":"2024-03-26T03:56:22.550881+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:56:22.550881+0000","last_undegraded":"2024-03-26T03:56:22.550881+0000","last_fullsized":"2024-03-26T03:56:22.550881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.666607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.666607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":291,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.493105+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:56:22.493105+0000","last_peered":"2024-03-26T03:56:22.493105+0000","last_clean":"2024-03-26T03:56:22.493105+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:56:22.493105+0000","last_undegraded":"2024-03-26T03:56:22.493105+0000","last_fullsized":"2024-03-26T03:56:22.493105+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":352,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471795+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T03:56:22.471795+0000","last_peered":"2024-03-26T03:56:22.471795+0000","last_clean":"2024-03-26T03:56:22.471795+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T03:56:22.471795+0000","last_undegraded":"2024-03-26T03:56:22.471795+0000","last_fullsized":"2024-03-26T03:56:22.471795+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"154'10","reported_seq":150,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.483107+0000","last_change":"2024-03-26T03:54:34.943516+0000","last_active":"2024-03-26T03:56:22.483107+0000","last_peered":"2024-03-26T03:56:22.483107+0000","last_clean":"2024-03-26T03:56:22.483107+0000","last_became_active":"2024-03-26T03:54:29.356520+0000","last_became_peered":"2024-03-26T03:54:29.356520+0000","last_unstale":"2024-03-26T03:56:22.483107+0000","last_undegraded":"2024-03-26T03:56:22.483107+0000","last_fullsized":"2024-03-26T03:56:22.483107+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"192'8284","reported_seq":10016,"reported_epoch":192,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.852845+0000","last_change":"2024-03-26T03:56:00.576519+0000","last_active":"2024-03-26T03:56:24.852845+0000","last_peered":"2024-03-26T03:56:24.852845+0000","last_clean":"2024-03-26T03:55:19.248623+0000","last_became_active":"2024-03-26T03:56:00.560122+0000","last_became_peered":"2024-03-26T03:56:00.560122+0000","last_unstale":"2024-03-26T03:56:24.852845+0000","last_undegraded":"2024-03-26T03:56:00.315936+0000","last_fullsized":"2024-03-26T03:56:00.286924+0000","mapping_epoch":181,"log_start":"170'4300","ondisk_log_start":"170'4300","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3984,"log_dups_size":0,"ondisk_log_size":3984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511836160,"num_objects":7810,"num_object_clones":0,"num_object_copies":23430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2276,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8284,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":822,"num_bytes_recovered":53870592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":5534}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":276,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472241+0000","last_change":"2024-03-26T03:54:29.432125+0000","last_active":"2024-03-26T03:56:22.472241+0000","last_peered":"2024-03-26T03:56:22.472241+0000","last_clean":"2024-03-26T03:56:22.472241+0000","last_became_active":"2024-03-26T03:54:29.431697+0000","last_became_peered":"2024-03-26T03:54:29.431697+0000","last_unstale":"2024-03-26T03:56:22.472241+0000","last_undegraded":"2024-03-26T03:56:22.472241+0000","last_fullsized":"2024-03-26T03:56:22.472241+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":97,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.551171+0000","last_change":"2024-03-26T03:55:59.294273+0000","last_active":"2024-03-26T03:56:22.551171+0000","last_peered":"2024-03-26T03:56:22.551171+0000","last_clean":"2024-03-26T03:56:22.551171+0000","last_became_active":"2024-03-26T03:55:59.293987+0000","last_became_peered":"2024-03-26T03:55:59.293987+0000","last_unstale":"2024-03-26T03:56:22.551171+0000","last_undegraded":"2024-03-26T03:56:22.551171+0000","last_fullsized":"2024-03-26T03:56:22.551171+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":214,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490804+0000","last_change":"2024-03-26T03:54:29.536599+0000","last_active":"2024-03-26T03:56:22.490804+0000","last_peered":"2024-03-26T03:56:22.490804+0000","last_clean":"2024-03-26T03:56:22.490804+0000","last_became_active":"2024-03-26T03:54:29.358363+0000","last_became_peered":"2024-03-26T03:54:29.358363+0000","last_unstale":"2024-03-26T03:56:22.490804+0000","last_undegraded":"2024-03-26T03:56:22.490804+0000","last_fullsized":"2024-03-26T03:56:22.490804+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":288,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471447+0000","last_change":"2024-03-26T03:55:22.211690+0000","last_active":"2024-03-26T03:56:22.471447+0000","last_peered":"2024-03-26T03:56:22.471447+0000","last_clean":"2024-03-26T03:56:22.471447+0000","last_became_active":"2024-03-26T03:54:29.346424+0000","last_became_peered":"2024-03-26T03:54:29.346424+0000","last_unstale":"2024-03-26T03:56:22.471447+0000","last_undegraded":"2024-03-26T03:56:22.471447+0000","last_fullsized":"2024-03-26T03:56:22.471447+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:30.099189+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"192'8538","reported_seq":8628,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.711155+0000","last_change":"2024-03-26T03:56:00.609756+0000","last_active":"2024-03-26T03:56:24.711155+0000","last_peered":"2024-03-26T03:56:24.711155+0000","last_clean":"2024-03-26T03:55:19.268201+0000","last_became_active":"2024-03-26T03:56:00.598736+0000","last_became_peered":"2024-03-26T03:56:00.598736+0000","last_unstale":"2024-03-26T03:56:24.711155+0000","last_undegraded":"2024-03-26T03:56:00.325858+0000","last_fullsized":"2024-03-26T03:56:00.284403+0000","mapping_epoch":181,"log_start":"170'4900","ondisk_log_start":"170'4900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3638,"log_dups_size":0,"ondisk_log_size":3638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528220160,"num_objects":8060,"num_object_clones":0,"num_object_copies":24180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8538,"num_write_kb":531136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":4799}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":205,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321163+0000","last_change":"2024-03-26T03:55:59.341468+0000","last_active":"2024-03-26T03:56:20.321163+0000","last_peered":"2024-03-26T03:56:20.321163+0000","last_clean":"2024-03-26T03:56:20.321163+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T03:56:20.321163+0000","last_undegraded":"2024-03-26T03:56:20.321163+0000","last_fullsized":"2024-03-26T03:56:20.321163+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":394,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472575+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T03:56:22.472575+0000","last_peered":"2024-03-26T03:56:22.472575+0000","last_clean":"2024-03-26T03:56:22.472575+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T03:56:22.472575+0000","last_undegraded":"2024-03-26T03:56:22.472575+0000","last_fullsized":"2024-03-26T03:56:22.472575+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":96,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.483611+0000","last_change":"2024-03-26T03:54:31.922380+0000","last_active":"2024-03-26T03:56:22.483611+0000","last_peered":"2024-03-26T03:56:22.483611+0000","last_clean":"2024-03-26T03:56:22.483611+0000","last_became_active":"2024-03-26T03:54:29.377662+0000","last_became_peered":"2024-03-26T03:54:29.377662+0000","last_unstale":"2024-03-26T03:56:22.483611+0000","last_undegraded":"2024-03-26T03:56:22.483611+0000","last_fullsized":"2024-03-26T03:56:22.483611+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":332,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.490848+0000","last_change":"2024-03-26T03:54:29.386220+0000","last_active":"2024-03-26T03:56:22.490848+0000","last_peered":"2024-03-26T03:56:22.490848+0000","last_clean":"2024-03-26T03:56:22.490848+0000","last_became_active":"2024-03-26T03:54:29.299380+0000","last_became_peered":"2024-03-26T03:54:29.299380+0000","last_unstale":"2024-03-26T03:56:22.490848+0000","last_undegraded":"2024-03-26T03:56:22.490848+0000","last_fullsized":"2024-03-26T03:56:22.490848+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.835587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.835587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"192'8243","reported_seq":11154,"reported_epoch":192,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.279967+0000","last_change":"2024-03-26T03:56:00.586196+0000","last_active":"2024-03-26T03:56:24.279967+0000","last_peered":"2024-03-26T03:56:24.279967+0000","last_clean":"2024-03-26T03:55:19.255604+0000","last_became_active":"2024-03-26T03:56:00.575261+0000","last_became_peered":"2024-03-26T03:56:00.575261+0000","last_unstale":"2024-03-26T03:56:24.279967+0000","last_undegraded":"2024-03-26T03:56:00.293512+0000","last_fullsized":"2024-03-26T03:56:00.279632+0000","mapping_epoch":181,"log_start":"170'4200","ondisk_log_start":"170'4200","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4043,"log_dups_size":0,"ondisk_log_size":4043,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":509149184,"num_objects":7769,"num_object_clones":0,"num_object_copies":23307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1767,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8243,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1411,"num_bytes_recovered":92471296,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":6002}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":103,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.320578+0000","last_change":"2024-03-26T03:55:59.342271+0000","last_active":"2024-03-26T03:56:20.320578+0000","last_peered":"2024-03-26T03:56:20.320578+0000","last_clean":"2024-03-26T03:56:20.320578+0000","last_became_active":"2024-03-26T03:55:59.341382+0000","last_became_peered":"2024-03-26T03:55:59.341382+0000","last_unstale":"2024-03-26T03:56:20.320578+0000","last_undegraded":"2024-03-26T03:56:20.320578+0000","last_fullsized":"2024-03-26T03:56:20.320578+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":247,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-03-26T03:56:20.321146+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T03:56:20.321146+0000","last_peered":"2024-03-26T03:56:20.321146+0000","last_clean":"2024-03-26T03:56:20.321146+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T03:56:20.321146+0000","last_undegraded":"2024-03-26T03:56:20.321146+0000","last_fullsized":"2024-03-26T03:56:20.321146+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":290,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471553+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:56:22.471553+0000","last_peered":"2024-03-26T03:56:22.471553+0000","last_clean":"2024-03-26T03:56:22.471553+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:56:22.471553+0000","last_undegraded":"2024-03-26T03:56:22.471553+0000","last_fullsized":"2024-03-26T03:56:22.471553+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":366,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472293+0000","last_change":"2024-03-26T03:55:59.289896+0000","last_active":"2024-03-26T03:56:22.472293+0000","last_peered":"2024-03-26T03:56:22.472293+0000","last_clean":"2024-03-26T03:56:22.472293+0000","last_became_active":"2024-03-26T03:55:59.289458+0000","last_became_peered":"2024-03-26T03:55:59.289458+0000","last_unstale":"2024-03-26T03:56:22.472293+0000","last_undegraded":"2024-03-26T03:56:22.472293+0000","last_fullsized":"2024-03-26T03:56:22.472293+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"192'8486","reported_seq":8570,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:24.748873+0000","last_change":"2024-03-26T03:54:38.382794+0000","last_active":"2024-03-26T03:56:24.748873+0000","last_peered":"2024-03-26T03:56:24.748873+0000","last_clean":"2024-03-26T03:56:24.748873+0000","last_became_active":"2024-03-26T03:54:38.382452+0000","last_became_peered":"2024-03-26T03:54:38.382452+0000","last_unstale":"2024-03-26T03:56:24.748873+0000","last_undegraded":"2024-03-26T03:56:24.748873+0000","last_fullsized":"2024-03-26T03:56:24.748873+0000","mapping_epoch":155,"log_start":"170'4900","ondisk_log_start":"170'4900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3586,"log_dups_size":0,"ondisk_log_size":3586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":523370496,"num_objects":7986,"num_object_clones":0,"num_object_copies":23958,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7986,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8486,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":300,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.499536+0000","last_change":"2024-03-26T03:55:55.887208+0000","last_active":"2024-03-26T03:56:22.499536+0000","last_peered":"2024-03-26T03:56:22.499536+0000","last_clean":"2024-03-26T03:56:22.499536+0000","last_became_active":"2024-03-26T03:54:29.296384+0000","last_became_peered":"2024-03-26T03:54:29.296384+0000","last_unstale":"2024-03-26T03:56:22.499536+0000","last_undegraded":"2024-03-26T03:56:22.499536+0000","last_fullsized":"2024-03-26T03:56:22.499536+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:55.887078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:55.887078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:11.476315+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":281,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.482040+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:56:22.482040+0000","last_peered":"2024-03-26T03:56:22.482040+0000","last_clean":"2024-03-26T03:56:22.482040+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:56:22.482040+0000","last_undegraded":"2024-03-26T03:56:22.482040+0000","last_fullsized":"2024-03-26T03:56:22.482040+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":235,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472343+0000","last_change":"2024-03-26T03:55:59.299222+0000","last_active":"2024-03-26T03:56:22.472343+0000","last_peered":"2024-03-26T03:56:22.472343+0000","last_clean":"2024-03-26T03:56:22.472343+0000","last_became_active":"2024-03-26T03:55:59.298700+0000","last_became_peered":"2024-03-26T03:55:59.298700+0000","last_unstale":"2024-03-26T03:56:22.472343+0000","last_undegraded":"2024-03-26T03:56:22.472343+0000","last_fullsized":"2024-03-26T03:56:22.472343+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":314,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472438+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:56:22.472438+0000","last_peered":"2024-03-26T03:56:22.472438+0000","last_clean":"2024-03-26T03:56:22.472438+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:56:22.472438+0000","last_undegraded":"2024-03-26T03:56:22.472438+0000","last_fullsized":"2024-03-26T03:56:22.472438+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"192'8358","reported_seq":8445,"reported_epoch":192,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:24.761966+0000","last_change":"2024-03-26T03:56:00.619485+0000","last_active":"2024-03-26T03:56:24.761966+0000","last_peered":"2024-03-26T03:56:24.761966+0000","last_clean":"2024-03-26T03:55:19.228504+0000","last_became_active":"2024-03-26T03:56:00.608636+0000","last_became_peered":"2024-03-26T03:56:00.608636+0000","last_unstale":"2024-03-26T03:56:24.761966+0000","last_undegraded":"2024-03-26T03:56:00.316145+0000","last_fullsized":"2024-03-26T03:56:00.283634+0000","mapping_epoch":181,"log_start":"170'4700","ondisk_log_start":"170'4700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3658,"log_dups_size":0,"ondisk_log_size":3658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514981888,"num_objects":7858,"num_object_clones":0,"num_object_copies":23574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8358,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":4574}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":278,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.472631+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:56:22.472631+0000","last_peered":"2024-03-26T03:56:22.472631+0000","last_clean":"2024-03-26T03:56:22.472631+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:56:22.472631+0000","last_undegraded":"2024-03-26T03:56:22.472631+0000","last_fullsized":"2024-03-26T03:56:22.472631+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":402,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.498510+0000","last_change":"2024-03-26T03:55:59.796875+0000","last_active":"2024-03-26T03:56:22.498510+0000","last_peered":"2024-03-26T03:56:22.498510+0000","last_clean":"2024-03-26T03:56:22.498510+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T03:56:22.498510+0000","last_undegraded":"2024-03-26T03:56:22.498510+0000","last_fullsized":"2024-03-26T03:56:22.498510+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:55:59.796779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:59.796779+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:23.173731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":96,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.492427+0000","last_change":"2024-03-26T03:54:33.815241+0000","last_active":"2024-03-26T03:56:22.492427+0000","last_peered":"2024-03-26T03:56:22.492427+0000","last_clean":"2024-03-26T03:56:22.492427+0000","last_became_active":"2024-03-26T03:54:29.399842+0000","last_became_peered":"2024-03-26T03:54:29.399842+0000","last_unstale":"2024-03-26T03:56:22.492427+0000","last_undegraded":"2024-03-26T03:56:22.492427+0000","last_fullsized":"2024-03-26T03:56:22.492427+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":337,"reported_epoch":192,"state":"active+clean","last_fresh":"2024-03-26T03:56:22.471980+0000","last_change":"2024-03-26T03:55:17.192246+0000","last_active":"2024-03-26T03:56:22.471980+0000","last_peered":"2024-03-26T03:56:22.471980+0000","last_clean":"2024-03-26T03:56:22.471980+0000","last_became_active":"2024-03-26T03:54:29.296174+0000","last_became_peered":"2024-03-26T03:54:29.296174+0000","last_unstale":"2024-03-26T03:56:22.471980+0000","last_undegraded":"2024-03-26T03:56:22.471980+0000","last_fullsized":"2024-03-26T03:56:22.471980+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:45.283846+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":8263630872,"num_objects":126094,"num_object_clones":0,"num_object_copies":378282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25618,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126094,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":133764,"num_write_kb":8315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3230,"num_bytes_recovered":211681280,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":23189401600,"data_stored":23189389384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64087,"ondisk_log_size":64087,"up":48,"acting":39,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":452,"num_read_kb":447,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070560,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2690300,"kb_used_data":2514676,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91054852,"statfs":{"total":95995035648,"available":93240168448,"internally_reserved":0,"allocated":2575028224,"data_stored":2574406481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":179763658},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364902,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574975,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3825992,"kb_used_data":3555968,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":89919160,"statfs":{"total":95995035648,"available":92077219840,"internally_reserved":0,"allocated":3641311232,"data_stored":3640677383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738500,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4324740,"kb_used_data":4022844,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":89420412,"statfs":{"total":95995035648,"available":91566501888,"internally_reserved":0,"allocated":4119392256,"data_stored":4118762455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411450,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5394516,"kb_used_data":5092620,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":88350636,"statfs":{"total":95995035648,"available":90471051264,"internally_reserved":0,"allocated":5214842880,"data_stored":5214259305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182516,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953583,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4850016,"kb_used_data":4561944,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":88895136,"statfs":{"total":95995035648,"available":91028619264,"internally_reserved":0,"allocated":4671430656,"data_stored":4670834819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113286,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3071004,"kb_used_data":2977552,"kb_used_omap":4,"kb_used_meta":93243,"kb_avail":90674148,"statfs":{"total":95995035648,"available":92850327552,"internally_reserved":0,"allocated":3049013248,"data_stored":3048418941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":95481504},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4109238272,"data_stored":4109238272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3637510144,"data_stored":3637510144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":5184163840,"data_stored":5184159768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4667998208,"data_stored":4667998208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3025997824,"data_stored":3025993752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2564493312,"data_stored":2564489240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:56:29.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-03-26T03:56:29.284 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:29.292 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:56:29.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:28 smithi086 ceph-mon[32965]: pgmap v329: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 35 MiB/s wr, 1.04k op/s; 25618/378972 objects degraded (6.760%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:56:29.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:56:29.387 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:56:29.433 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:56:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:29.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:29.474 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:56:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:56:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:56:29.480 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:56:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:56:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:56:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:56:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:56:29.481 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:56:29.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:29.586 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:29.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:29.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:29.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:29.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:29.861 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:29.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:29.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:30.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:30.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:30.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:30.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:30.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:30.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:29 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2476479740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 25618/378972 objects degraded (6.760%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:30.352 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:29 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2476479740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:30.352 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 25618/378972 objects degraded (6.760%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:29 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2476479740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 25618/378972 objects degraded (6.760%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:30.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:30.439 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:30.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:30.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:30.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:30.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:30.721 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:30.806 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:30.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:30.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:30.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:30.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:30.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:30.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:31.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:31.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.130 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:31.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:31.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:31.251 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:30 smithi006 ceph-mon[29446]: pgmap v330: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 MiB/s wr, 636 op/s; 25196/376077 objects degraded (6.700%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T03:56:31.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:30 smithi006 ceph-mon[37501]: pgmap v330: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 MiB/s wr, 636 op/s; 25196/376077 objects degraded (6.700%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T03:56:31.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:31.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:30 smithi086 ceph-mon[32965]: pgmap v330: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.6 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 6.5 MiB/s wr, 636 op/s; 25196/376077 objects degraded (6.700%); 8.3 MiB/s, 132 objects/s recovering 2024-03-26T03:56:31.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:31.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:31.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:31.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:31.557 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:31.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:31.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:31.712 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:31.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:31.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:31.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:31.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:31.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:31.967 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:31.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:31.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:31.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:32.091 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:32.130 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:32.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:32.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:32.232 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:31 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:32.232 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:31 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:32.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:31 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:32.233 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:31 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:32.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:31 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:31 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:32.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:32.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:32.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:32.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:56:32.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:32.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:32.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:32.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:32.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:32.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:32.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.821 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:32.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:32.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:32.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:32.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:32.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:33.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:33.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:33.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:33.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:33.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:33.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:32 smithi086 ceph-mon[32965]: pgmap v331: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:32 smithi086 ceph-mon[32965]: osdmap e193: 8 total, 8 up, 6 in 2024-03-26T03:56:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:33.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:32 smithi006 ceph-mon[37501]: pgmap v331: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:32 smithi006 ceph-mon[37501]: osdmap e193: 8 total, 8 up, 6 in 2024-03-26T03:56:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:32 smithi006 ceph-mon[29446]: pgmap v331: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:32 smithi006 ceph-mon[29446]: osdmap e193: 8 total, 8 up, 6 in 2024-03-26T03:56:33.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:33.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:33.653 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:33.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:33.819 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:33.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:33.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:33.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:33.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:34.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:34.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:34.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.088 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:34.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:33 smithi006 ceph-mon[37501]: pgmap v333: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:34.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:33 smithi006 ceph-mon[29446]: pgmap v333: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:34.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:34.238 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:34.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:34.321 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:56:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:33 smithi086 ceph-mon[32965]: pgmap v333: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.4 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 5.8 MiB/s wr, 985 op/s; 24765/363729 objects degraded (6.809%); 10 MiB/s, 162 objects/s recovering 2024-03-26T03:56:34.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:34.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:34.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:34.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:34.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:34.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:34.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:34.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:34.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:34.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:34.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:34.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:34.929 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:34.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:34.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:34.957 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:35.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:35.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.069 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:35.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:35.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:35.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:35.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:56:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 24765/363729 objects degraded (6.809%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:56:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: osdmap e194: 8 total, 8 up, 6 in 2024-03-26T03:56:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:34 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:35.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.349 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:35.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 24765/363729 objects degraded (6.809%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: osdmap e194: 8 total, 8 up, 6 in 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 24765/363729 objects degraded (6.809%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1174876687' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: osdmap e194: 8 total, 8 up, 6 in 2024-03-26T03:56:35.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:34 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:35.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:35.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:35.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:35.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:35.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:35.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:35.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:35.898 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:35.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:35.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:36.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:36.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:36.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:36.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:36.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:36.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:35 smithi086 ceph-mon[32965]: pgmap v335: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 23710/347979 objects degraded (6.814%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:56:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:35 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:35 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[37501]: pgmap v335: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 23710/347979 objects degraded (6.814%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:56:36.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:36.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[29446]: pgmap v335: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s; 23710/347979 objects degraded (6.814%); 15 MiB/s, 238 objects/s recovering 2024-03-26T03:56:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:35 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:36.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:36.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:36.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:36.583 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:36.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:36.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:36.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:36.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:36.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:36.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:36.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:36.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:36.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:37.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:37.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:37.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:36 smithi006 ceph-mon[37501]: osdmap e195: 8 total, 8 up, 6 in 2024-03-26T03:56:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:36 smithi006 ceph-mon[29446]: osdmap e195: 8 total, 8 up, 6 in 2024-03-26T03:56:37.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:37.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:36 smithi086 ceph-mon[32965]: osdmap e195: 8 total, 8 up, 6 in 2024-03-26T03:56:37.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:37.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:37.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:37.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:37.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:37.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:37.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:37.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:37.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:37.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:37.944 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:37.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:37.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:38.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:38.081 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:38.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:38.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:38.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:38.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:37 smithi086 ceph-mon[32965]: pgmap v337: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 875 op/s; 23710/347979 objects degraded (6.814%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:56:38.341 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:37 smithi006 ceph-mon[29446]: pgmap v337: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 875 op/s; 23710/347979 objects degraded (6.814%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:56:38.342 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:37 smithi006 ceph-mon[37501]: pgmap v337: 148 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 875 op/s; 23710/347979 objects degraded (6.814%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:56:38.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:145393 8.c 8:35da9ee9:::benchmark_data_smithi006_103169_object15462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.141722+0000", 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.014749722, 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014783543999999999, 2024-03-26T03:56:38.458 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 145393 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141722+0000", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141722+0000", 2024-03-26T03:56:38.459 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141724+0000", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.359e-06 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141730+0000", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9649999999999996e-06 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141742+0000", 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.253e-05 2024-03-26T03:56:38.460 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.141750+0000", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1440000000000003e-06 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.142088+0000", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033766599999999998 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.142231+0000", 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014287999999999999 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.461 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.143431+0000", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001199697 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154963+0000", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011532489 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.155047+0000", 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4146000000000004e-05 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.462 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:145397 8.c 8:352b1833:::benchmark_data_smithi006_103169_object15466:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.144804+0000", 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.011668137, 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011806159, 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 145397 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144804+0000", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144804+0000", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144805+0000", 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.457e-06 2024-03-26T03:56:38.464 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144812+0000", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7039999999999997e-06 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144824+0000", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1623e-05 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.144832+0000", 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4980000000000006e-06 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.465 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148603+0000", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037706810000000001 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148730+0000", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012741400000000001 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148853+0000", 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012251800000000001 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.466 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.155010+0000", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.006156847 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.155083+0000", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2933000000000001e-05 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:145389 8.1 e195/155 8:872e2c8e:::benchmark_data_smithi006_103169_object15458:head v 195'9211, mlcod=195'9209)", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.147246+0000", 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0092257090000000003, 2024-03-26T03:56:38.467 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0094763820000000002, 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147246+0000", 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147246+0000", 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.468 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147247+0000", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6199999999999998e-07 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147252+0000", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9150000000000004e-06 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147260+0000", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5550000000000003e-06 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147268+0000", 2024-03-26T03:56:38.469 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2479999999999999e-06 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147598+0000", 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033040100000000002 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.147652+0000", 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4259999999999999e-05 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.470 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:145399 8.6 8:69961428:::benchmark_data_smithi006_103169_object15468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.148412+0000", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0080603889999999994, 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0083844680000000008, 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 145399 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.471 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148412+0000", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148412+0000", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148413+0000", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.57e-06 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148420+0000", 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0700000000000001e-06 2024-03-26T03:56:38.472 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148441+0000", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0089999999999999e-05 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148449+0000", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3960000000000006e-06 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148519+0000", 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0137999999999998e-05 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.473 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148640+0000", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121304 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148765+0000", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000124554 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.151593+0000", 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002827623 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.474 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.153292+0000", 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016994759999999999 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:145400 8.1 e195/155 8:81718689:::benchmark_data_smithi006_103169_object15469:head v 195'9212, mlcod=195'9209)", 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.148807+0000", 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0076654519999999997, 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080901129999999995, 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.475 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148807+0000", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148807+0000", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148807+0000", 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2600000000000002e-07 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148814+0000", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2999999999999998e-06 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148823+0000", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.628e-06 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.148831+0000", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5310000000000004e-06 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.150867+0000", 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002036431 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.150929+0000", 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1793999999999998e-05 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:145401 8.8 8:19503032:::benchmark_data_smithi006_103169_object15470:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.152364+0000", 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0041085139999999997, 2024-03-26T03:56:38.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046084400000000001, 2024-03-26T03:56:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:56:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:56:38.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 145401 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152364+0000", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152364+0000", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152365+0000", 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5340000000000001e-06 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152371+0000", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9370000000000001e-06 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152387+0000", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6285000000000001e-05 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152395+0000", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7959999999999996e-06 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152457+0000", 2024-03-26T03:56:38.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2123000000000001e-05 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152618+0000", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016092499999999999 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.152874+0000", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025617199999999999 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.156781+0000", 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039069930000000001 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.484 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:145403 8.2 e195/181 8:40acd5b9:::benchmark_data_smithi006_103169_object15472:head v 195'9002, mlcod=177'6419)", 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:56:36.154318+0000", 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0021539509999999999, 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027516369999999999, 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154318+0000", 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154318+0000", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154319+0000", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.09e-07 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154325+0000", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9560000000000002e-06 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:56:38.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154335+0000", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0366999999999999e-05 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154343+0000", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4309999999999997e-06 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154420+0000", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7296e-05 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:56:36.154445+0000", 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4893999999999999e-05 2024-03-26T03:56:38.487 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 7 2024-03-26T03:56:38.488 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:56:38.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:38.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:38.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:38.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:38.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:38.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:38.793 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:38.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:38.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:38.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:39.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:39.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:39.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T03:56:39.164 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:39.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.219 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:39.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:39.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:39.359 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:39.362 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:39.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:39.475 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:39.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:39.605 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:39.663 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:39.706 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:39.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:39.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:39.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:39.859 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:39.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:39.987 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:39.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:40.030 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 8 pgp_num to 16 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[37501]: osdmap e196: 8 total, 8 up, 6 in 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 23710/347979 objects degraded (6.814%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[37501]: pgmap v339: 148 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 22947/345246 objects degraded (6.647%); 19 MiB/s, 302 objects/s recovering 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2539351735' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.048 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[29446]: osdmap e196: 8 total, 8 up, 6 in 2024-03-26T03:56:40.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 23710/347979 objects degraded (6.814%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:40.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[29446]: pgmap v339: 148 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 22947/345246 objects degraded (6.647%); 19 MiB/s, 302 objects/s recovering 2024-03-26T03:56:40.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2539351735' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.049 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:40 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:40.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:40.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:40.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:40.273 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:40.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:40 smithi086 ceph-mon[32965]: osdmap e196: 8 total, 8 up, 6 in 2024-03-26T03:56:40.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:40 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 23710/347979 objects degraded (6.814%), 9 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:40 smithi086 ceph-mon[32965]: pgmap v339: 148 pgs: 1 peering, 6 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 139 active+clean; 7.0 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 22947/345246 objects degraded (6.647%); 19 MiB/s, 302 objects/s recovering 2024-03-26T03:56:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2539351735' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:40 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:56:40.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:40.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:56:40.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:40.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:40.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:40.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:40.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:40.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:40.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:40.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:40.882 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:40.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:40.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:41.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:41.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:41.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:41.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:41.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:41 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:56:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:41 smithi086 ceph-mon[32965]: osdmap e197: 8 total, 8 up, 6 in 2024-03-26T03:56:41.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:41 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:56:41.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:41 smithi006 ceph-mon[37501]: osdmap e197: 8 total, 8 up, 6 in 2024-03-26T03:56:41.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:41 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:56:41.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:41 smithi006 ceph-mon[29446]: osdmap e197: 8 total, 8 up, 6 in 2024-03-26T03:56:41.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:41.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:41.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:41.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:41.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:41.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:41.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:41.868 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:41.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:41.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:42.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:42.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:42.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:42.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:42.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:42.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.297 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:42.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:42 smithi086 ceph-mon[32965]: pgmap v341: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 22248/334320 objects degraded (6.655%); 15 MiB/s, 243 objects/s recovering 2024-03-26T03:56:42.361 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:42 smithi006 ceph-mon[37501]: pgmap v341: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 22248/334320 objects degraded (6.655%); 15 MiB/s, 243 objects/s recovering 2024-03-26T03:56:42.361 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:42 smithi006 ceph-mon[29446]: pgmap v341: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 759 op/s; 22248/334320 objects degraded (6.655%); 15 MiB/s, 243 objects/s recovering 2024-03-26T03:56:42.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:42.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:42.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:42.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:42.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:42.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:42.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:42.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:42.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:42.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:42.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:42.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:42.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:43.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:43.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:43.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:43.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:43.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:43.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:43.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:43.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:43.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:43.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:43.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:43.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:43.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:43.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:43.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:44.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:44.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:44.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:44.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:44.203 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:44.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:44.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:44.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:44.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:44.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:44.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:44.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[37501]: pgmap v342: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 619 op/s; 22248/334320 objects degraded (6.655%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:44.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 22248/334320 objects degraded (6.655%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:44.587 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[37501]: osdmap e198: 8 total, 8 up, 6 in 2024-03-26T03:56:44.587 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[29446]: pgmap v342: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 619 op/s; 22248/334320 objects degraded (6.655%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:44.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 22248/334320 objects degraded (6.655%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:44.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-mon[29446]: osdmap e198: 8 total, 8 up, 6 in 2024-03-26T03:56:44.588 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:44.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:44.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:44.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:44.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:44 smithi086 ceph-mon[32965]: pgmap v342: 148 pgs: 1 peering, 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 139 active+clean; 6.8 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 696 B/s rd, 0 B/s wr, 619 op/s; 22248/334320 objects degraded (6.655%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 22248/334320 objects degraded (6.655%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:44 smithi086 ceph-mon[32965]: osdmap e198: 8 total, 8 up, 6 in 2024-03-26T03:56:44.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:44.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:44.860 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:44.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:44.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:44.949 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:56:44.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:44.998 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:56:45.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:45.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:45.037 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:56:45.079 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:56:45.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:45.120 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:56:45.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:45.162 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:56:45.202 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:56:45.254 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:56:45.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:56:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:56:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:56:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:56:45.262 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:56:45.263 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:56:45.263 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:56:45.263 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:56:45.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:45.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:45.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:45.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:45.564 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:45.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:45.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:45.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:45.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:45.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2999547307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:45.709 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:45.709 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2999547307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:45.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2999547307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:56:45.853 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:45.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:45.989 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:45.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:46.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:46.123 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:46.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:46.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:46.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:46.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:46.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:46.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:46.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:46.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:46.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:46.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:46.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:46 smithi086 ceph-mon[32965]: pgmap v344: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.52k op/s; 21361/319161 objects degraded (6.693%); 23 MiB/s, 370 objects/s recovering 2024-03-26T03:56:46.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:46 smithi006 ceph-mon[37501]: pgmap v344: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.52k op/s; 21361/319161 objects degraded (6.693%); 23 MiB/s, 370 objects/s recovering 2024-03-26T03:56:46.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:46 smithi006 ceph-mon[29446]: pgmap v344: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.52k op/s; 21361/319161 objects degraded (6.693%); 23 MiB/s, 370 objects/s recovering 2024-03-26T03:56:46.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:46.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:46.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:46.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:47.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:47.083 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:56:47.083 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:56:47.083 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-03-26T03:56:47.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:56:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:47.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:47.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:47.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:47.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:47.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:47.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:47.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:47.537 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:47.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:47.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:47.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:47.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:47.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:47.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:47.949 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:47.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:48.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:48.090 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:48.194 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:48.219 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:48.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:48.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:48.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:48.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:56:48.457 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:48.531 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:48.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:48.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:48.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:48.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:48.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:48.814 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:48.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:48 smithi086 ceph-mon[32965]: pgmap v345: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 21361/319161 objects degraded (6.693%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:48.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:48 smithi006 ceph-mon[37501]: pgmap v345: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 21361/319161 objects degraded (6.693%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:48.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:48 smithi006 ceph-mon[29446]: pgmap v345: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.5 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s; 21361/319161 objects degraded (6.693%); 12 MiB/s, 198 objects/s recovering 2024-03-26T03:56:48.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:48.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:48.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:48.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:49.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:49.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.102 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:49.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:49.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:49.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:49.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:49.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:49.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:49.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:49.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:49.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:49.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 21361/319161 objects degraded (6.693%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:49.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 21361/319161 objects degraded (6.693%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:49.857 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 21361/319161 objects degraded (6.693%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:49.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:49.909 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:49.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:49.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:50.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:50.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:50.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:50.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:50.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:50.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:50.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:50.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:50.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:50.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:50.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:50.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:50.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:50.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:50.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:50.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.725 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:50.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:50.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:50 smithi086 ceph-mon[32965]: pgmap v346: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 21361/316149 objects degraded (6.757%); 5.9 MiB/s, 95 objects/s recovering 2024-03-26T03:56:50.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:50 smithi006 ceph-mon[37501]: pgmap v346: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 21361/316149 objects degraded (6.757%); 5.9 MiB/s, 95 objects/s recovering 2024-03-26T03:56:50.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:50 smithi006 ceph-mon[29446]: pgmap v346: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 650 op/s; 21361/316149 objects degraded (6.757%); 5.9 MiB/s, 95 objects/s recovering 2024-03-26T03:56:50.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:50.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:50.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:50.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:51.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:51.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:51.143 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:51.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:51.285 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:51.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:51.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:51.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:51.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:51.545 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:51.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:51.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:51.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:51.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:51.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:51.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:51.961 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:51.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:52.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:52.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:52.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:52.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:52.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:52.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:52.329 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:56:52.329 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:56:52.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:52.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.431 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:52.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:52.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:52.603 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:52.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:52 smithi006 ceph-mon[29446]: pgmap v347: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:52.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:56:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:52 smithi006 ceph-mon[37501]: pgmap v347: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:56:52.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:52.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:52 smithi086 ceph-mon[32965]: pgmap v347: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:56:52.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:52.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:52.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:52.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:52.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:53.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:53.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.175 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:53.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:53.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:56:53.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:56:53.341 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:53.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T03:56:53.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:53.449 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:56:53.526 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:53.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:53.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:53.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:53.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:53.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:53.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:53.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:53.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:53.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:56:53.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:53.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:53.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:54.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:54.081 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:54.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:54.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:54.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:54.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:54.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:54.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:54.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:54.628 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:54.641 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:56:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: pgmap v348: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: osdmap e199: 8 total, 8 up, 6 in 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:56:54.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 20486/303273 objects degraded (6.755%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[37501]: osdmap e200: 8 total, 8 up, 6 in 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: pgmap v348: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: osdmap e199: 8 total, 8 up, 6 in 2024-03-26T03:56:54.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 20486/303273 objects degraded (6.755%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:54.644 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:54 smithi006 ceph-mon[29446]: osdmap e200: 8 total, 8 up, 6 in 2024-03-26T03:56:54.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:54.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:54.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:54.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: pgmap v348: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s; 20486/303273 objects degraded (6.755%); 11 MiB/s, 176 objects/s recovering 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/564407389' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: osdmap e199: 8 total, 8 up, 6 in 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:56:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:56:54.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:56:54.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:56:54.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 20486/303273 objects degraded (6.755%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:54.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:54 smithi086 ceph-mon[32965]: osdmap e200: 8 total, 8 up, 6 in 2024-03-26T03:56:54.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:54.913 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:54.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:55.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:55.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:55.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:55.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:55.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:55.328 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:55.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:55.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:55.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:55.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:55.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:55.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:55.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:55.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:55.848 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:55.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:56:55.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:55.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:55.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:56.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:56.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:56.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:56.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:56.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:56.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:56.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:56.401 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:56.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:56.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:56.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:56.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:56.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:56.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:56 smithi006 ceph-mon[37501]: pgmap v351: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:56.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:56 smithi006 ceph-mon[29446]: pgmap v351: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:56.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:56.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:56.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:56:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:56.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:56:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:56 smithi086 ceph-mon[32965]: pgmap v351: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:56.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:56.940 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:56.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:56.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:57.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:57.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:57.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:57.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:57.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:57.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:57.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:57.341 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:57.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:57.483 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:57.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:57.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:57.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:57.759 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:57.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:57.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:57.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:58.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:58.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:58.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:58.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:58.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:58.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:58.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:58.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:58.569 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:58.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:58.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:56:58.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:58 smithi086 ceph-mon[32965]: pgmap v352: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:58.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:58 smithi006 ceph-mon[37501]: pgmap v352: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:58 smithi006 ceph-mon[29446]: pgmap v352: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.13k op/s; 19544/288924 objects degraded (6.764%); 14 MiB/s, 226 objects/s recovering 2024-03-26T03:56:58.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:58.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:58.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:58.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:59.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:59.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:59.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:56:59.257 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:59.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:56:59.274 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:56:59.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:56:59.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:56:59.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:56:59.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:56:59.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:56:59.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:56:59.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:56:59.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:56:59.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:56:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:59 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 19544/288924 objects degraded (6.764%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:56:59 smithi086 ceph-mon[32965]: osdmap e201: 8 total, 8 up, 6 in 2024-03-26T03:56:59.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:56:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:59 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 19544/288924 objects degraded (6.764%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:59.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:56:59 smithi006 ceph-mon[37501]: osdmap e201: 8 total, 8 up, 6 in 2024-03-26T03:56:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:59 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 19544/288924 objects degraded (6.764%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T03:56:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:56:59 smithi006 ceph-mon[29446]: osdmap e201: 8 total, 8 up, 6 in 2024-03-26T03:56:59.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:59.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:56:59.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:00.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:00.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:00.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:00.165 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:00.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:00.293 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:00.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:00.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:00.431 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:00.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:00.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:00.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:00.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:00.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:00.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:00.714 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:00 smithi006 ceph-mon[37501]: pgmap v354: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 950 op/s; 19544/286182 objects degraded (6.829%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:57:00.714 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:00 smithi006 ceph-mon[37501]: osdmap e202: 8 total, 8 up, 6 in 2024-03-26T03:57:00.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:00 smithi006 ceph-mon[29446]: pgmap v354: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 950 op/s; 19544/286182 objects degraded (6.829%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:57:00.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:00 smithi006 ceph-mon[29446]: osdmap e202: 8 total, 8 up, 6 in 2024-03-26T03:57:00.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:00.819 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:00 smithi086 ceph-mon[32965]: pgmap v354: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.8 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 950 op/s; 19544/286182 objects degraded (6.829%); 9.8 MiB/s, 156 objects/s recovering 2024-03-26T03:57:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:00 smithi086 ceph-mon[32965]: osdmap e202: 8 total, 8 up, 6 in 2024-03-26T03:57:00.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:00.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:00.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:01.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:01.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:01.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:01.205 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:01.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:01.295 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:01.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:01.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:01.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:01.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:01.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:01.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:01.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:01.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:01.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:01.742 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:01.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:01.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:01.882 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:01.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:01.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:01.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:02.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:02.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:57:02.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:02.169 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:02.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:02.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:02.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.309 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:02.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:02 smithi006 ceph-mon[37501]: osdmap e203: 8 total, 8 up, 6 in 2024-03-26T03:57:02.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:02 smithi006 ceph-mon[37501]: pgmap v357: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 18665/275361 objects degraded (6.778%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:57:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:02 smithi006 ceph-mon[29446]: osdmap e203: 8 total, 8 up, 6 in 2024-03-26T03:57:02.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:02 smithi006 ceph-mon[29446]: pgmap v357: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 18665/275361 objects degraded (6.778%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:57:02.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:02.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:02.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:02.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:02.576 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:02.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:02 smithi086 ceph-mon[32965]: osdmap e203: 8 total, 8 up, 6 in 2024-03-26T03:57:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:02 smithi086 ceph-mon[32965]: pgmap v357: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 754 op/s; 18665/275361 objects degraded (6.778%); 9.2 MiB/s, 146 objects/s recovering 2024-03-26T03:57:02.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:02.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:02.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:02.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:02.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:02.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:02.995 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:02.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:03.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:03.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.138 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:03.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:03.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:03.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:03.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:03.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:03.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:03.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:03.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:03.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:03.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:03.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:03.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:03.926 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:03.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:03.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:04.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:04.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:04.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:04.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:04.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:04.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:04.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:04.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:04.380 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:04.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:04.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:04.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:57:04.530 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:57:04.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:04.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:04.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[37501]: pgmap v358: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s; 18308/274194 objects degraded (6.677%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 18308/274194 objects degraded (6.677%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[37501]: osdmap e204: 8 total, 8 up, 6 in 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[29446]: pgmap v358: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s; 18308/274194 objects degraded (6.677%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 18308/274194 objects degraded (6.677%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:04 smithi006 ceph-mon[29446]: osdmap e204: 8 total, 8 up, 6 in 2024-03-26T03:57:04.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.721 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:04.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:04.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:04.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:04 smithi086 ceph-mon[32965]: pgmap v358: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 5.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 818 op/s; 18308/274194 objects degraded (6.677%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:04 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 18308/274194 objects degraded (6.677%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:04 smithi086 ceph-mon[32965]: osdmap e204: 8 total, 8 up, 6 in 2024-03-26T03:57:04.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:04.862 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:04.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:04.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.013 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:05.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:05.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:05.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:05.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:05.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:05.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:05.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:05.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:05.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:05.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:05.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:05.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:05 smithi006 ceph-mon[37501]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:57:05.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:05.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:05 smithi006 ceph-mon[29446]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:57:05.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:05.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:05.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.699 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:05.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:05.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:05 smithi086 ceph-mon[32965]: from='client.15156 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:57:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:05.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:05.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:05.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:05.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:05.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:05.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:06.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:06.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:06.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:06.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:06.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.255 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:06.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:06.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:06.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:06.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:06.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:06.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:06.617 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:57:06.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:06.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:06.668 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:57:06.713 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:57:06.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:06 smithi006 ceph-mon[29446]: pgmap v360: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 17287/264555 objects degraded (6.534%); 24 MiB/s, 376 objects/s recovering 2024-03-26T03:57:06.714 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:06 smithi006 ceph-mon[37501]: pgmap v360: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 17287/264555 objects degraded (6.534%); 24 MiB/s, 376 objects/s recovering 2024-03-26T03:57:06.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:06.752 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:57:06.792 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:57:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.810 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:06.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:06 smithi086 ceph-mon[32965]: pgmap v360: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.20k op/s; 17287/264555 objects degraded (6.534%); 24 MiB/s, 376 objects/s recovering 2024-03-26T03:57:06.858 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:57:06.906 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:57:06.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:06.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:06.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:06.949 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:57:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:57:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:57:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:57:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:57:06.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:57:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:57:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:57:06.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:57:07.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:07.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44831 7.7 7:e27452a6:::benchmark_data_smithi006_77260_object44830:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.204538+0000", 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.950306415, 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15058026999999999, 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.160 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44831 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204538+0000", 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204538+0000", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204540+0000", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4449999999999999e-06 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204562+0000", 2024-03-26T03:57:07.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2671e-05 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204574+0000", 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1622e-05 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.204581+0000", 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1219999999999997e-06 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205429+0000", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00084776600000000001 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205551+0000", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012258399999999999 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206516+0000", 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00096463199999999999 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208336+0000", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018195959999999999 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354946+0000", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14661073399999999 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355018+0000", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2108999999999997e-05 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355079+0000", 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0161000000000002e-05 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355118+0000", 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9827999999999999e-05 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44833 7.6 7:6193598d:::benchmark_data_smithi006_77260_object44832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.205002+0000", 2024-03-26T03:57:07.166 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94984274400002, 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15000279999999999, 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44833 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205002+0000", 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205002+0000", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205003+0000", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8299999999999996e-07 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205048+0000", 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5701000000000002e-05 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205056+0000", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9070000000000007e-06 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205091+0000", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4378999999999997e-05 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206565+0000", 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001474277 2024-03-26T03:57:07.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206682+0000", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117129 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206820+0000", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013763800000000001 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208418+0000", 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001598934 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352860+0000", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14444116300000001 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354903+0000", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002043628 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354959+0000", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6161e-05 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355005+0000", 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5099999999999998e-05 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44835 7.9 7:96ace094:::benchmark_data_smithi006_77260_object44834:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.205362+0000", 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94948235499999, 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14931718099999999, 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.172 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44835 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205362+0000", 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205362+0000", 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205363+0000", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0070000000000001e-06 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205445+0000", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1802e-05 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205456+0000", 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1056999999999999e-05 2024-03-26T03:57:07.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205463+0000", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4200000000000001e-06 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206018+0000", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055445099999999997 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206135+0000", 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116777 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206257+0000", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012265 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354475+0000", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14821806000000001 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354517+0000", 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1956999999999998e-05 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354529+0000", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1442e-05 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354633+0000", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000104128 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354679+0000", 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6430000000000001e-05 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44836 7.9 7:9e429f71:::benchmark_data_smithi006_77260_object44835:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.205481+0000", 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94936389899999, 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.155985188, 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44836 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.178 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205481+0000", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205481+0000", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205481+0000", 2024-03-26T03:57:07.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1500000000000004e-07 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205501+0000", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9205e-05 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205507+0000", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7490000000000001e-06 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.205513+0000", 2024-03-26T03:57:07.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6529999999999999e-06 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.207898+0000", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023855 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208023+0000", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000124447 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208287+0000", 2024-03-26T03:57:07.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026416500000000002 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354690+0000", 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14640284000000001 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354959+0000", 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026898100000000001 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.361357+0000", 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063979989999999997 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.361417+0000", 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9775000000000003e-05 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.361466+0000", 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9159000000000001e-05 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:44832 7.e e108/95 7:778b6221:::benchmark_data_smithi006_77260_object44831:head v 108'2734, mlcod=108'2732)", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.206217+0000", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94862796299998, 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.148650595, 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206217+0000", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206217+0000", 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206217+0000", 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.23e-07 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206241+0000", 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3339000000000001e-05 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206251+0000", 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9499999999999996e-06 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206257+0000", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6789999999999999e-06 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206400+0000", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014309500000000001 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206420+0000", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9607999999999999e-05 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354834+0000", 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.148414195 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354867+0000", 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3006000000000002e-05 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:44838 7.f 7:f0e0816e:::benchmark_data_smithi006_77260_object44837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.206609+0000", 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94823573899998, 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14897956700000001, 2024-03-26T03:57:07.188 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 44838 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206609+0000", 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206609+0000", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206610+0000", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04e-06 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206672+0000", 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2681999999999994e-05 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206682+0000", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7489999999999993e-06 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206690+0000", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9899999999999997e-06 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206747+0000", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7229999999999999e-05 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206858+0000", 2024-03-26T03:57:07.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110152 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.206982+0000", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012424300000000001 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354745+0000", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.147762959 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354769+0000", 2024-03-26T03:57:07.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4173e-05 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355038+0000", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026921699999999998 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355546+0000", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00050781199999999996 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.355588+0000", 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2320000000000001e-05 2024-03-26T03:57:07.193 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:44835 7.9 e108/95 ondisk, result = 0)", 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.208077+0000", 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94676703099998, 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14640866299999999, 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208077+0000", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208077+0000", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208078+0000", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.61e-07 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208083+0000", 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6269999999999999e-06 2024-03-26T03:57:07.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208092+0000", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4239999999999999e-06 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208099+0000", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8260000000000004e-06 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354460+0000", 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.146360671 2024-03-26T03:57:07.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354469+0000", 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0299999999999999e-06 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354486+0000", 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7323999999999999e-05 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.24704.0:44837 7.d e108/96 7:b7237241:::benchmark_data_smithi006_77260_object44836:head v 108'2750, mlcod=108'2747)", 2024-03-26T03:57:07.197 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.208212+0000", 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94663262099999, 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14653396499999999, 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208212+0000", 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208212+0000", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208212+0000", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9500000000000002e-07 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208233+0000", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0304000000000001e-05 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208241+0000", 2024-03-26T03:57:07.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7789999999999996e-06 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208246+0000", 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8470000000000001e-06 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208380+0000", 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013361100000000001 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208401+0000", 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0803999999999999e-05 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354709+0000", 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.146307941 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354746+0000", 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7083999999999999e-05 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:44838 7.f e108/95 ondisk, result = 0)", 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.208482+0000", 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.94636292400003, 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.146270601, 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208482+0000", 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208482+0000", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208482+0000", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2800000000000002e-07 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208486+0000", 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.613e-06 2024-03-26T03:57:07.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208494+0000", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6649999999999992e-06 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208501+0000", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4139999999999999e-06 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354732+0000", 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14623122299999999 2024-03-26T03:57:07.204 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354739+0000", 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2810000000000003e-06 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354752+0000", 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2977e-05 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.205 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.24704.0:44835 7.9 e108/95 ondisk, result = 0)", 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:21.208545+0000", 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 345.946299184, 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14613778799999999, 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208545+0000", 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.206 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208545+0000", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208546+0000", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2200000000000004e-07 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.208549+0000", 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1389999999999999e-06 2024-03-26T03:57:07.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352718+0000", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.144169466 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.352731+0000", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2849e-05 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354507+0000", 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001775616 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354513+0000", 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0039999999999996e-06 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:21.354683+0000", 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000170192 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46147 7.f 7:f549e31b:::benchmark_data_smithi006_77260_object46146:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:57:07.209 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.442388+0000", 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.71245647299997, 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14495195699999999, 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46147 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.210 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.442388+0000", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.442388+0000", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.442389+0000", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4900000000000004e-07 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.211 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.442413+0000", 2024-03-26T03:57:07.212 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4032e-05 2024-03-26T03:57:07.212 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.212 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.212 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.563278+0000", 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.12086509099999999 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.563316+0000", 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.82e-05 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.213 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.580294+0000", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016977900000000001 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.580401+0000", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010673 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.580524+0000", 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012344499999999999 2024-03-26T03:57:07.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.584251+0000", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003726806 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.586513+0000", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022613490000000002 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.587224+0000", 2024-03-26T03:57:07.215 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00071133100000000005 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.587293+0000", 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9350000000000005e-05 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.587340+0000", 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6774000000000003e-05 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.216 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46477 7.7 7:edd3f9ef:::benchmark_data_smithi006_77260_object46476:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.780223+0000", 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.37462114599998, 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15478266399999999, 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46477 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.217 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780223+0000", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780223+0000", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780225+0000", 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3349999999999999e-06 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.218 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780254+0000", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9771999999999999e-05 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780265+0000", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0813e-05 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.780273+0000", 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1869999999999997e-06 2024-03-26T03:57:07.219 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785739+0000", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0054654609999999996 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785819+0000", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0332000000000003e-05 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785965+0000", 2024-03-26T03:57:07.220 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014552099999999999 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790352+0000", 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043869720000000003 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790538+0000", 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00018637999999999999 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.221 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934848+0000", 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.144310036 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934955+0000", 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106634 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935006+0000", 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1221000000000003e-05 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.222 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46479 7.7 7:e04d6df7:::benchmark_data_smithi006_77260_object46478:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.784714+0000", 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.37013028199999, 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15095299600000001, 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46479 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.223 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784714+0000", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784714+0000", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784716+0000", 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.305e-06 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.224 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784737+0000", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1172e-05 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784747+0000", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0526999999999999e-05 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.784755+0000", 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4309999999999997e-06 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.225 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.787932+0000", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003177119 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.788202+0000", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027040199999999999 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.788347+0000", 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000144346 2024-03-26T03:57:07.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935056+0000", 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14670970699999999 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935141+0000", 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5067000000000004e-05 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.227 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935547+0000", 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00040540900000000002 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935628+0000", 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1593000000000003e-05 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935667+0000", 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8918000000000002e-05 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46483 7.7 7:e3a68a80:::benchmark_data_smithi006_77260_object46482:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.785439+0000", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.36940546400001, 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15034114700000001, 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46483 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.229 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785439+0000", 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785439+0000", 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785440+0000", 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6300000000000005e-07 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785524+0000", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4380000000000002e-05 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785533+0000", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1749999999999994e-06 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.785540+0000", 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2840000000000001e-06 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.231 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790565+0000", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050250599999999996 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790671+0000", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010532800000000001 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790795+0000", 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012378399999999999 2024-03-26T03:57:07.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935101+0000", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.144306604 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935181+0000", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0142000000000006e-05 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935675+0000", 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00049343399999999995 2024-03-26T03:57:07.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935746+0000", 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1074000000000004e-05 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935780+0000", 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4319000000000001e-05 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46487 7.7 7:e0beb510:::benchmark_data_smithi006_77260_object46486:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.789486+0000", 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.36535820900002, 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15730486399999999, 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.235 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46487 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789486+0000", 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789486+0000", 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789488+0000", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.677e-06 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789543+0000", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5408000000000002e-05 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789556+0000", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2529e-05 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789565+0000", 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4059999999999994e-06 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792899+0000", 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033332840000000002 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793111+0000", 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021214699999999999 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,3", 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793288+0000", 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017771799999999999 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935534+0000", 2024-03-26T03:57:07.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.142245911 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943806+0000", 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082715459999999994 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946629+0000", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002822794 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946746+0000", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011715899999999999 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946791+0000", 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5284999999999998e-05 2024-03-26T03:57:07.241 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46488 7.6 7:6828ee54:::benchmark_data_smithi006_77260_object46487:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.789585+0000", 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.36525930800002, 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.1451105, 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46488 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789585+0000", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789585+0000", 2024-03-26T03:57:07.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789586+0000", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1799999999999995e-07 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789628+0000", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2400000000000001e-05 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789637+0000", 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7080000000000003e-06 2024-03-26T03:57:07.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.789643+0000", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5729999999999996e-06 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790145+0000", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00050174900000000001 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790266+0000", 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012094100000000001 2024-03-26T03:57:07.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.790552+0000", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00028578000000000002 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793174+0000", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026221249999999999 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.793445+0000", 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00027141300000000003 2024-03-26T03:57:07.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934119+0000", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.140673251 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934639+0000", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00052059899999999995 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.934696+0000", 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6342999999999999e-05 2024-03-26T03:57:07.247 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46493 7.f 7:f7346e28:::benchmark_data_smithi006_77260_object46492:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.791936+0000", 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.36290845299999, 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.15461847200000001, 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46493 2024-03-26T03:57:07.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791936+0000", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791936+0000", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791937+0000", 2024-03-26T03:57:07.249 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.297e-06 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791961+0000", 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3377999999999999e-05 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791972+0000", 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.093e-05 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.791979+0000", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7139999999999993e-06 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792706+0000", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00072618499999999996 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792817+0000", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111526 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.792945+0000", 2024-03-26T03:57:07.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012836 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935220+0000", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.14227501000000001 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943797+0000", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.008576129 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946448+0000", 2024-03-26T03:57:07.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026516420000000001 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946513+0000", 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4953e-05 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946555+0000", 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1347999999999998e-05 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24704.0:46494 7.f 7:f20dd325:::benchmark_data_smithi006_77260_object46493:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:51:22.794260+0000", 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 344.36058441, 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.152771875, 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24704", 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 46494 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.254 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794260+0000", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794260+0000", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794261+0000", 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.361e-06 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.794285+0000", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3693000000000002e-05 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933478+0000", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.13919266899999999 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.933493+0000", 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5099000000000001e-05 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935256+0000", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017635470000000001 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.935377+0000", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012082 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.936045+0000", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00066808000000000004 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943301+0000", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0072554769999999998 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.943817+0000", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051601000000000001 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946694+0000", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028767189999999998 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:07.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.946817+0000", 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000123769 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:51:22.947032+0000", 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021463100000000001 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1969 5.3 5:c609908c:::notify.5:head [watch ping cookie 94232629225728 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136387+0000", 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 158.01845709400001, 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.30655227899999998, 2024-03-26T03:57:07.259 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1969 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136387+0000", 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136387+0000", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136389+0000", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2550000000000001e-06 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136394+0000", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4750000000000001e-06 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136408+0000", 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4248e-05 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136421+0000", 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2261e-05 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136554+0000", 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013327 2024-03-26T03:57:07.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136642+0000", 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8004999999999995e-05 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401102+0000", 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.26446012499999999 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401407+0000", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030506799999999997 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442774+0000", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041367379000000003 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442858+0000", 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4009000000000002e-05 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442940+0000", 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1184000000000001e-05 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1979 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94232629225728 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136815+0000", 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 158.01802937900001, 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32657255899999998, 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:07.265 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1979 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136815+0000", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136815+0000", 2024-03-26T03:57:07.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136816+0000", 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7899999999999997e-07 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136820+0000", 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3789999999999999e-06 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:07.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136829+0000", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7630000000000006e-06 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136836+0000", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3540000000000002e-06 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136897+0000", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0992999999999997e-05 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136910+0000", 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2948999999999999e-05 2024-03-26T03:57:07.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401439+0000", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.26452896599999998 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401489+0000", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9790000000000003e-05 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442966+0000", 2024-03-26T03:57:07.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041476987 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.443039+0000", 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2958000000000002e-05 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.463388+0000", 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020348641000000001 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.271 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:07.271 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:07.271 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:57:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:07.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:07.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:07.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:07.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:07.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:07.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:07.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:07.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:07 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/570563563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:07.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:07 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/570563563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:07.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:07.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:07.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:07.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:07.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:07.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:07 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/570563563' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:07.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:07.977 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:07.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:08.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:08.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:08.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:08.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:08.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:08.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:08.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:08.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:08.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:08.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:08.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.646 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:08.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:08.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:08.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:08.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:08 smithi086 ceph-mon[32965]: pgmap v361: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 728 B/s rd, 0 B/s wr, 555 op/s; 17287/263655 objects degraded (6.557%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:57:08.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:08 smithi006 ceph-mon[37501]: pgmap v361: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 728 B/s rd, 0 B/s wr, 555 op/s; 17287/263655 objects degraded (6.557%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:57:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:08 smithi006 ceph-mon[29446]: pgmap v361: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 728 B/s rd, 0 B/s wr, 555 op/s; 17287/263655 objects degraded (6.557%); 12 MiB/s, 195 objects/s recovering 2024-03-26T03:57:08.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:08.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:08.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:09.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:09.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:09.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:09.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:09.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:09.334 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:09.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:09.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:09.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:09.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:09.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:09.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:09.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:09.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:09.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:09 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 17287/263655 objects degraded (6.557%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:09.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:09.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:09.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:09 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 17287/263655 objects degraded (6.557%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:09 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 17287/263655 objects degraded (6.557%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:09.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:10.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:10.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:10.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:10.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:10.286 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:10.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:10.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:10.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:10.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:10.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:10.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:10.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.681 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:10.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:10.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:10 smithi086 ceph-mon[32965]: pgmap v362: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 606 op/s; 17287/260769 objects degraded (6.629%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:57:10.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:10 smithi006 ceph-mon[37501]: pgmap v362: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 606 op/s; 17287/260769 objects degraded (6.629%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:57:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:10 smithi006 ceph-mon[29446]: pgmap v362: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 606 op/s; 17287/260769 objects degraded (6.629%); 11 MiB/s, 171 objects/s recovering 2024-03-26T03:57:10.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:10.960 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:10.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:11.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:11.095 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:11.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:11.232 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:11.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:11.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:11.375 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:11.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:11.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:11.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:11.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:11.649 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:11.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:11.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:11.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:11.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:11.915 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:11.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:12.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:12.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:12.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:12.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:12.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:12.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:12.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:12.319 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:12.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:12.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:12.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:12.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:12.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:12.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:12.728 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:12.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:12 smithi086 ceph-mon[32965]: pgmap v363: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s; 16575/248934 objects degraded (6.658%); 13 MiB/s, 208 objects/s recovering 2024-03-26T03:57:12.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:12.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:12 smithi006 ceph-mon[37501]: pgmap v363: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s; 16575/248934 objects degraded (6.658%); 13 MiB/s, 208 objects/s recovering 2024-03-26T03:57:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:12 smithi006 ceph-mon[29446]: pgmap v363: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 881 op/s; 16575/248934 objects degraded (6.658%); 13 MiB/s, 208 objects/s recovering 2024-03-26T03:57:12.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:12.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:12.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:13.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:13.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:13.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:13.254 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:13.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:13.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:13.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:13.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:13.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:13.518 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:13.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:13.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:13.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:13.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:13.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:13.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:13.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:13.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:14.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:14.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:14.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:14.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:14.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:14.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.320 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:14.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:14.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:14.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:14.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:14.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:14.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:14.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:14 smithi006 ceph-mon[37501]: pgmap v364: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16575/248067 objects degraded (6.682%); 11 MiB/s, 173 objects/s recovering 2024-03-26T03:57:14.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:14 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 16575/248067 objects degraded (6.682%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:14 smithi006 ceph-mon[29446]: pgmap v364: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16575/248067 objects degraded (6.682%); 11 MiB/s, 173 objects/s recovering 2024-03-26T03:57:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:14 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 16575/248067 objects degraded (6.682%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:14.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:14.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:14.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:14 smithi086 ceph-mon[32965]: pgmap v364: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 5.0 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16575/248067 objects degraded (6.682%); 11 MiB/s, 173 objects/s recovering 2024-03-26T03:57:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:14 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 16575/248067 objects degraded (6.682%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:14.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:14.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:14.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:14.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:14.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:15.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:15.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:15.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:15.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:15.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:15.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:15.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:15.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:15.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:15.557 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:15.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.683 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:15.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:15.813 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:15.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:15.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:15.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:15.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:16.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:16.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:16.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:16.205 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:16.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:16.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.355 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:16.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:16.383 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-03-26T03:57:04.315347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5975316928,"num_objects":91398,"num_object_clones":0,"num_object_copies":274194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91398,"num_whiteouts":0,"num_read":1708,"num_read_kb":1522,"num_write":169497,"num_write_kb":8317854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10730,"num_bytes_recovered":693115127,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64476,"ondisk_log_size":64476,"up":444,"acting":437,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":18013956,"kb_used_data":16370884,"kb_used_omap":12,"kb_used_meta":1642931,"kb_avail":544456956,"statfs":{"total":575970213888,"available":557523922944,"internally_reserved":0,"allocated":16763785216,"data_stored":16759838357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12418,"internal_metadata":1682362238},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-321781760,"num_objects":-4910,"num_object_clones":0,"num_object_copies":-14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4910,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4910,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":81002496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.004459"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312687+0000","last_change":"2024-03-26T03:55:59.300239+0000","last_active":"2024-03-26T03:56:59.312687+0000","last_peered":"2024-03-26T03:56:59.312687+0000","last_clean":"2024-03-26T03:56:59.312687+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T03:56:59.312687+0000","last_undegraded":"2024-03-26T03:56:59.312687+0000","last_fullsized":"2024-03-26T03:56:59.312687+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312288+0000","last_change":"2024-03-26T03:55:59.291583+0000","last_active":"2024-03-26T03:56:59.312288+0000","last_peered":"2024-03-26T03:56:59.312288+0000","last_clean":"2024-03-26T03:56:59.312288+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T03:56:59.312288+0000","last_undegraded":"2024-03-26T03:56:59.312288+0000","last_fullsized":"2024-03-26T03:56:59.312288+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":317,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331527+0000","last_change":"2024-03-26T03:55:59.307852+0000","last_active":"2024-03-26T03:56:59.331527+0000","last_peered":"2024-03-26T03:56:59.331527+0000","last_clean":"2024-03-26T03:56:59.331527+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T03:56:59.331527+0000","last_undegraded":"2024-03-26T03:56:59.331527+0000","last_fullsized":"2024-03-26T03:56:59.331527+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":368,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.311990+0000","last_change":"2024-03-26T03:55:32.246310+0000","last_active":"2024-03-26T03:56:59.311990+0000","last_peered":"2024-03-26T03:56:59.311990+0000","last_clean":"2024-03-26T03:56:59.311990+0000","last_became_active":"2024-03-26T03:54:29.438463+0000","last_became_peered":"2024-03-26T03:54:29.438463+0000","last_unstale":"2024-03-26T03:56:59.311990+0000","last_undegraded":"2024-03-26T03:56:59.311990+0000","last_fullsized":"2024-03-26T03:56:59.311990+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":185,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.318984+0000","last_change":"2024-03-26T03:55:20.740072+0000","last_active":"2024-03-26T03:57:01.318984+0000","last_peered":"2024-03-26T03:57:01.318984+0000","last_clean":"2024-03-26T03:57:01.318984+0000","last_became_active":"2024-03-26T03:54:29.357724+0000","last_became_peered":"2024-03-26T03:54:29.357724+0000","last_unstale":"2024-03-26T03:57:01.318984+0000","last_undegraded":"2024-03-26T03:57:01.318984+0000","last_fullsized":"2024-03-26T03:57:01.318984+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.311881+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T03:56:59.311881+0000","last_peered":"2024-03-26T03:56:59.311881+0000","last_clean":"2024-03-26T03:56:59.311881+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:56:59.311881+0000","last_undegraded":"2024-03-26T03:56:59.311881+0000","last_fullsized":"2024-03-26T03:56:59.311881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:14.310169+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":193,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.321964+0000","last_change":"2024-03-26T03:55:56.852293+0000","last_active":"2024-03-26T03:56:59.321964+0000","last_peered":"2024-03-26T03:56:59.321964+0000","last_clean":"2024-03-26T03:56:59.321964+0000","last_became_active":"2024-03-26T03:54:29.377365+0000","last_became_peered":"2024-03-26T03:54:29.377365+0000","last_unstale":"2024-03-26T03:56:59.321964+0000","last_undegraded":"2024-03-26T03:56:59.321964+0000","last_fullsized":"2024-03-26T03:56:59.321964+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:26.353468+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":232,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.320300+0000","last_change":"2024-03-26T03:55:59.310364+0000","last_active":"2024-03-26T03:56:59.320300+0000","last_peered":"2024-03-26T03:56:59.320300+0000","last_clean":"2024-03-26T03:56:59.320300+0000","last_became_active":"2024-03-26T03:55:59.309970+0000","last_became_peered":"2024-03-26T03:55:59.309970+0000","last_unstale":"2024-03-26T03:56:59.320300+0000","last_undegraded":"2024-03-26T03:56:59.320300+0000","last_fullsized":"2024-03-26T03:56:59.320300+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.340155+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T03:56:59.340155+0000","last_peered":"2024-03-26T03:56:59.340155+0000","last_clean":"2024-03-26T03:56:59.340155+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:56:59.340155+0000","last_undegraded":"2024-03-26T03:56:59.340155+0000","last_fullsized":"2024-03-26T03:56:59.340155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:56:59.995323+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":294,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327279+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T03:57:01.327279+0000","last_peered":"2024-03-26T03:57:01.327279+0000","last_clean":"2024-03-26T03:57:01.327279+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T03:57:01.327279+0000","last_undegraded":"2024-03-26T03:57:01.327279+0000","last_fullsized":"2024-03-26T03:57:01.327279+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326890+0000","last_change":"2024-03-26T03:55:59.304086+0000","last_active":"2024-03-26T03:57:01.326890+0000","last_peered":"2024-03-26T03:57:01.326890+0000","last_clean":"2024-03-26T03:57:01.326890+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T03:57:01.326890+0000","last_undegraded":"2024-03-26T03:57:01.326890+0000","last_fullsized":"2024-03-26T03:57:01.326890+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":365,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312973+0000","last_change":"2024-03-26T03:54:29.551157+0000","last_active":"2024-03-26T03:56:59.312973+0000","last_peered":"2024-03-26T03:56:59.312973+0000","last_clean":"2024-03-26T03:56:59.312973+0000","last_became_active":"2024-03-26T03:54:29.444189+0000","last_became_peered":"2024-03-26T03:54:29.444189+0000","last_unstale":"2024-03-26T03:56:59.312973+0000","last_undegraded":"2024-03-26T03:56:59.312973+0000","last_fullsized":"2024-03-26T03:56:59.312973+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":264,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326451+0000","last_change":"2024-03-26T03:55:59.298016+0000","last_active":"2024-03-26T03:57:01.326451+0000","last_peered":"2024-03-26T03:57:01.326451+0000","last_clean":"2024-03-26T03:57:01.326451+0000","last_became_active":"2024-03-26T03:55:59.297675+0000","last_became_peered":"2024-03-26T03:55:59.297675+0000","last_unstale":"2024-03-26T03:57:01.326451+0000","last_undegraded":"2024-03-26T03:57:01.326451+0000","last_fullsized":"2024-03-26T03:57:01.326451+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":304,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319149+0000","last_change":"2024-03-26T03:54:11.544712+0000","last_active":"2024-03-26T03:57:01.319149+0000","last_peered":"2024-03-26T03:57:01.319149+0000","last_clean":"2024-03-26T03:57:01.319149+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:57:01.319149+0000","last_undegraded":"2024-03-26T03:57:01.319149+0000","last_fullsized":"2024-03-26T03:57:01.319149+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.544637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.544637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326663+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T03:57:01.326663+0000","last_peered":"2024-03-26T03:57:01.326663+0000","last_clean":"2024-03-26T03:57:01.326663+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T03:57:01.326663+0000","last_undegraded":"2024-03-26T03:57:01.326663+0000","last_fullsized":"2024-03-26T03:57:01.326663+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":343,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313073+0000","last_change":"2024-03-26T03:55:18.237943+0000","last_active":"2024-03-26T03:56:59.313073+0000","last_peered":"2024-03-26T03:56:59.313073+0000","last_clean":"2024-03-26T03:56:59.313073+0000","last_became_active":"2024-03-26T03:54:29.360309+0000","last_became_peered":"2024-03-26T03:54:29.360309+0000","last_unstale":"2024-03-26T03:56:59.313073+0000","last_undegraded":"2024-03-26T03:56:59.313073+0000","last_fullsized":"2024-03-26T03:56:59.313073+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":310,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312714+0000","last_change":"2024-03-26T03:55:59.295858+0000","last_active":"2024-03-26T03:56:59.312714+0000","last_peered":"2024-03-26T03:56:59.312714+0000","last_clean":"2024-03-26T03:56:59.312714+0000","last_became_active":"2024-03-26T03:55:59.295544+0000","last_became_peered":"2024-03-26T03:55:59.295544+0000","last_unstale":"2024-03-26T03:56:59.312714+0000","last_undegraded":"2024-03-26T03:56:59.312714+0000","last_fullsized":"2024-03-26T03:56:59.312714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":299,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.315428+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T03:56:59.315428+0000","last_peered":"2024-03-26T03:56:59.315428+0000","last_clean":"2024-03-26T03:56:59.315428+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:56:59.315428+0000","last_undegraded":"2024-03-26T03:56:59.315428+0000","last_fullsized":"2024-03-26T03:56:59.315428+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":316,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331231+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T03:56:59.331231+0000","last_peered":"2024-03-26T03:56:59.331231+0000","last_clean":"2024-03-26T03:56:59.331231+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:56:59.331231+0000","last_undegraded":"2024-03-26T03:56:59.331231+0000","last_fullsized":"2024-03-26T03:56:59.331231+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:05.710428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":367,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.328765+0000","last_change":"2024-03-26T03:55:30.771720+0000","last_active":"2024-03-26T03:56:59.328765+0000","last_peered":"2024-03-26T03:56:59.328765+0000","last_clean":"2024-03-26T03:56:59.328765+0000","last_became_active":"2024-03-26T03:54:29.346374+0000","last_became_peered":"2024-03-26T03:54:29.346374+0000","last_unstale":"2024-03-26T03:56:59.328765+0000","last_undegraded":"2024-03-26T03:56:59.328765+0000","last_fullsized":"2024-03-26T03:56:59.328765+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:30.771573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.771573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":134,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327403+0000","last_change":"2024-03-26T03:55:59.320775+0000","last_active":"2024-03-26T03:57:01.327403+0000","last_peered":"2024-03-26T03:57:01.327403+0000","last_clean":"2024-03-26T03:57:01.327403+0000","last_became_active":"2024-03-26T03:55:59.318416+0000","last_became_peered":"2024-03-26T03:55:59.318416+0000","last_unstale":"2024-03-26T03:57:01.327403+0000","last_undegraded":"2024-03-26T03:57:01.327403+0000","last_fullsized":"2024-03-26T03:57:01.327403+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313123+0000","last_change":"2024-03-26T03:54:29.360637+0000","last_active":"2024-03-26T03:56:59.313123+0000","last_peered":"2024-03-26T03:56:59.313123+0000","last_clean":"2024-03-26T03:56:59.313123+0000","last_became_active":"2024-03-26T03:54:29.360289+0000","last_became_peered":"2024-03-26T03:54:29.360289+0000","last_unstale":"2024-03-26T03:56:59.313123+0000","last_undegraded":"2024-03-26T03:56:59.313123+0000","last_fullsized":"2024-03-26T03:56:59.313123+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313080+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:56:59.313080+0000","last_peered":"2024-03-26T03:56:59.313080+0000","last_clean":"2024-03-26T03:56:59.313080+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:56:59.313080+0000","last_undegraded":"2024-03-26T03:56:59.313080+0000","last_fullsized":"2024-03-26T03:56:59.313080+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":184,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312142+0000","last_change":"2024-03-26T03:55:59.312779+0000","last_active":"2024-03-26T03:56:59.312142+0000","last_peered":"2024-03-26T03:56:59.312142+0000","last_clean":"2024-03-26T03:56:59.312142+0000","last_became_active":"2024-03-26T03:55:59.312386+0000","last_became_peered":"2024-03-26T03:55:59.312386+0000","last_unstale":"2024-03-26T03:56:59.312142+0000","last_undegraded":"2024-03-26T03:56:59.312142+0000","last_fullsized":"2024-03-26T03:56:59.312142+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.320025+0000","last_change":"2024-03-26T03:55:59.300737+0000","last_active":"2024-03-26T03:57:01.320025+0000","last_peered":"2024-03-26T03:57:01.320025+0000","last_clean":"2024-03-26T03:57:01.320025+0000","last_became_active":"2024-03-26T03:55:59.299335+0000","last_became_peered":"2024-03-26T03:55:59.299335+0000","last_unstale":"2024-03-26T03:57:01.320025+0000","last_undegraded":"2024-03-26T03:57:01.320025+0000","last_fullsized":"2024-03-26T03:57:01.320025+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:54:15.688518+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.688518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":348,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.314948+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T03:56:59.314948+0000","last_peered":"2024-03-26T03:56:59.314948+0000","last_clean":"2024-03-26T03:56:59.314948+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:56:59.314948+0000","last_undegraded":"2024-03-26T03:56:59.314948+0000","last_fullsized":"2024-03-26T03:56:59.314948+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":318,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312174+0000","last_change":"2024-03-26T03:55:20.749859+0000","last_active":"2024-03-26T03:56:59.312174+0000","last_peered":"2024-03-26T03:56:59.312174+0000","last_clean":"2024-03-26T03:56:59.312174+0000","last_became_active":"2024-03-26T03:54:29.324000+0000","last_became_peered":"2024-03-26T03:54:29.324000+0000","last_unstale":"2024-03-26T03:56:59.312174+0000","last_undegraded":"2024-03-26T03:56:59.312174+0000","last_fullsized":"2024-03-26T03:56:59.312174+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:55:20.749583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.749583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":352,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.334273+0000","last_change":"2024-03-26T03:54:36.814787+0000","last_active":"2024-03-26T03:56:59.334273+0000","last_peered":"2024-03-26T03:56:59.334273+0000","last_clean":"2024-03-26T03:56:59.334273+0000","last_became_active":"2024-03-26T03:54:29.521665+0000","last_became_peered":"2024-03-26T03:54:29.521665+0000","last_unstale":"2024-03-26T03:56:59.334273+0000","last_undegraded":"2024-03-26T03:56:59.334273+0000","last_fullsized":"2024-03-26T03:56:59.334273+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.311945+0000","last_change":"2024-03-26T03:55:59.343699+0000","last_active":"2024-03-26T03:56:59.311945+0000","last_peered":"2024-03-26T03:56:59.311945+0000","last_clean":"2024-03-26T03:56:59.311945+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T03:56:59.311945+0000","last_undegraded":"2024-03-26T03:56:59.311945+0000","last_fullsized":"2024-03-26T03:56:59.311945+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":301,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.334529+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:56:59.334529+0000","last_peered":"2024-03-26T03:56:59.334529+0000","last_clean":"2024-03-26T03:56:59.334529+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:56:59.334529+0000","last_undegraded":"2024-03-26T03:56:59.334529+0000","last_fullsized":"2024-03-26T03:56:59.334529+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.340109+0000","last_change":"2024-03-26T03:54:31.825491+0000","last_active":"2024-03-26T03:56:59.340109+0000","last_peered":"2024-03-26T03:56:59.340109+0000","last_clean":"2024-03-26T03:56:59.340109+0000","last_became_active":"2024-03-26T03:54:29.453692+0000","last_became_peered":"2024-03-26T03:54:29.453692+0000","last_unstale":"2024-03-26T03:56:59.340109+0000","last_undegraded":"2024-03-26T03:56:59.340109+0000","last_fullsized":"2024-03-26T03:56:59.340109+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":298,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327183+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T03:57:01.327183+0000","last_peered":"2024-03-26T03:57:01.327183+0000","last_clean":"2024-03-26T03:57:01.327183+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T03:57:01.327183+0000","last_undegraded":"2024-03-26T03:57:01.327183+0000","last_fullsized":"2024-03-26T03:57:01.327183+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":300,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319208+0000","last_change":"2024-03-26T03:54:19.645849+0000","last_active":"2024-03-26T03:57:01.319208+0000","last_peered":"2024-03-26T03:57:01.319208+0000","last_clean":"2024-03-26T03:57:01.319208+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:57:01.319208+0000","last_undegraded":"2024-03-26T03:57:01.319208+0000","last_fullsized":"2024-03-26T03:57:01.319208+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:19.645746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.645746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":238,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312454+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:56:59.312454+0000","last_peered":"2024-03-26T03:56:59.312454+0000","last_clean":"2024-03-26T03:56:59.312454+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:56:59.312454+0000","last_undegraded":"2024-03-26T03:56:59.312454+0000","last_fullsized":"2024-03-26T03:56:59.312454+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":323,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331713+0000","last_change":"2024-03-26T03:55:59.307951+0000","last_active":"2024-03-26T03:56:59.331713+0000","last_peered":"2024-03-26T03:56:59.331713+0000","last_clean":"2024-03-26T03:56:59.331713+0000","last_became_active":"2024-03-26T03:55:59.307490+0000","last_became_peered":"2024-03-26T03:55:59.307490+0000","last_unstale":"2024-03-26T03:56:59.331713+0000","last_undegraded":"2024-03-26T03:56:59.331713+0000","last_fullsized":"2024-03-26T03:56:59.331713+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.323872+0000","last_change":"2024-03-26T03:55:59.299317+0000","last_active":"2024-03-26T03:56:59.323872+0000","last_peered":"2024-03-26T03:56:59.323872+0000","last_clean":"2024-03-26T03:56:59.323872+0000","last_became_active":"2024-03-26T03:55:59.298906+0000","last_became_peered":"2024-03-26T03:55:59.298906+0000","last_unstale":"2024-03-26T03:56:59.323872+0000","last_undegraded":"2024-03-26T03:56:59.323872+0000","last_fullsized":"2024-03-26T03:56:59.323872+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":181,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.330711+0000","last_change":"2024-03-26T03:55:29.312054+0000","last_active":"2024-03-26T03:56:59.330711+0000","last_peered":"2024-03-26T03:56:59.330711+0000","last_clean":"2024-03-26T03:56:59.330711+0000","last_became_active":"2024-03-26T03:54:29.361440+0000","last_became_peered":"2024-03-26T03:54:29.361440+0000","last_unstale":"2024-03-26T03:56:59.330711+0000","last_undegraded":"2024-03-26T03:56:59.330711+0000","last_fullsized":"2024-03-26T03:56:59.330711+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":310,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319395+0000","last_change":"2024-03-26T03:55:25.766015+0000","last_active":"2024-03-26T03:57:01.319395+0000","last_peered":"2024-03-26T03:57:01.319395+0000","last_clean":"2024-03-26T03:57:01.319395+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:57:01.319395+0000","last_undegraded":"2024-03-26T03:57:01.319395+0000","last_fullsized":"2024-03-26T03:57:01.319395+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.765904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.765904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":277,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327328+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T03:57:01.327328+0000","last_peered":"2024-03-26T03:57:01.327328+0000","last_clean":"2024-03-26T03:57:01.327328+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T03:57:01.327328+0000","last_undegraded":"2024-03-26T03:57:01.327328+0000","last_fullsized":"2024-03-26T03:57:01.327328+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":221,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319159+0000","last_change":"2024-03-26T03:55:59.303323+0000","last_active":"2024-03-26T03:57:01.319159+0000","last_peered":"2024-03-26T03:57:01.319159+0000","last_clean":"2024-03-26T03:57:01.319159+0000","last_became_active":"2024-03-26T03:55:59.302965+0000","last_became_peered":"2024-03-26T03:55:59.302965+0000","last_unstale":"2024-03-26T03:57:01.319159+0000","last_undegraded":"2024-03-26T03:57:01.319159+0000","last_fullsized":"2024-03-26T03:57:01.319159+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":305,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.339871+0000","last_change":"2024-03-26T03:55:59.324307+0000","last_active":"2024-03-26T03:56:59.339871+0000","last_peered":"2024-03-26T03:56:59.339871+0000","last_clean":"2024-03-26T03:56:59.339871+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T03:56:59.339871+0000","last_undegraded":"2024-03-26T03:56:59.339871+0000","last_fullsized":"2024-03-26T03:56:59.339871+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":124,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.330810+0000","last_change":"2024-03-26T03:56:00.569951+0000","last_active":"2024-03-26T03:56:59.330810+0000","last_peered":"2024-03-26T03:56:59.330810+0000","last_clean":"2024-03-26T03:56:59.330810+0000","last_became_active":"2024-03-26T03:55:59.308759+0000","last_became_peered":"2024-03-26T03:55:59.308759+0000","last_unstale":"2024-03-26T03:56:59.330810+0000","last_undegraded":"2024-03-26T03:56:59.330810+0000","last_fullsized":"2024-03-26T03:56:59.330810+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:12.010641+0000","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":318,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319927+0000","last_change":"2024-03-26T03:54:29.530213+0000","last_active":"2024-03-26T03:57:01.319927+0000","last_peered":"2024-03-26T03:57:01.319927+0000","last_clean":"2024-03-26T03:57:01.319927+0000","last_became_active":"2024-03-26T03:54:29.529851+0000","last_became_peered":"2024-03-26T03:54:29.529851+0000","last_unstale":"2024-03-26T03:57:01.319927+0000","last_undegraded":"2024-03-26T03:57:01.319927+0000","last_fullsized":"2024-03-26T03:57:01.319927+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:24.672762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:24.672762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"152'13","reported_seq":347,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313633+0000","last_change":"2024-03-26T03:55:30.839253+0000","last_active":"2024-03-26T03:56:59.313633+0000","last_peered":"2024-03-26T03:56:59.313633+0000","last_clean":"2024-03-26T03:56:59.313633+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:56:59.313633+0000","last_undegraded":"2024-03-26T03:56:59.313633+0000","last_fullsized":"2024-03-26T03:56:59.313633+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:55:30.839058+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.839058+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":125,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.332704+0000","last_change":"2024-03-26T03:55:59.311115+0000","last_active":"2024-03-26T03:56:59.332704+0000","last_peered":"2024-03-26T03:56:59.332704+0000","last_clean":"2024-03-26T03:56:59.332704+0000","last_became_active":"2024-03-26T03:55:59.310758+0000","last_became_peered":"2024-03-26T03:55:59.310758+0000","last_unstale":"2024-03-26T03:56:59.332704+0000","last_undegraded":"2024-03-26T03:56:59.332704+0000","last_fullsized":"2024-03-26T03:56:59.332704+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.804652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313489+0000","last_change":"2024-03-26T03:55:59.302276+0000","last_active":"2024-03-26T03:56:59.313489+0000","last_peered":"2024-03-26T03:56:59.313489+0000","last_clean":"2024-03-26T03:56:59.313489+0000","last_became_active":"2024-03-26T03:55:59.299889+0000","last_became_peered":"2024-03-26T03:55:59.299889+0000","last_unstale":"2024-03-26T03:56:59.313489+0000","last_undegraded":"2024-03-26T03:56:59.313489+0000","last_fullsized":"2024-03-26T03:56:59.313489+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:40.835525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:52:40.835525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312641+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:56:59.312641+0000","last_peered":"2024-03-26T03:56:59.312641+0000","last_clean":"2024-03-26T03:56:59.312641+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:56:59.312641+0000","last_undegraded":"2024-03-26T03:56:59.312641+0000","last_fullsized":"2024-03-26T03:56:59.312641+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":342,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313187+0000","last_change":"2024-03-26T03:55:59.300136+0000","last_active":"2024-03-26T03:56:59.313187+0000","last_peered":"2024-03-26T03:56:59.313187+0000","last_clean":"2024-03-26T03:56:59.313187+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T03:56:59.313187+0000","last_undegraded":"2024-03-26T03:56:59.313187+0000","last_fullsized":"2024-03-26T03:56:59.313187+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.333947+0000","last_change":"2024-03-26T03:54:29.521980+0000","last_active":"2024-03-26T03:56:59.333947+0000","last_peered":"2024-03-26T03:56:59.333947+0000","last_clean":"2024-03-26T03:56:59.333947+0000","last_became_active":"2024-03-26T03:54:29.520018+0000","last_became_peered":"2024-03-26T03:54:29.520018+0000","last_unstale":"2024-03-26T03:56:59.333947+0000","last_undegraded":"2024-03-26T03:56:59.333947+0000","last_fullsized":"2024-03-26T03:56:59.333947+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":393,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326588+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T03:57:01.326588+0000","last_peered":"2024-03-26T03:57:01.326588+0000","last_clean":"2024-03-26T03:57:01.326588+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T03:57:01.326588+0000","last_undegraded":"2024-03-26T03:57:01.326588+0000","last_fullsized":"2024-03-26T03:57:01.326588+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319101+0000","last_change":"2024-03-26T03:55:57.916905+0000","last_active":"2024-03-26T03:57:01.319101+0000","last_peered":"2024-03-26T03:57:01.319101+0000","last_clean":"2024-03-26T03:57:01.319101+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:57:01.319101+0000","last_undegraded":"2024-03-26T03:57:01.319101+0000","last_fullsized":"2024-03-26T03:57:01.319101+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.916786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.916786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:27.322837+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":189,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312517+0000","last_change":"2024-03-26T03:54:32.911388+0000","last_active":"2024-03-26T03:56:59.312517+0000","last_peered":"2024-03-26T03:56:59.312517+0000","last_clean":"2024-03-26T03:56:59.312517+0000","last_became_active":"2024-03-26T03:54:29.390009+0000","last_became_peered":"2024-03-26T03:54:29.390009+0000","last_unstale":"2024-03-26T03:56:59.312517+0000","last_undegraded":"2024-03-26T03:56:59.312517+0000","last_fullsized":"2024-03-26T03:56:59.312517+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":305,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.315107+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T03:56:59.315107+0000","last_peered":"2024-03-26T03:56:59.315107+0000","last_clean":"2024-03-26T03:56:59.315107+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T03:56:59.315107+0000","last_undegraded":"2024-03-26T03:56:59.315107+0000","last_fullsized":"2024-03-26T03:56:59.315107+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":180,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319963+0000","last_change":"2024-03-26T03:55:13.805008+0000","last_active":"2024-03-26T03:57:01.319963+0000","last_peered":"2024-03-26T03:57:01.319963+0000","last_clean":"2024-03-26T03:57:01.319963+0000","last_became_active":"2024-03-26T03:54:29.389636+0000","last_became_peered":"2024-03-26T03:54:29.389636+0000","last_unstale":"2024-03-26T03:57:01.319963+0000","last_undegraded":"2024-03-26T03:57:01.319963+0000","last_fullsized":"2024-03-26T03:57:01.319963+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":334,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331743+0000","last_change":"2024-03-26T03:55:53.234770+0000","last_active":"2024-03-26T03:56:59.331743+0000","last_peered":"2024-03-26T03:56:59.331743+0000","last_clean":"2024-03-26T03:56:59.331743+0000","last_became_active":"2024-03-26T03:54:29.345136+0000","last_became_peered":"2024-03-26T03:54:29.345136+0000","last_unstale":"2024-03-26T03:56:59.331743+0000","last_undegraded":"2024-03-26T03:56:59.331743+0000","last_fullsized":"2024-03-26T03:56:59.331743+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:03.979602+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1347,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":317,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326763+0000","last_change":"2024-03-26T03:55:59.342176+0000","last_active":"2024-03-26T03:57:01.326763+0000","last_peered":"2024-03-26T03:57:01.326763+0000","last_clean":"2024-03-26T03:57:01.326763+0000","last_became_active":"2024-03-26T03:55:59.340772+0000","last_became_peered":"2024-03-26T03:55:59.340772+0000","last_unstale":"2024-03-26T03:57:01.326763+0000","last_undegraded":"2024-03-26T03:57:01.326763+0000","last_fullsized":"2024-03-26T03:57:01.326763+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.329071+0000","last_change":"2024-03-26T03:54:36.963991+0000","last_active":"2024-03-26T03:56:59.329071+0000","last_peered":"2024-03-26T03:56:59.329071+0000","last_clean":"2024-03-26T03:56:59.329071+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:56:59.329071+0000","last_undegraded":"2024-03-26T03:56:59.329071+0000","last_fullsized":"2024-03-26T03:56:59.329071+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:54:36.963833+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.963833+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.314947+0000","last_change":"2024-03-26T03:55:19.712372+0000","last_active":"2024-03-26T03:56:59.314947+0000","last_peered":"2024-03-26T03:56:59.314947+0000","last_clean":"2024-03-26T03:56:59.314947+0000","last_became_active":"2024-03-26T03:54:29.532138+0000","last_became_peered":"2024-03-26T03:54:29.532138+0000","last_unstale":"2024-03-26T03:56:59.314947+0000","last_undegraded":"2024-03-26T03:56:59.314947+0000","last_fullsized":"2024-03-26T03:56:59.314947+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":318,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319076+0000","last_change":"2024-03-26T03:53:29.687145+0000","last_active":"2024-03-26T03:57:01.319076+0000","last_peered":"2024-03-26T03:57:01.319076+0000","last_clean":"2024-03-26T03:57:01.319076+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:57:01.319076+0000","last_undegraded":"2024-03-26T03:57:01.319076+0000","last_fullsized":"2024-03-26T03:57:01.319076+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.687067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.687067+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":364,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.334376+0000","last_change":"2024-03-26T03:55:25.160341+0000","last_active":"2024-03-26T03:56:59.334376+0000","last_peered":"2024-03-26T03:56:59.334376+0000","last_clean":"2024-03-26T03:56:59.334376+0000","last_became_active":"2024-03-26T03:54:29.521088+0000","last_became_peered":"2024-03-26T03:54:29.521088+0000","last_unstale":"2024-03-26T03:56:59.334376+0000","last_undegraded":"2024-03-26T03:56:59.334376+0000","last_fullsized":"2024-03-26T03:56:59.334376+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":307,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319770+0000","last_change":"2024-03-26T03:55:59.300900+0000","last_active":"2024-03-26T03:57:01.319770+0000","last_peered":"2024-03-26T03:57:01.319770+0000","last_clean":"2024-03-26T03:57:01.319770+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T03:57:01.319770+0000","last_undegraded":"2024-03-26T03:57:01.319770+0000","last_fullsized":"2024-03-26T03:57:01.319770+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:17.657828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:17.657828+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":268,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327458+0000","last_change":"2024-03-26T03:55:59.305418+0000","last_active":"2024-03-26T03:57:01.327458+0000","last_peered":"2024-03-26T03:57:01.327458+0000","last_clean":"2024-03-26T03:57:01.327458+0000","last_became_active":"2024-03-26T03:55:59.304158+0000","last_became_peered":"2024-03-26T03:55:59.304158+0000","last_unstale":"2024-03-26T03:57:01.327458+0000","last_undegraded":"2024-03-26T03:57:01.327458+0000","last_fullsized":"2024-03-26T03:57:01.327458+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":253,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.340060+0000","last_change":"2024-03-26T03:55:59.325776+0000","last_active":"2024-03-26T03:56:59.340060+0000","last_peered":"2024-03-26T03:56:59.340060+0000","last_clean":"2024-03-26T03:56:59.340060+0000","last_became_active":"2024-03-26T03:55:59.325344+0000","last_became_peered":"2024-03-26T03:55:59.325344+0000","last_unstale":"2024-03-26T03:56:59.340060+0000","last_undegraded":"2024-03-26T03:56:59.340060+0000","last_fullsized":"2024-03-26T03:56:59.340060+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":170,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319829+0000","last_change":"2024-03-26T03:55:19.697165+0000","last_active":"2024-03-26T03:57:01.319829+0000","last_peered":"2024-03-26T03:57:01.319829+0000","last_clean":"2024-03-26T03:57:01.319829+0000","last_became_active":"2024-03-26T03:54:29.289163+0000","last_became_peered":"2024-03-26T03:54:29.289163+0000","last_unstale":"2024-03-26T03:57:01.319829+0000","last_undegraded":"2024-03-26T03:57:01.319829+0000","last_fullsized":"2024-03-26T03:57:01.319829+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.4","version":"203'10623","reported_seq":17156,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:02.062774+0000","last_change":"2024-03-26T03:57:00.336002+0000","last_active":"2024-03-26T03:57:02.062774+0000","last_peered":"2024-03-26T03:57:02.062774+0000","last_clean":"2024-03-26T03:57:02.062774+0000","last_became_active":"2024-03-26T03:57:00.322652+0000","last_became_peered":"2024-03-26T03:57:00.322652+0000","last_unstale":"2024-03-26T03:57:02.062774+0000","last_undegraded":"2024-03-26T03:57:02.062774+0000","last_fullsized":"2024-03-26T03:57:02.062774+0000","mapping_epoch":201,"log_start":"176'6384","ondisk_log_start":"176'6384","created":155,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4239,"log_dups_size":0,"ondisk_log_size":4239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371392512,"num_objects":5667,"num_object_clones":0,"num_object_copies":17001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10623,"num_write_kb":521280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":210173952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":261,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327017+0000","last_change":"2024-03-26T03:55:59.325043+0000","last_active":"2024-03-26T03:57:01.327017+0000","last_peered":"2024-03-26T03:57:01.327017+0000","last_clean":"2024-03-26T03:57:01.327017+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T03:57:01.327017+0000","last_undegraded":"2024-03-26T03:57:01.327017+0000","last_fullsized":"2024-03-26T03:57:01.327017+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":324,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319741+0000","last_change":"2024-03-26T03:54:16.696733+0000","last_active":"2024-03-26T03:57:01.319741+0000","last_peered":"2024-03-26T03:57:01.319741+0000","last_clean":"2024-03-26T03:57:01.319741+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:57:01.319741+0000","last_undegraded":"2024-03-26T03:57:01.319741+0000","last_fullsized":"2024-03-26T03:57:01.319741+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:54:16.696602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.696602+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":334,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326506+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T03:57:01.326506+0000","last_peered":"2024-03-26T03:57:01.326506+0000","last_clean":"2024-03-26T03:57:01.326506+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T03:57:01.326506+0000","last_undegraded":"2024-03-26T03:57:01.326506+0000","last_fullsized":"2024-03-26T03:57:01.326506+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":431,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319627+0000","last_change":"2024-03-26T03:55:52.900155+0000","last_active":"2024-03-26T03:57:01.319627+0000","last_peered":"2024-03-26T03:57:01.319627+0000","last_clean":"2024-03-26T03:57:01.319627+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:57:01.319627+0000","last_undegraded":"2024-03-26T03:57:01.319627+0000","last_fullsized":"2024-03-26T03:57:01.319627+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:52.900040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:52.900040+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:06.366979+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"200'10549","reported_seq":10652,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312901+0000","last_change":"2024-03-26T03:54:38.786252+0000","last_active":"2024-03-26T03:56:59.312901+0000","last_peered":"2024-03-26T03:56:59.312901+0000","last_clean":"2024-03-26T03:56:59.312901+0000","last_became_active":"2024-03-26T03:54:38.785773+0000","last_became_peered":"2024-03-26T03:54:38.785773+0000","last_unstale":"2024-03-26T03:56:59.312901+0000","last_undegraded":"2024-03-26T03:56:59.312901+0000","last_fullsized":"2024-03-26T03:56:59.312901+0000","mapping_epoch":155,"log_start":"178'6600","ondisk_log_start":"178'6600","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3949,"log_dups_size":0,"ondisk_log_size":3949,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367329280,"num_objects":5605,"num_object_clones":0,"num_object_copies":16815,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5605,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10549,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":120,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319328+0000","last_change":"2024-03-26T03:54:30.757009+0000","last_active":"2024-03-26T03:57:01.319328+0000","last_peered":"2024-03-26T03:57:01.319328+0000","last_clean":"2024-03-26T03:57:01.319328+0000","last_became_active":"2024-03-26T03:54:29.304565+0000","last_became_peered":"2024-03-26T03:54:29.304565+0000","last_unstale":"2024-03-26T03:57:01.319328+0000","last_undegraded":"2024-03-26T03:57:01.319328+0000","last_fullsized":"2024-03-26T03:57:01.319328+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":312,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.320075+0000","last_change":"2024-03-26T03:54:18.653671+0000","last_active":"2024-03-26T03:57:01.320075+0000","last_peered":"2024-03-26T03:57:01.320075+0000","last_clean":"2024-03-26T03:57:01.320075+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:57:01.320075+0000","last_undegraded":"2024-03-26T03:57:01.320075+0000","last_fullsized":"2024-03-26T03:57:01.320075+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.653555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.653555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":373,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313471+0000","last_change":"2024-03-26T03:55:14.285475+0000","last_active":"2024-03-26T03:56:59.313471+0000","last_peered":"2024-03-26T03:56:59.313471+0000","last_clean":"2024-03-26T03:56:59.313471+0000","last_became_active":"2024-03-26T03:54:29.346977+0000","last_became_peered":"2024-03-26T03:54:29.346977+0000","last_unstale":"2024-03-26T03:56:59.313471+0000","last_undegraded":"2024-03-26T03:56:59.313471+0000","last_fullsized":"2024-03-26T03:56:59.313471+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.319581+0000","last_change":"2024-03-26T03:55:19.828552+0000","last_active":"2024-03-26T03:56:59.319581+0000","last_peered":"2024-03-26T03:56:59.319581+0000","last_clean":"2024-03-26T03:56:59.319581+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:56:59.319581+0000","last_undegraded":"2024-03-26T03:56:59.319581+0000","last_fullsized":"2024-03-26T03:56:59.319581+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.828444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.828444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"201'10344","reported_seq":10448,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.506526+0000","last_change":"2024-03-26T03:54:38.785847+0000","last_active":"2024-03-26T03:56:59.506526+0000","last_peered":"2024-03-26T03:56:59.506526+0000","last_clean":"2024-03-26T03:56:59.506526+0000","last_became_active":"2024-03-26T03:54:38.785420+0000","last_became_peered":"2024-03-26T03:54:38.785420+0000","last_unstale":"2024-03-26T03:56:59.506526+0000","last_undegraded":"2024-03-26T03:56:59.506526+0000","last_fullsized":"2024-03-26T03:56:59.506526+0000","mapping_epoch":155,"log_start":"176'6300","ondisk_log_start":"176'6300","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371326976,"num_objects":5666,"num_object_clones":0,"num_object_copies":16998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10344,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.318850+0000","last_change":"2024-03-26T03:54:29.359240+0000","last_active":"2024-03-26T03:57:01.318850+0000","last_peered":"2024-03-26T03:57:01.318850+0000","last_clean":"2024-03-26T03:57:01.318850+0000","last_became_active":"2024-03-26T03:54:29.358827+0000","last_became_peered":"2024-03-26T03:54:29.358827+0000","last_unstale":"2024-03-26T03:57:01.318850+0000","last_undegraded":"2024-03-26T03:57:01.318850+0000","last_fullsized":"2024-03-26T03:57:01.318850+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.664206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.664206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":317,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.320117+0000","last_change":"2024-03-26T03:55:59.304842+0000","last_active":"2024-03-26T03:56:59.320117+0000","last_peered":"2024-03-26T03:56:59.320117+0000","last_clean":"2024-03-26T03:56:59.320117+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T03:56:59.320117+0000","last_undegraded":"2024-03-26T03:56:59.320117+0000","last_fullsized":"2024-03-26T03:56:59.320117+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.876322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.876322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":333,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.320027+0000","last_change":"2024-03-26T03:55:59.299719+0000","last_active":"2024-03-26T03:57:01.320027+0000","last_peered":"2024-03-26T03:57:01.320027+0000","last_clean":"2024-03-26T03:57:01.320027+0000","last_became_active":"2024-03-26T03:55:59.299160+0000","last_became_peered":"2024-03-26T03:55:59.299160+0000","last_unstale":"2024-03-26T03:57:01.320027+0000","last_undegraded":"2024-03-26T03:57:01.320027+0000","last_fullsized":"2024-03-26T03:57:01.320027+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.612556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.612556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313368+0000","last_change":"2024-03-26T03:55:59.302041+0000","last_active":"2024-03-26T03:56:59.313368+0000","last_peered":"2024-03-26T03:56:59.313368+0000","last_clean":"2024-03-26T03:56:59.313368+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T03:56:59.313368+0000","last_undegraded":"2024-03-26T03:56:59.313368+0000","last_fullsized":"2024-03-26T03:56:59.313368+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:55:14.773103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773103+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"201'10512","reported_seq":10616,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.445924+0000","last_change":"2024-03-26T03:54:38.382909+0000","last_active":"2024-03-26T03:56:59.445924+0000","last_peered":"2024-03-26T03:56:59.445924+0000","last_clean":"2024-03-26T03:56:59.445924+0000","last_became_active":"2024-03-26T03:54:38.382277+0000","last_became_peered":"2024-03-26T03:54:38.382277+0000","last_unstale":"2024-03-26T03:56:59.445924+0000","last_undegraded":"2024-03-26T03:56:59.445924+0000","last_fullsized":"2024-03-26T03:56:59.445924+0000","mapping_epoch":155,"log_start":"182'6900","ondisk_log_start":"182'6900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3612,"log_dups_size":0,"ondisk_log_size":3612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367001600,"num_objects":5600,"num_object_clones":0,"num_object_copies":16800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10512,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.332938+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T03:56:59.332938+0000","last_peered":"2024-03-26T03:56:59.332938+0000","last_clean":"2024-03-26T03:56:59.332938+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:56:59.332938+0000","last_undegraded":"2024-03-26T03:56:59.332938+0000","last_fullsized":"2024-03-26T03:56:59.332938+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.320115+0000","last_change":"2024-03-26T03:54:33.962451+0000","last_active":"2024-03-26T03:56:59.320115+0000","last_peered":"2024-03-26T03:56:59.320115+0000","last_clean":"2024-03-26T03:56:59.320115+0000","last_became_active":"2024-03-26T03:54:29.428035+0000","last_became_peered":"2024-03-26T03:54:29.428035+0000","last_unstale":"2024-03-26T03:56:59.320115+0000","last_undegraded":"2024-03-26T03:56:59.320115+0000","last_fullsized":"2024-03-26T03:56:59.320115+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.962364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.962364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":290,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326562+0000","last_change":"2024-03-26T03:55:59.341967+0000","last_active":"2024-03-26T03:57:01.326562+0000","last_peered":"2024-03-26T03:57:01.326562+0000","last_clean":"2024-03-26T03:57:01.326562+0000","last_became_active":"2024-03-26T03:55:59.341062+0000","last_became_peered":"2024-03-26T03:55:59.341062+0000","last_unstale":"2024-03-26T03:57:01.326562+0000","last_undegraded":"2024-03-26T03:57:01.326562+0000","last_fullsized":"2024-03-26T03:57:01.326562+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":359,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.332098+0000","last_change":"2024-03-26T03:55:59.309207+0000","last_active":"2024-03-26T03:56:59.332098+0000","last_peered":"2024-03-26T03:56:59.332098+0000","last_clean":"2024-03-26T03:56:59.332098+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T03:56:59.332098+0000","last_undegraded":"2024-03-26T03:56:59.332098+0000","last_fullsized":"2024-03-26T03:56:59.332098+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"201'10296","reported_seq":10550,"reported_epoch":201,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.957465+0000","last_change":"2024-03-26T03:56:59.104904+0000","last_active":"2024-03-26T03:56:59.957465+0000","last_peered":"2024-03-26T03:56:59.957465+0000","last_clean":"2024-03-26T03:55:19.234210+0000","last_became_active":"2024-03-26T03:56:00.572420+0000","last_became_peered":"2024-03-26T03:56:00.572420+0000","last_unstale":"2024-03-26T03:56:59.957465+0000","last_undegraded":"2024-03-26T03:56:00.323147+0000","last_fullsized":"2024-03-26T03:56:00.288259+0000","mapping_epoch":181,"log_start":"176'6277","ondisk_log_start":"176'6277","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4019,"log_dups_size":0,"ondisk_log_size":4019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370671616,"num_objects":5656,"num_object_clones":0,"num_object_copies":16968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3096,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10296,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":4718592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":2560}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":125,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312363+0000","last_change":"2024-03-26T03:55:59.289528+0000","last_active":"2024-03-26T03:56:59.312363+0000","last_peered":"2024-03-26T03:56:59.312363+0000","last_clean":"2024-03-26T03:56:59.312363+0000","last_became_active":"2024-03-26T03:55:59.288910+0000","last_became_peered":"2024-03-26T03:55:59.288910+0000","last_unstale":"2024-03-26T03:56:59.312363+0000","last_undegraded":"2024-03-26T03:56:59.312363+0000","last_fullsized":"2024-03-26T03:56:59.312363+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":264,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326975+0000","last_change":"2024-03-26T03:55:59.316898+0000","last_active":"2024-03-26T03:57:01.326975+0000","last_peered":"2024-03-26T03:57:01.326975+0000","last_clean":"2024-03-26T03:57:01.326975+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T03:57:01.326975+0000","last_undegraded":"2024-03-26T03:57:01.326975+0000","last_fullsized":"2024-03-26T03:57:01.326975+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":314,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312405+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T03:56:59.312405+0000","last_peered":"2024-03-26T03:56:59.312405+0000","last_clean":"2024-03-26T03:56:59.312405+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:56:59.312405+0000","last_undegraded":"2024-03-26T03:56:59.312405+0000","last_fullsized":"2024-03-26T03:56:59.312405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"154'35","reported_seq":493,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312415+0000","last_change":"2024-03-26T03:54:29.460582+0000","last_active":"2024-03-26T03:56:59.312415+0000","last_peered":"2024-03-26T03:56:59.312415+0000","last_clean":"2024-03-26T03:56:59.312415+0000","last_became_active":"2024-03-26T03:54:29.371147+0000","last_became_peered":"2024-03-26T03:54:29.371147+0000","last_unstale":"2024-03-26T03:56:59.312415+0000","last_undegraded":"2024-03-26T03:56:59.312415+0000","last_fullsized":"2024-03-26T03:56:59.312415+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":300,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331048+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:56:59.331048+0000","last_peered":"2024-03-26T03:56:59.331048+0000","last_clean":"2024-03-26T03:56:59.331048+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:56:59.331048+0000","last_undegraded":"2024-03-26T03:56:59.331048+0000","last_fullsized":"2024-03-26T03:56:59.331048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"200'10730","reported_seq":10834,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.311991+0000","last_change":"2024-03-26T03:54:38.382935+0000","last_active":"2024-03-26T03:56:59.311991+0000","last_peered":"2024-03-26T03:56:59.311991+0000","last_clean":"2024-03-26T03:56:59.311991+0000","last_became_active":"2024-03-26T03:54:38.382581+0000","last_became_peered":"2024-03-26T03:54:38.382581+0000","last_unstale":"2024-03-26T03:56:59.311991+0000","last_undegraded":"2024-03-26T03:56:59.311991+0000","last_fullsized":"2024-03-26T03:56:59.311991+0000","mapping_epoch":155,"log_start":"177'6700","ondisk_log_start":"177'6700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4030,"log_dups_size":0,"ondisk_log_size":4030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":386138112,"num_objects":5892,"num_object_clones":0,"num_object_copies":17676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10730,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":321,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.320337+0000","last_change":"2024-03-26T03:55:59.310549+0000","last_active":"2024-03-26T03:57:01.320337+0000","last_peered":"2024-03-26T03:57:01.320337+0000","last_clean":"2024-03-26T03:57:01.320337+0000","last_became_active":"2024-03-26T03:55:59.310313+0000","last_became_peered":"2024-03-26T03:55:59.310313+0000","last_unstale":"2024-03-26T03:57:01.320337+0000","last_undegraded":"2024-03-26T03:57:01.320337+0000","last_fullsized":"2024-03-26T03:57:01.320337+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.773109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.323786+0000","last_change":"2024-03-26T03:55:29.863335+0000","last_active":"2024-03-26T03:56:59.323786+0000","last_peered":"2024-03-26T03:56:59.323786+0000","last_clean":"2024-03-26T03:56:59.323786+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:56:59.323786+0000","last_undegraded":"2024-03-26T03:56:59.323786+0000","last_fullsized":"2024-03-26T03:56:59.323786+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.863201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.863201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.320267+0000","last_change":"2024-03-26T03:54:29.333438+0000","last_active":"2024-03-26T03:56:59.320267+0000","last_peered":"2024-03-26T03:56:59.320267+0000","last_clean":"2024-03-26T03:56:59.320267+0000","last_became_active":"2024-03-26T03:54:29.319429+0000","last_became_peered":"2024-03-26T03:54:29.319429+0000","last_unstale":"2024-03-26T03:56:59.320267+0000","last_undegraded":"2024-03-26T03:56:59.320267+0000","last_fullsized":"2024-03-26T03:56:59.320267+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.962798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.962798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":343,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.314695+0000","last_change":"2024-03-26T03:55:59.300086+0000","last_active":"2024-03-26T03:56:59.314695+0000","last_peered":"2024-03-26T03:56:59.314695+0000","last_clean":"2024-03-26T03:56:59.314695+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T03:56:59.314695+0000","last_undegraded":"2024-03-26T03:56:59.314695+0000","last_fullsized":"2024-03-26T03:56:59.314695+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.322119+0000","last_change":"2024-03-26T03:55:59.310466+0000","last_active":"2024-03-26T03:56:59.322119+0000","last_peered":"2024-03-26T03:56:59.322119+0000","last_clean":"2024-03-26T03:56:59.322119+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T03:56:59.322119+0000","last_undegraded":"2024-03-26T03:56:59.322119+0000","last_fullsized":"2024-03-26T03:56:59.322119+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.387809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.387809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"201'10701","reported_seq":10815,"reported_epoch":201,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.473280+0000","last_change":"2024-03-26T03:56:00.607744+0000","last_active":"2024-03-26T03:56:59.473280+0000","last_peered":"2024-03-26T03:56:59.473280+0000","last_clean":"2024-03-26T03:55:19.247499+0000","last_became_active":"2024-03-26T03:56:00.597387+0000","last_became_peered":"2024-03-26T03:56:00.597387+0000","last_unstale":"2024-03-26T03:56:59.473280+0000","last_undegraded":"2024-03-26T03:56:00.318184+0000","last_fullsized":"2024-03-26T03:56:00.286579+0000","mapping_epoch":181,"log_start":"177'6700","ondisk_log_start":"177'6700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4001,"log_dups_size":0,"ondisk_log_size":4001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380960768,"num_objects":5813,"num_object_clones":0,"num_object_copies":17439,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3268,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5813,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10701,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2545}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":264,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327307+0000","last_change":"2024-03-26T03:55:59.296154+0000","last_active":"2024-03-26T03:57:01.327307+0000","last_peered":"2024-03-26T03:57:01.327307+0000","last_clean":"2024-03-26T03:57:01.327307+0000","last_became_active":"2024-03-26T03:55:59.294575+0000","last_became_peered":"2024-03-26T03:55:59.294575+0000","last_unstale":"2024-03-26T03:57:01.327307+0000","last_undegraded":"2024-03-26T03:57:01.327307+0000","last_fullsized":"2024-03-26T03:57:01.327307+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":577,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319058+0000","last_change":"2024-03-26T03:55:17.691688+0000","last_active":"2024-03-26T03:57:01.319058+0000","last_peered":"2024-03-26T03:57:01.319058+0000","last_clean":"2024-03-26T03:57:01.319058+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:57:01.319058+0000","last_undegraded":"2024-03-26T03:57:01.319058+0000","last_fullsized":"2024-03-26T03:57:01.319058+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:55:17.691502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.691502+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":139,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326555+0000","last_change":"2024-03-26T03:55:59.298495+0000","last_active":"2024-03-26T03:57:01.326555+0000","last_peered":"2024-03-26T03:57:01.326555+0000","last_clean":"2024-03-26T03:57:01.326555+0000","last_became_active":"2024-03-26T03:55:59.298187+0000","last_became_peered":"2024-03-26T03:55:59.298187+0000","last_unstale":"2024-03-26T03:57:01.326555+0000","last_undegraded":"2024-03-26T03:57:01.326555+0000","last_fullsized":"2024-03-26T03:57:01.326555+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":374,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312764+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T03:56:59.312764+0000","last_peered":"2024-03-26T03:56:59.312764+0000","last_clean":"2024-03-26T03:56:59.312764+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:56:59.312764+0000","last_undegraded":"2024-03-26T03:56:59.312764+0000","last_fullsized":"2024-03-26T03:56:59.312764+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":313,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.320066+0000","last_change":"2024-03-26T03:55:59.299269+0000","last_active":"2024-03-26T03:57:01.320066+0000","last_peered":"2024-03-26T03:57:01.320066+0000","last_clean":"2024-03-26T03:57:01.320066+0000","last_became_active":"2024-03-26T03:55:59.298950+0000","last_became_peered":"2024-03-26T03:55:59.298950+0000","last_unstale":"2024-03-26T03:57:01.320066+0000","last_undegraded":"2024-03-26T03:57:01.320066+0000","last_fullsized":"2024-03-26T03:57:01.320066+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.678344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.678344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"201'10410","reported_seq":10521,"reported_epoch":201,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.490835+0000","last_change":"2024-03-26T03:56:00.604282+0000","last_active":"2024-03-26T03:56:59.490835+0000","last_peered":"2024-03-26T03:56:59.490835+0000","last_clean":"2024-03-26T03:55:19.256252+0000","last_became_active":"2024-03-26T03:56:00.597388+0000","last_became_peered":"2024-03-26T03:56:00.597388+0000","last_unstale":"2024-03-26T03:56:59.490835+0000","last_undegraded":"2024-03-26T03:56:00.387509+0000","last_fullsized":"2024-03-26T03:56:00.283703+0000","mapping_epoch":181,"log_start":"179'6800","ondisk_log_start":"179'6800","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371195904,"num_objects":5664,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3208,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10410,"num_write_kb":514368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2456}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312047+0000","last_change":"2024-03-26T03:55:59.300224+0000","last_active":"2024-03-26T03:56:59.312047+0000","last_peered":"2024-03-26T03:56:59.312047+0000","last_clean":"2024-03-26T03:56:59.312047+0000","last_became_active":"2024-03-26T03:55:59.299592+0000","last_became_peered":"2024-03-26T03:55:59.299592+0000","last_unstale":"2024-03-26T03:56:59.312047+0000","last_undegraded":"2024-03-26T03:56:59.312047+0000","last_fullsized":"2024-03-26T03:56:59.312047+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.332624+0000","last_change":"2024-03-26T03:55:59.309172+0000","last_active":"2024-03-26T03:56:59.332624+0000","last_peered":"2024-03-26T03:56:59.332624+0000","last_clean":"2024-03-26T03:56:59.332624+0000","last_became_active":"2024-03-26T03:55:59.308797+0000","last_became_peered":"2024-03-26T03:55:59.308797+0000","last_unstale":"2024-03-26T03:56:59.332624+0000","last_undegraded":"2024-03-26T03:56:59.332624+0000","last_fullsized":"2024-03-26T03:56:59.332624+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":825,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:02.412289+0000","last_change":"2024-03-26T03:55:58.942740+0000","last_active":"2024-03-26T03:57:02.412289+0000","last_peered":"2024-03-26T03:57:02.412289+0000","last_clean":"2024-03-26T03:57:02.412289+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:57:02.412289+0000","last_undegraded":"2024-03-26T03:57:02.412289+0000","last_fullsized":"2024-03-26T03:57:02.412289+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:55:58.942635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.942635+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:03.755920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":492,"num_read_kb":487,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.334069+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T03:56:59.334069+0000","last_peered":"2024-03-26T03:56:59.334069+0000","last_clean":"2024-03-26T03:56:59.334069+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:56:59.334069+0000","last_undegraded":"2024-03-26T03:56:59.334069+0000","last_fullsized":"2024-03-26T03:56:59.334069+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":251,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327324+0000","last_change":"2024-03-26T03:55:59.311162+0000","last_active":"2024-03-26T03:57:01.327324+0000","last_peered":"2024-03-26T03:57:01.327324+0000","last_clean":"2024-03-26T03:57:01.327324+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T03:57:01.327324+0000","last_undegraded":"2024-03-26T03:57:01.327324+0000","last_fullsized":"2024-03-26T03:57:01.327324+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"201'10717","reported_seq":10821,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.509757+0000","last_change":"2024-03-26T03:54:38.384402+0000","last_active":"2024-03-26T03:56:59.509757+0000","last_peered":"2024-03-26T03:56:59.509757+0000","last_clean":"2024-03-26T03:56:59.509757+0000","last_became_active":"2024-03-26T03:54:38.384087+0000","last_became_peered":"2024-03-26T03:54:38.384087+0000","last_unstale":"2024-03-26T03:56:59.509757+0000","last_undegraded":"2024-03-26T03:56:59.509757+0000","last_fullsized":"2024-03-26T03:56:59.509757+0000","mapping_epoch":155,"log_start":"175'6100","ondisk_log_start":"175'6100","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4617,"log_dups_size":0,"ondisk_log_size":4617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371261440,"num_objects":5665,"num_object_clones":0,"num_object_copies":16995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10717,"num_write_kb":524224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312829+0000","last_change":"2024-03-26T03:55:15.751265+0000","last_active":"2024-03-26T03:56:59.312829+0000","last_peered":"2024-03-26T03:56:59.312829+0000","last_clean":"2024-03-26T03:56:59.312829+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:56:59.312829+0000","last_undegraded":"2024-03-26T03:56:59.312829+0000","last_fullsized":"2024-03-26T03:56:59.312829+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:15.751136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:15.751136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":316,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312250+0000","last_change":"2024-03-26T03:55:59.301777+0000","last_active":"2024-03-26T03:56:59.312250+0000","last_peered":"2024-03-26T03:56:59.312250+0000","last_clean":"2024-03-26T03:56:59.312250+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T03:56:59.312250+0000","last_undegraded":"2024-03-26T03:56:59.312250+0000","last_fullsized":"2024-03-26T03:56:59.312250+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":361,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331327+0000","last_change":"2024-03-26T03:55:59.309535+0000","last_active":"2024-03-26T03:56:59.331327+0000","last_peered":"2024-03-26T03:56:59.331327+0000","last_clean":"2024-03-26T03:56:59.331327+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T03:56:59.331327+0000","last_undegraded":"2024-03-26T03:56:59.331327+0000","last_fullsized":"2024-03-26T03:56:59.331327+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":247,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331658+0000","last_change":"2024-03-26T03:55:59.308367+0000","last_active":"2024-03-26T03:56:59.331658+0000","last_peered":"2024-03-26T03:56:59.331658+0000","last_clean":"2024-03-26T03:56:59.331658+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T03:56:59.331658+0000","last_undegraded":"2024-03-26T03:56:59.331658+0000","last_fullsized":"2024-03-26T03:56:59.331658+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"201'10529","reported_seq":10633,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.486851+0000","last_change":"2024-03-26T03:54:38.379391+0000","last_active":"2024-03-26T03:56:59.486851+0000","last_peered":"2024-03-26T03:56:59.486851+0000","last_clean":"2024-03-26T03:56:59.486851+0000","last_became_active":"2024-03-26T03:54:38.379010+0000","last_became_peered":"2024-03-26T03:54:38.379010+0000","last_unstale":"2024-03-26T03:56:59.486851+0000","last_undegraded":"2024-03-26T03:56:59.486851+0000","last_fullsized":"2024-03-26T03:56:59.486851+0000","mapping_epoch":155,"log_start":"177'6500","ondisk_log_start":"177'6500","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4029,"log_dups_size":0,"ondisk_log_size":4029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370868224,"num_objects":5659,"num_object_clones":0,"num_object_copies":16977,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5659,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10529,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":304,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.339931+0000","last_change":"2024-03-26T03:55:59.324661+0000","last_active":"2024-03-26T03:56:59.339931+0000","last_peered":"2024-03-26T03:56:59.339931+0000","last_clean":"2024-03-26T03:56:59.339931+0000","last_became_active":"2024-03-26T03:55:59.324158+0000","last_became_peered":"2024-03-26T03:55:59.324158+0000","last_unstale":"2024-03-26T03:56:59.339931+0000","last_undegraded":"2024-03-26T03:56:59.339931+0000","last_fullsized":"2024-03-26T03:56:59.339931+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":276,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327074+0000","last_change":"2024-03-26T03:55:59.304498+0000","last_active":"2024-03-26T03:57:01.327074+0000","last_peered":"2024-03-26T03:57:01.327074+0000","last_clean":"2024-03-26T03:57:01.327074+0000","last_became_active":"2024-03-26T03:55:59.298094+0000","last_became_peered":"2024-03-26T03:55:59.298094+0000","last_unstale":"2024-03-26T03:57:01.327074+0000","last_undegraded":"2024-03-26T03:57:01.327074+0000","last_fullsized":"2024-03-26T03:57:01.327074+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313558+0000","last_change":"2024-03-26T03:55:59.299304+0000","last_active":"2024-03-26T03:56:59.313558+0000","last_peered":"2024-03-26T03:56:59.313558+0000","last_clean":"2024-03-26T03:56:59.313558+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T03:56:59.313558+0000","last_undegraded":"2024-03-26T03:56:59.313558+0000","last_fullsized":"2024-03-26T03:56:59.313558+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"181'12","reported_seq":431,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331082+0000","last_change":"2024-03-26T03:55:59.310069+0000","last_active":"2024-03-26T03:56:59.331082+0000","last_peered":"2024-03-26T03:56:59.331082+0000","last_clean":"2024-03-26T03:56:59.331082+0000","last_became_active":"2024-03-26T03:55:59.309116+0000","last_became_peered":"2024-03-26T03:55:59.309116+0000","last_unstale":"2024-03-26T03:56:59.331082+0000","last_undegraded":"2024-03-26T03:56:59.331082+0000","last_fullsized":"2024-03-26T03:56:59.331082+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"201'10504","reported_seq":12683,"reported_epoch":201,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.764434+0000","last_change":"2024-03-26T03:56:38.491697+0000","last_active":"2024-03-26T03:56:59.764434+0000","last_peered":"2024-03-26T03:56:59.764434+0000","last_clean":"2024-03-26T03:55:19.255545+0000","last_became_active":"2024-03-26T03:56:00.572469+0000","last_became_peered":"2024-03-26T03:56:00.572469+0000","last_unstale":"2024-03-26T03:56:59.764434+0000","last_undegraded":"2024-03-26T03:56:00.311081+0000","last_fullsized":"2024-03-26T03:56:00.282491+0000","mapping_epoch":181,"log_start":"174'5900","ondisk_log_start":"174'5900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4604,"log_dups_size":0,"ondisk_log_size":4604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371130392,"num_objects":5664,"num_object_clones":0,"num_object_copies":16992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2140,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5664,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10504,"num_write_kb":517313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1034,"num_bytes_recovered":67764224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":3524}],"blocked_by":[],"up_primary":5,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":306,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319455+0000","last_change":"2024-03-26T03:53:34.666684+0000","last_active":"2024-03-26T03:57:01.319455+0000","last_peered":"2024-03-26T03:57:01.319455+0000","last_clean":"2024-03-26T03:57:01.319455+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:57:01.319455+0000","last_undegraded":"2024-03-26T03:57:01.319455+0000","last_fullsized":"2024-03-26T03:57:01.319455+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:34.666607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.666607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":309,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.334168+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:56:59.334168+0000","last_peered":"2024-03-26T03:56:59.334168+0000","last_clean":"2024-03-26T03:56:59.334168+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:56:59.334168+0000","last_undegraded":"2024-03-26T03:56:59.334168+0000","last_fullsized":"2024-03-26T03:56:59.334168+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":370,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313374+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T03:56:59.313374+0000","last_peered":"2024-03-26T03:56:59.313374+0000","last_clean":"2024-03-26T03:56:59.313374+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T03:56:59.313374+0000","last_undegraded":"2024-03-26T03:56:59.313374+0000","last_fullsized":"2024-03-26T03:56:59.313374+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"154'10","reported_seq":176,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.323379+0000","last_change":"2024-03-26T03:54:34.943516+0000","last_active":"2024-03-26T03:56:59.323379+0000","last_peered":"2024-03-26T03:56:59.323379+0000","last_clean":"2024-03-26T03:56:59.323379+0000","last_became_active":"2024-03-26T03:54:29.356520+0000","last_became_peered":"2024-03-26T03:54:29.356520+0000","last_unstale":"2024-03-26T03:56:59.323379+0000","last_undegraded":"2024-03-26T03:56:59.323379+0000","last_fullsized":"2024-03-26T03:56:59.323379+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.c","version":"201'10456","reported_seq":16662,"reported_epoch":201,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.961125+0000","last_change":"2024-03-26T03:56:00.576519+0000","last_active":"2024-03-26T03:56:59.961125+0000","last_peered":"2024-03-26T03:56:59.961125+0000","last_clean":"2024-03-26T03:55:19.248623+0000","last_became_active":"2024-03-26T03:56:00.560122+0000","last_became_peered":"2024-03-26T03:56:00.560122+0000","last_unstale":"2024-03-26T03:56:59.961125+0000","last_undegraded":"2024-03-26T03:56:00.315936+0000","last_fullsized":"2024-03-26T03:56:00.286924+0000","mapping_epoch":181,"log_start":"177'6500","ondisk_log_start":"177'6500","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3956,"log_dups_size":0,"ondisk_log_size":3956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369491968,"num_objects":5638,"num_object_clones":0,"num_object_copies":16914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":51,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5638,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10456,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3047,"num_bytes_recovered":199688192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":5587}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312735+0000","last_change":"2024-03-26T03:54:29.432125+0000","last_active":"2024-03-26T03:56:59.312735+0000","last_peered":"2024-03-26T03:56:59.312735+0000","last_clean":"2024-03-26T03:56:59.312735+0000","last_became_active":"2024-03-26T03:54:29.431697+0000","last_became_peered":"2024-03-26T03:54:29.431697+0000","last_unstale":"2024-03-26T03:56:59.312735+0000","last_undegraded":"2024-03-26T03:56:59.312735+0000","last_fullsized":"2024-03-26T03:56:59.312735+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":120,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319570+0000","last_change":"2024-03-26T03:55:59.294273+0000","last_active":"2024-03-26T03:57:01.319570+0000","last_peered":"2024-03-26T03:57:01.319570+0000","last_clean":"2024-03-26T03:57:01.319570+0000","last_became_active":"2024-03-26T03:55:59.293987+0000","last_became_peered":"2024-03-26T03:55:59.293987+0000","last_unstale":"2024-03-26T03:57:01.319570+0000","last_undegraded":"2024-03-26T03:57:01.319570+0000","last_fullsized":"2024-03-26T03:57:01.319570+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":237,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.319301+0000","last_change":"2024-03-26T03:54:29.536599+0000","last_active":"2024-03-26T03:57:01.319301+0000","last_peered":"2024-03-26T03:57:01.319301+0000","last_clean":"2024-03-26T03:57:01.319301+0000","last_became_active":"2024-03-26T03:54:29.358363+0000","last_became_peered":"2024-03-26T03:54:29.358363+0000","last_unstale":"2024-03-26T03:57:01.319301+0000","last_undegraded":"2024-03-26T03:57:01.319301+0000","last_fullsized":"2024-03-26T03:57:01.319301+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312637+0000","last_change":"2024-03-26T03:55:22.211690+0000","last_active":"2024-03-26T03:56:59.312637+0000","last_peered":"2024-03-26T03:56:59.312637+0000","last_clean":"2024-03-26T03:56:59.312637+0000","last_became_active":"2024-03-26T03:54:29.346424+0000","last_became_peered":"2024-03-26T03:54:29.346424+0000","last_unstale":"2024-03-26T03:56:59.312637+0000","last_undegraded":"2024-03-26T03:56:59.312637+0000","last_fullsized":"2024-03-26T03:56:59.312637+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"201'10769","reported_seq":10879,"reported_epoch":201,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.520582+0000","last_change":"2024-03-26T03:56:00.609756+0000","last_active":"2024-03-26T03:56:59.520582+0000","last_peered":"2024-03-26T03:56:59.520582+0000","last_clean":"2024-03-26T03:55:19.268201+0000","last_became_active":"2024-03-26T03:56:00.598736+0000","last_became_peered":"2024-03-26T03:56:00.598736+0000","last_unstale":"2024-03-26T03:56:59.520582+0000","last_undegraded":"2024-03-26T03:56:00.325858+0000","last_fullsized":"2024-03-26T03:56:00.284403+0000","mapping_epoch":181,"log_start":"182'7100","ondisk_log_start":"182'7100","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3669,"log_dups_size":0,"ondisk_log_size":3669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":382009344,"num_objects":5829,"num_object_clones":0,"num_object_copies":17487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3261,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10769,"num_write_kb":531136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2568}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":229,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327006+0000","last_change":"2024-03-26T03:55:59.341468+0000","last_active":"2024-03-26T03:57:01.327006+0000","last_peered":"2024-03-26T03:57:01.327006+0000","last_clean":"2024-03-26T03:57:01.327006+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T03:57:01.327006+0000","last_undegraded":"2024-03-26T03:57:01.327006+0000","last_fullsized":"2024-03-26T03:57:01.327006+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":419,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312814+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T03:56:59.312814+0000","last_peered":"2024-03-26T03:56:59.312814+0000","last_clean":"2024-03-26T03:56:59.312814+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T03:56:59.312814+0000","last_undegraded":"2024-03-26T03:56:59.312814+0000","last_fullsized":"2024-03-26T03:56:59.312814+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.323959+0000","last_change":"2024-03-26T03:54:31.922380+0000","last_active":"2024-03-26T03:56:59.323959+0000","last_peered":"2024-03-26T03:56:59.323959+0000","last_clean":"2024-03-26T03:56:59.323959+0000","last_became_active":"2024-03-26T03:54:29.377662+0000","last_became_peered":"2024-03-26T03:54:29.377662+0000","last_unstale":"2024-03-26T03:56:59.323959+0000","last_undegraded":"2024-03-26T03:56:59.323959+0000","last_fullsized":"2024-03-26T03:56:59.323959+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":350,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.314789+0000","last_change":"2024-03-26T03:54:29.386220+0000","last_active":"2024-03-26T03:56:59.314789+0000","last_peered":"2024-03-26T03:56:59.314789+0000","last_clean":"2024-03-26T03:56:59.314789+0000","last_became_active":"2024-03-26T03:54:29.299380+0000","last_became_peered":"2024-03-26T03:54:29.299380+0000","last_unstale":"2024-03-26T03:56:59.314789+0000","last_undegraded":"2024-03-26T03:56:59.314789+0000","last_fullsized":"2024-03-26T03:56:59.314789+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.835587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.835587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"200'10409","reported_seq":16881,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312519+0000","last_change":"2024-03-26T03:56:40.032243+0000","last_active":"2024-03-26T03:56:59.312519+0000","last_peered":"2024-03-26T03:56:59.312519+0000","last_clean":"2024-03-26T03:56:59.312519+0000","last_became_active":"2024-03-26T03:56:40.023931+0000","last_became_peered":"2024-03-26T03:56:40.023931+0000","last_unstale":"2024-03-26T03:56:59.312519+0000","last_undegraded":"2024-03-26T03:56:59.312519+0000","last_fullsized":"2024-03-26T03:56:59.312519+0000","mapping_epoch":196,"log_start":"177'6400","ondisk_log_start":"177'6400","created":155,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4009,"log_dups_size":0,"ondisk_log_size":4009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":367198208,"num_objects":5603,"num_object_clones":0,"num_object_copies":16809,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5603,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10409,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3178,"num_bytes_recovered":208273408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":127,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.326785+0000","last_change":"2024-03-26T03:55:59.342271+0000","last_active":"2024-03-26T03:57:01.326785+0000","last_peered":"2024-03-26T03:57:01.326785+0000","last_clean":"2024-03-26T03:57:01.326785+0000","last_became_active":"2024-03-26T03:55:59.341382+0000","last_became_peered":"2024-03-26T03:55:59.341382+0000","last_unstale":"2024-03-26T03:57:01.326785+0000","last_undegraded":"2024-03-26T03:57:01.326785+0000","last_fullsized":"2024-03-26T03:57:01.326785+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":271,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-03-26T03:57:01.327204+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T03:57:01.327204+0000","last_peered":"2024-03-26T03:57:01.327204+0000","last_clean":"2024-03-26T03:57:01.327204+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T03:57:01.327204+0000","last_undegraded":"2024-03-26T03:57:01.327204+0000","last_fullsized":"2024-03-26T03:57:01.327204+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312996+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:56:59.312996+0000","last_peered":"2024-03-26T03:56:59.312996+0000","last_clean":"2024-03-26T03:56:59.312996+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:56:59.312996+0000","last_undegraded":"2024-03-26T03:56:59.312996+0000","last_fullsized":"2024-03-26T03:56:59.312996+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":384,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312605+0000","last_change":"2024-03-26T03:55:59.289896+0000","last_active":"2024-03-26T03:56:59.312605+0000","last_peered":"2024-03-26T03:56:59.312605+0000","last_clean":"2024-03-26T03:56:59.312605+0000","last_became_active":"2024-03-26T03:55:59.289458+0000","last_became_peered":"2024-03-26T03:55:59.289458+0000","last_unstale":"2024-03-26T03:56:59.312605+0000","last_undegraded":"2024-03-26T03:56:59.312605+0000","last_fullsized":"2024-03-26T03:56:59.312605+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"201'10660","reported_seq":10764,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.492422+0000","last_change":"2024-03-26T03:54:38.382794+0000","last_active":"2024-03-26T03:56:59.492422+0000","last_peered":"2024-03-26T03:56:59.492422+0000","last_clean":"2024-03-26T03:56:59.492422+0000","last_became_active":"2024-03-26T03:54:38.382452+0000","last_became_peered":"2024-03-26T03:54:38.382452+0000","last_unstale":"2024-03-26T03:56:59.492422+0000","last_undegraded":"2024-03-26T03:56:59.492422+0000","last_fullsized":"2024-03-26T03:56:59.492422+0000","mapping_epoch":155,"log_start":"179'7000","ondisk_log_start":"179'7000","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":380895232,"num_objects":5812,"num_object_clones":0,"num_object_copies":17436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10660,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":318,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.322319+0000","last_change":"2024-03-26T03:55:55.887208+0000","last_active":"2024-03-26T03:56:59.322319+0000","last_peered":"2024-03-26T03:56:59.322319+0000","last_clean":"2024-03-26T03:56:59.322319+0000","last_became_active":"2024-03-26T03:54:29.296384+0000","last_became_peered":"2024-03-26T03:54:29.296384+0000","last_unstale":"2024-03-26T03:56:59.322319+0000","last_undegraded":"2024-03-26T03:56:59.322319+0000","last_fullsized":"2024-03-26T03:56:59.322319+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:55.887078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:55.887078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:11.476315+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":299,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312605+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:56:59.312605+0000","last_peered":"2024-03-26T03:56:59.312605+0000","last_clean":"2024-03-26T03:56:59.312605+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:56:59.312605+0000","last_undegraded":"2024-03-26T03:56:59.312605+0000","last_fullsized":"2024-03-26T03:56:59.312605+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":253,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312876+0000","last_change":"2024-03-26T03:55:59.299222+0000","last_active":"2024-03-26T03:56:59.312876+0000","last_peered":"2024-03-26T03:56:59.312876+0000","last_clean":"2024-03-26T03:56:59.312876+0000","last_became_active":"2024-03-26T03:55:59.298700+0000","last_became_peered":"2024-03-26T03:55:59.298700+0000","last_unstale":"2024-03-26T03:56:59.312876+0000","last_undegraded":"2024-03-26T03:56:59.312876+0000","last_fullsized":"2024-03-26T03:56:59.312876+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":332,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.313017+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:56:59.313017+0000","last_peered":"2024-03-26T03:56:59.313017+0000","last_clean":"2024-03-26T03:56:59.313017+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:56:59.313017+0000","last_undegraded":"2024-03-26T03:56:59.313017+0000","last_fullsized":"2024-03-26T03:56:59.313017+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"201'10481","reported_seq":10587,"reported_epoch":201,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-26T03:56:59.476179+0000","last_change":"2024-03-26T03:56:00.619485+0000","last_active":"2024-03-26T03:56:59.476179+0000","last_peered":"2024-03-26T03:56:59.476179+0000","last_clean":"2024-03-26T03:55:19.228504+0000","last_became_active":"2024-03-26T03:56:00.608636+0000","last_became_peered":"2024-03-26T03:56:00.608636+0000","last_unstale":"2024-03-26T03:56:59.476179+0000","last_undegraded":"2024-03-26T03:56:00.316145+0000","last_fullsized":"2024-03-26T03:56:00.283634+0000","mapping_epoch":181,"log_start":"179'6900","ondisk_log_start":"179'6900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":3581,"log_dups_size":0,"ondisk_log_size":3581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375848960,"num_objects":5735,"num_object_clones":0,"num_object_copies":17205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3284,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10481,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":2451}],"blocked_by":[],"up_primary":5,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":296,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312946+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:56:59.312946+0000","last_peered":"2024-03-26T03:56:59.312946+0000","last_clean":"2024-03-26T03:56:59.312946+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:56:59.312946+0000","last_undegraded":"2024-03-26T03:56:59.312946+0000","last_fullsized":"2024-03-26T03:56:59.312946+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":427,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.327387+0000","last_change":"2024-03-26T03:55:59.796875+0000","last_active":"2024-03-26T03:56:59.327387+0000","last_peered":"2024-03-26T03:56:59.327387+0000","last_clean":"2024-03-26T03:56:59.327387+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T03:56:59.327387+0000","last_undegraded":"2024-03-26T03:56:59.327387+0000","last_fullsized":"2024-03-26T03:56:59.327387+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:55:59.796779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:59.796779+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:57:23.173731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":115,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.331428+0000","last_change":"2024-03-26T03:54:33.815241+0000","last_active":"2024-03-26T03:56:59.331428+0000","last_peered":"2024-03-26T03:56:59.331428+0000","last_clean":"2024-03-26T03:56:59.331428+0000","last_became_active":"2024-03-26T03:54:29.399842+0000","last_became_peered":"2024-03-26T03:54:29.399842+0000","last_unstale":"2024-03-26T03:56:59.331428+0000","last_undegraded":"2024-03-26T03:56:59.331428+0000","last_fullsized":"2024-03-26T03:56:59.331428+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":355,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-26T03:56:59.312761+0000","last_change":"2024-03-26T03:55:17.192246+0000","last_active":"2024-03-26T03:56:59.312761+0000","last_peered":"2024-03-26T03:56:59.312761+0000","last_clean":"2024-03-26T03:56:59.312761+0000","last_became_active":"2024-03-26T03:54:29.296174+0000","last_became_peered":"2024-03-26T03:54:29.296174+0000","last_unstale":"2024-03-26T03:56:59.312761+0000","last_undegraded":"2024-03-26T03:56:59.312761+0000","last_fullsized":"2024-03-26T03:56:59.312761+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":5974720536,"num_objects":91168,"num_object_clones":0,"num_object_copies":273504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91168,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":168690,"num_write_kb":8315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10540,"num_bytes_recovered":690749440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16685281280,"data_stored":16685269064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":63629,"ondisk_log_size":63629,"up":48,"acting":41,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":492,"num_read_kb":487,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1393,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":2822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":621,"ondisk_log_size":621,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070568,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1957120,"kb_used_data":1749624,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":91788032,"statfs":{"total":95995035648,"available":93990944768,"internally_reserved":0,"allocated":1791614976,"data_stored":1790936543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":212466122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364910,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574982,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2899176,"kb_used_data":2588896,"kb_used_omap":1,"kb_used_meta":310206,"kb_avail":90845976,"statfs":{"total":95995035648,"available":93026279424,"internally_reserved":0,"allocated":2651029504,"data_stored":2650354645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":317651397},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738507,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3225716,"kb_used_data":2909928,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90519436,"statfs":{"total":95995035648,"available":92691902464,"internally_reserved":0,"allocated":2979766272,"data_stored":2979090258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411457,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4027636,"kb_used_data":3681324,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":89717516,"statfs":{"total":95995035648,"available":91870736384,"internally_reserved":0,"allocated":3769675776,"data_stored":3769051191,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182523,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953590,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3615936,"kb_used_data":3318200,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":90129216,"statfs":{"total":95995035648,"available":92292317184,"internally_reserved":0,"allocated":3397836800,"data_stored":3397199953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113294,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2288372,"kb_used_data":2122912,"kb_used_omap":4,"kb_used_meta":165435,"kb_avail":91456780,"statfs":{"total":95995035648,"available":93651742720,"internally_reserved":0,"allocated":2173861888,"data_stored":2173205767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":169406112},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2977759232,"data_stored":2977759232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2633367552,"data_stored":2633367552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3747024896,"data_stored":3747020824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3366780928,"data_stored":3366780928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2171211776,"data_stored":2171207704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1789136896,"data_stored":1789132824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:57:16.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-03-26T03:57:16.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:16.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:16.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:16.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:16.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:16.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:16.741 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:16.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:16 smithi086 ceph-mon[32965]: pgmap v365: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.15k op/s; 15830/236250 objects degraded (6.701%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:57:16.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:16.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:16.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:16.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:16 smithi006 ceph-mon[37501]: pgmap v365: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.15k op/s; 15830/236250 objects degraded (6.701%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:57:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:16 smithi006 ceph-mon[29446]: pgmap v365: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 927 B/s rd, 0 B/s wr, 1.15k op/s; 15830/236250 objects degraded (6.701%); 14 MiB/s, 224 objects/s recovering 2024-03-26T03:57:16.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:17.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:17.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:17.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:17.140 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:17.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:17.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:17.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:17.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:17.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:17.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T03:57:17.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:17.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:17.559 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:17.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.705 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:17.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:17.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:17.830 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:17.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:17.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:17.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:17.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:18.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:18.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:18.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:18.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:18.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:18.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:18.381 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:18.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:18.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:18.523 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:18.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:18.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:18.663 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:18.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:18.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:18.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:18.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:18 smithi086 ceph-mon[32965]: pgmap v366: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 809 op/s; 15830/235428 objects degraded (6.724%); 7.6 MiB/s, 121 objects/s recovering 2024-03-26T03:57:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:18 smithi006 ceph-mon[29446]: pgmap v366: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 809 op/s; 15830/235428 objects degraded (6.724%); 7.6 MiB/s, 121 objects/s recovering 2024-03-26T03:57:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:18 smithi006 ceph-mon[37501]: pgmap v366: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 809 op/s; 15830/235428 objects degraded (6.724%); 7.6 MiB/s, 121 objects/s recovering 2024-03-26T03:57:18.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:18.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:18.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:19.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:19.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:19.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:19.205 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:19.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:19.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:19.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:19.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:19.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:19.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:19.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:19.570 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:19.608 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:19.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:19.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:19.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:19.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:19.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:19 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:57:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:19 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:57:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:19 smithi086 ceph-mon[32965]: osdmap e205: 8 total, 8 up, 6 in 2024-03-26T03:57:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:19 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 15830/235428 objects degraded (6.724%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:19.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:19.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:19.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:57:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:57:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[37501]: osdmap e205: 8 total, 8 up, 6 in 2024-03-26T03:57:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 15830/235428 objects degraded (6.724%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:57:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:57:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[29446]: osdmap e205: 8 total, 8 up, 6 in 2024-03-26T03:57:19.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:19 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 15830/235428 objects degraded (6.724%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:19.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:20.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:20.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:20.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:20.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:20.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:20.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:20.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:20.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:20.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:20.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:20.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:20.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:20.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:20.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:20 smithi086 ceph-mon[32965]: pgmap v368: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 15830/232962 objects degraded (6.795%); 9.1 MiB/s, 145 objects/s recovering 2024-03-26T03:57:20.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:20 smithi006 ceph-mon[37501]: pgmap v368: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 15830/232962 objects degraded (6.795%); 9.1 MiB/s, 145 objects/s recovering 2024-03-26T03:57:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:20 smithi006 ceph-mon[29446]: pgmap v368: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 927 op/s; 15830/232962 objects degraded (6.795%); 9.1 MiB/s, 145 objects/s recovering 2024-03-26T03:57:20.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:20.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:20.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:21.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:21.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:21.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:21.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:21.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:21.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:21.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:21.470 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:21.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:21.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.614 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:21.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:21.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.755 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:21.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:21.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:21.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:21.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:21.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:22.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:22.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:22.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:22.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:22.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:22.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:22.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:22.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:22.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:22.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:22.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:22.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:22.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:22.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:22.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:22.830 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:22 smithi086 ceph-mon[32965]: pgmap v369: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s; 15042/223284 objects degraded (6.737%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:22.861 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:22.865 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:22 smithi006 ceph-mon[29446]: pgmap v369: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s; 15042/223284 objects degraded (6.737%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:22.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:22 smithi006 ceph-mon[37501]: pgmap v369: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 855 op/s; 15042/223284 objects degraded (6.737%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:22.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:22.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:22.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:23.011 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:57:23.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:23.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:23.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:23.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:23.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:23.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:23.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:23.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:23.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:23.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:23.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:23.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:23.824 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:23.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:23.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:23.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:23.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:24.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:24.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:24.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:24.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:24.227 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:24.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:24.229 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:24.329 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:24.382 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:24.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:24.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:24.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:24.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: pgmap v370: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 15042/222261 objects degraded (6.768%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: osdmap e206: 8 total, 8 up, 6 in 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 15042/222261 objects degraded (6.768%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: pgmap v370: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 15042/222261 objects degraded (6.768%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: osdmap e206: 8 total, 8 up, 6 in 2024-03-26T03:57:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:24.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 15042/222261 objects degraded (6.768%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:24.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:24.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:24.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:24.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: pgmap v370: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.5 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 860 op/s; 15042/222261 objects degraded (6.768%); 9.6 MiB/s, 153 objects/s recovering 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: osdmap e206: 8 total, 8 up, 6 in 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 15042/222261 objects degraded (6.768%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:24.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:24.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:24.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:24.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:25.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:25.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:25.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:25.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:25.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:25.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:25.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:25.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:25.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:25.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:25.483 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:25.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:25.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:25.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:25.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:25.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:25.765 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:25.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:25.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:25.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:25.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:26.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:26.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:26.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:26.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:26.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.287 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:26.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:26.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:26.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:26.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:26.525 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:57:26.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:26.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:26.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:26.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:26.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:26 smithi086 ceph-mon[32965]: pgmap v372: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14116/207123 objects degraded (6.815%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:26 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2565578398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:26 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:26.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[37501]: pgmap v372: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14116/207123 objects degraded (6.815%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2565578398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[29446]: pgmap v372: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 14116/207123 objects degraded (6.815%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2565578398' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:26 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:57:26.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:26.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:27.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:27.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:27.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:27.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:27.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:27.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:27.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.407 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:27.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:27.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:27.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:27.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:27.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:27.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:27.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:27 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:27 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:57:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:27 smithi086 ceph-mon[32965]: osdmap e207: 8 total, 8 up, 6 in 2024-03-26T03:57:27.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:27.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:57:27.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[37501]: osdmap e207: 8 total, 8 up, 6 in 2024-03-26T03:57:27.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:27.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:57:27.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:27 smithi006 ceph-mon[29446]: osdmap e207: 8 total, 8 up, 6 in 2024-03-26T03:57:27.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:27.924 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:27.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:27.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:28.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:28.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:28.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:28.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:28.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:28.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:28.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:28.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:28.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:28.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:28.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:28.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:28.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:28.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:28.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:28 smithi006 ceph-mon[37501]: pgmap v374: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 14116/206121 objects degraded (6.848%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:28.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:28 smithi006 ceph-mon[37501]: osdmap e208: 8 total, 8 up, 6 in 2024-03-26T03:57:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:28 smithi006 ceph-mon[29446]: pgmap v374: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 14116/206121 objects degraded (6.848%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:28.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:28 smithi006 ceph-mon[29446]: osdmap e208: 8 total, 8 up, 6 in 2024-03-26T03:57:28.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:28.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:28.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:28.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:28.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:28 smithi086 ceph-mon[32965]: pgmap v374: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.2 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 14116/206121 objects degraded (6.848%); 13 MiB/s, 214 objects/s recovering 2024-03-26T03:57:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:28 smithi086 ceph-mon[32965]: osdmap e208: 8 total, 8 up, 6 in 2024-03-26T03:57:28.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:28.963 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:28.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:28.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:29.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:29.104 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:29.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:29.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:29.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:29.377 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:29.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:29.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:29.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:29.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:29.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:29 smithi086 ceph-mon[32965]: osdmap e209: 8 total, 8 up, 6 in 2024-03-26T03:57:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 14116/206121 objects degraded (6.848%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:29.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:29 smithi006 ceph-mon[29446]: osdmap e209: 8 total, 8 up, 6 in 2024-03-26T03:57:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 14116/206121 objects degraded (6.848%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:29 smithi006 ceph-mon[37501]: osdmap e209: 8 total, 8 up, 6 in 2024-03-26T03:57:29.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 14116/206121 objects degraded (6.848%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:29.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:30.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:30.046 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:30.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:30.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:30.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:30.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:30.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:30.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:30.453 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:30.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:30.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:30.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:30.710 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:30.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:30.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:30 smithi086 ceph-mon[32965]: pgmap v377: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 14116/203754 objects degraded (6.928%); 9.9 MiB/s, 157 objects/s recovering 2024-03-26T03:57:30.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:30 smithi006 ceph-mon[37501]: pgmap v377: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 14116/203754 objects degraded (6.928%); 9.9 MiB/s, 157 objects/s recovering 2024-03-26T03:57:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:30 smithi006 ceph-mon[29446]: pgmap v377: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.05k op/s; 14116/203754 objects degraded (6.928%); 9.9 MiB/s, 157 objects/s recovering 2024-03-26T03:57:30.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:30.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:30.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:30.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:31.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:31.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:31.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:31.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:31.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:31.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:31.400 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:31.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:31.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:31.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:31.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:31.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:31.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:31.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:31.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:31.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:31.948 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:31.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:31.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:32.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:32.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:32.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:32.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:32.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:32.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:32.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:32.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:32.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:32.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:32.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:32.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:32.626 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:32.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:32.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:32.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:32 smithi086 ceph-mon[32965]: pgmap v378: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 13317/195438 objects degraded (6.814%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:57:32.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:32 smithi006 ceph-mon[29446]: pgmap v378: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 13317/195438 objects degraded (6.814%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:57:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:32 smithi006 ceph-mon[37501]: pgmap v378: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 649 op/s; 13317/195438 objects degraded (6.814%); 8.3 MiB/s, 133 objects/s recovering 2024-03-26T03:57:32.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:32.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:32.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:32.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:33.037 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:33.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:33.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:33.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:33.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:33.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:33.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:33.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 8 pgp_num to 16 2024-03-26T03:57:33.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:33.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:33.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:33.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3555257606' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:33 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:33.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3555257606' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:33 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3555257606' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:33 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T03:57:33.940 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:33.980 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:33.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:34.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:57:34.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:34.142 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:34.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:34.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:34.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:34.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:34.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:34.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:34.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:34.532 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:34.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:34.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:34.696 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:34.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:34.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:34.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:34.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:34.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:34 smithi086 ceph-mon[32965]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 667 op/s; 13037/193413 objects degraded (6.740%); 9.8 MiB/s, 157 objects/s recovering 2024-03-26T03:57:34.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:34 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:57:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:34 smithi086 ceph-mon[32965]: osdmap e210: 8 total, 8 up, 6 in 2024-03-26T03:57:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 13037/193413 objects degraded (6.740%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:34.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[37501]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 667 op/s; 13037/193413 objects degraded (6.740%); 9.8 MiB/s, 157 objects/s recovering 2024-03-26T03:57:34.876 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:57:34.876 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[37501]: osdmap e210: 8 total, 8 up, 6 in 2024-03-26T03:57:34.876 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 13037/193413 objects degraded (6.740%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:34.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[29446]: pgmap v379: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.9 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 747 B/s rd, 0 B/s wr, 667 op/s; 13037/193413 objects degraded (6.740%); 9.8 MiB/s, 157 objects/s recovering 2024-03-26T03:57:34.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T03:57:34.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[29446]: osdmap e210: 8 total, 8 up, 6 in 2024-03-26T03:57:34.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 13037/193413 objects degraded (6.740%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:34.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:34.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:34.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:34.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:35.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:35.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:35.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:35.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:35.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:35.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:35.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:35.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:35.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:35.474 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201340 8.a 8:5ca8a93b:::benchmark_data_smithi006_103169_object71409:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.449060+0000", 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.022139549000000001, 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.022173724999999998, 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201340 2024-03-26T03:57:35.475 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449060+0000", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449060+0000", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449062+0000", 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.387e-06 2024-03-26T03:57:35.476 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449068+0000", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3609999999999997e-06 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449078+0000", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0556e-05 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.449086+0000", 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9920000000000007e-06 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.477 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454190+0000", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051037310000000002 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454361+0000", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017063500000000001 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.456084+0000", 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001723069 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.478 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201344 8.3 8:c7b6e3d8:::benchmark_data_smithi006_103169_object71413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.454586+0000", 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.016614119, 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016746475, 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201344 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.479 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454586+0000", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454586+0000", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454587+0000", 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.269e-06 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.480 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454593+0000", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5910000000000001e-06 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454607+0000", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3427999999999999e-05 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.454615+0000", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5920000000000001e-06 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.464062+0000", 2024-03-26T03:57:35.481 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009446121 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.464218+0000", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015641799999999999 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.464368+0000", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014950799999999999 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468617+0000", 2024-03-26T03:57:35.482 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0042496269999999997 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:201324 8.a e210/181)", 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.455953+0000", 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.015247047, 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015464208, 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.483 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455953+0000", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455953+0000", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455954+0000", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1340000000000001e-06 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455961+0000", 2024-03-26T03:57:35.484 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6950000000000003e-06 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455970+0000", 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7210000000000006e-06 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.455978+0000", 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4050000000000003e-06 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.485 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201348 8.1 8:88c3d26d:::benchmark_data_smithi006_103169_object71417:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.457110+0000", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.014089328, 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014360316999999999, 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201348 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.486 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457110+0000", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457110+0000", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457112+0000", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.201e-06 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457117+0000", 2024-03-26T03:57:35.487 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7289999999999997e-06 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457127+0000", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7799999999999995e-06 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.457134+0000", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2699999999999999e-06 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459726+0000", 2024-03-26T03:57:35.488 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025919229999999999 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459870+0000", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014334699999999999 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.460004+0000", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013411 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.463928+0000", 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039242549999999998 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.489 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468551+0000", 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046229890000000001 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:201340 8.a e210/181)", 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.459696+0000", 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.011503406000000001, 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011859344000000001, 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.490 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459696+0000", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459696+0000", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459697+0000", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7599999999999997e-07 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.491 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459702+0000", 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7160000000000002e-06 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459711+0000", 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1500000000000005e-06 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.459717+0000", 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3069999999999999e-06 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.492 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDPGPushReply(8.3 210/181 [PushReplyOp(8:caec9d4c:::benchmark_data_smithi006_103169_object85225:head)])", 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.466529+0000", 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0046711510000000001, 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050765949999999997, 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466529+0000", 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.493 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466529+0000", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466530+0000", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4729999999999999e-06 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466539+0000", 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2189999999999993e-06 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.494 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466555+0000", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5556000000000001e-05 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.466564+0000", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3829999999999998e-06 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.470870+0000", 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004305344 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.495 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:201353 8.b e210/155 8:db63dddb:::benchmark_data_smithi006_103169_object71422:head v 210'12853, mlcod=210'12851)", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.468408+0000", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0027914630000000001, 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003253059, 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468408+0000", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.496 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468408+0000", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468409+0000", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9900000000000009e-07 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468415+0000", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7490000000000004e-06 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468425+0000", 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.533e-06 2024-03-26T03:57:35.497 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468432+0000", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6389999999999992e-06 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468648+0000", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021618700000000001 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468666+0000", 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7510999999999999e-05 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201351 8.7 8:e5793545:::benchmark_data_smithi006_103169_object71420:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.468755+0000", 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.002444706, 2024-03-26T03:57:35.499 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029680209999999999, 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201351 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468755+0000", 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.500 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468755+0000", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468757+0000", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5650000000000001e-06 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468762+0000", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6239999999999999e-06 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.501 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468774+0000", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1457000000000001e-05 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468782+0000", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8970000000000008e-06 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.470980+0000", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002198527 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.502 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.471106+0000", 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012541399999999999 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.471234+0000", 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012810400000000001 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201354 8.0 8:06310363:::benchmark_data_smithi006_103169_object71423:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.468802+0000", 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0023979819999999999, 2024-03-26T03:57:35.503 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029986850000000001, 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201354 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468802+0000", 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.504 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468802+0000", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468802+0000", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8100000000000003e-07 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468805+0000", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6539999999999999e-06 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.505 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468811+0000", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9989999999999997e-06 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468816+0000", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.48e-06 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.468885+0000", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8554999999999997e-05 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.469062+0000", 2024-03-26T03:57:35.506 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017745099999999999 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,5", 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.469230+0000", 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016782799999999999 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:201324 8.a 8:536aaaa0:::benchmark_data_smithi006_103169_object71393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e210)", 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:35.435247+0000", 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.035952980000000002, 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.036629836999999998, 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:35.507 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 201324 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435247+0000", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:35.508 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435247+0000", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435248+0000", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6099999999999999e-07 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435251+0000", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6009999999999999e-06 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435260+0000", 2024-03-26T03:57:35.509 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2779999999999993e-06 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.435267+0000", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1489999999999997e-06 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.451585+0000", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016317773000000001 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.451690+0000", 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010512400000000001 2024-03-26T03:57:35.510 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:35.511 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:35.451839+0000", 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014957499999999999 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:35.513 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:57:35.514 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 10 2024-03-26T03:57:35.514 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:57:35.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:35.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:35.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:35.661 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:35.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:35.701 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:57:35.743 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:57:35.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:35.744 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:35.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:35.781 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:57:35.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:35.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:35.829 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:57:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:35.870 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:57:35.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:35.914 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:57:35.954 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:57:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:35.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:35.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:35.995 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:57:36.001 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:57:36.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:36.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:36.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:36.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.253 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:36.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:36.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:36.394 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:36.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:36.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T03:57:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:36.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:36.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:36.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:36.763 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:57:36.763 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:57:36.763 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[37501]: pgmap v381: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 12265/178965 objects degraded (6.853%); 15 MiB/s, 236 objects/s recovering 2024-03-26T03:57:36.763 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[37501]: osdmap e211: 8 total, 8 up, 6 in 2024-03-26T03:57:36.763 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1892854897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:36.764 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:57:36.764 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:57:36.764 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[29446]: pgmap v381: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 12265/178965 objects degraded (6.853%); 15 MiB/s, 236 objects/s recovering 2024-03-26T03:57:36.765 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[29446]: osdmap e211: 8 total, 8 up, 6 in 2024-03-26T03:57:36.765 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1892854897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:36.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:36.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:36 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:57:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:36 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:57:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:36 smithi086 ceph-mon[32965]: pgmap v381: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.16k op/s; 12265/178965 objects degraded (6.853%); 15 MiB/s, 236 objects/s recovering 2024-03-26T03:57:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:36 smithi086 ceph-mon[32965]: osdmap e211: 8 total, 8 up, 6 in 2024-03-26T03:57:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1892854897' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:36.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:36.981 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:36.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:37.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:37.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:37.207 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:37.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:37.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:37.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:37.272 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:37.356 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:37.401 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:37.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:37.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:37.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:37.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:37.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:37.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:37.805 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:37.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:37.951 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:37.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:38.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:38.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:38.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:38.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:38.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:38.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:38.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:38.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:38.513 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:38.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:38.644 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:38.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:38.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:38 smithi086 ceph-mon[32965]: pgmap v383: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 12265/177096 objects degraded (6.926%); 14 MiB/s, 231 objects/s recovering 2024-03-26T03:57:38.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:38 smithi006 ceph-mon[37501]: pgmap v383: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 12265/177096 objects degraded (6.926%); 14 MiB/s, 231 objects/s recovering 2024-03-26T03:57:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:38 smithi006 ceph-mon[29446]: pgmap v383: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s; 12265/177096 objects degraded (6.926%); 14 MiB/s, 231 objects/s recovering 2024-03-26T03:57:38.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:38.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:39.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:39.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:39.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:39.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:39.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:57:39.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:39.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:39.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:39.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:39.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:39.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:39.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:39.608 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:39.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:39.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:39.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:39.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:39.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:39.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:39.924 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7] out_osds: [6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:57:39.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:57:39.925 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-26T03:57:39.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd in 3 2024-03-26T03:57:39.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:40.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:40.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:40.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:40.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:40.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:40.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:40.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 12265/177096 objects degraded (6.926%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:39 smithi086 ceph-mon[32965]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 12265/174192 objects degraded (7.041%); 8.2 MiB/s, 131 objects/s recovering 2024-03-26T03:57:40.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:40.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 12265/177096 objects degraded (6.926%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[29446]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 12265/174192 objects degraded (7.041%); 8.2 MiB/s, 131 objects/s recovering 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 12265/177096 objects degraded (6.926%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:40.388 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:39 smithi006 ceph-mon[37501]: pgmap v384: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 885 op/s; 12265/174192 objects degraded (7.041%); 8.2 MiB/s, 131 objects/s recovering 2024-03-26T03:57:40.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:40.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:40.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:40.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:40.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:40.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:40.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:40.851 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:40.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:40.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:40.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:40.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:40.953 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:57:40.954 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:57:40.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:41.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:41.023 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:57:41.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:41.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:41.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:41.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:41.302 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:41 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1863748251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:41 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:41 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:57:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:41 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:57:41.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1863748251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:57:41.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:57:41.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1863748251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:57:41.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:57:41.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:41 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:57:41.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:41.437 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:41.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:41.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:41.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:41.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:41.726 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:41.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:41.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:41.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:41.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:42.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:42.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:42.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:42.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[37501]: osdmap e212: 8 total, 8 up, 6 in 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[37501]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 11432/163347 objects degraded (6.999%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[29446]: osdmap e212: 8 total, 8 up, 6 in 2024-03-26T03:57:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:42 smithi006 ceph-mon[29446]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 11432/163347 objects degraded (6.999%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:42.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:42.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:42.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:42 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:57:42.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:42 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:57:42.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:42 smithi086 ceph-mon[32965]: osdmap e212: 8 total, 8 up, 6 in 2024-03-26T03:57:42.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:42 smithi086 ceph-mon[32965]: pgmap v386: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.28k op/s; 11432/163347 objects degraded (6.999%); 13 MiB/s, 205 objects/s recovering 2024-03-26T03:57:42.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:42.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:42.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:42.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:42.659 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:42.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:42.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:42.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:42.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T03:57:42.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:42.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:43.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:43.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:43.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:43.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:43.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:43.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:43.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:43.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:43.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:43.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:43.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:43.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:43.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:43.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:43.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:43.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:43.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:57:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:43.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:44.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:44.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:44.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:44.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:44.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:44.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:44.253 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:44.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:44.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:44.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:44.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:44.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:44.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:44 smithi006 ceph-mon[37501]: pgmap v387: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 11432/161073 objects degraded (7.097%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T03:57:44.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 11432/161073 objects degraded (7.097%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:44.589 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:44.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:44 smithi006 ceph-mon[29446]: pgmap v387: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 11432/161073 objects degraded (7.097%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T03:57:44.590 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 11432/161073 objects degraded (7.097%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:44.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:44.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:44.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:44 smithi086 ceph-mon[32965]: pgmap v387: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 745 op/s; 11432/161073 objects degraded (7.097%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T03:57:44.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 11432/161073 objects degraded (7.097%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:44.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:44.751 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:44.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:44.894 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:44.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:44.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:45.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:45.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:45.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:45.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:57:45.204 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:57:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:57:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:57:45.216 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:57:45.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:45.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:45.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:45.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:45.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:45.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:45.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:45.614 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:45.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:45.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:45.753 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:45 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4096467190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:45.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:45.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:45.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:45 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4096467190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:45.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:45.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:45 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4096467190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:57:45.893 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:45.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:45.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:46.034 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:46.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:46.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:46.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:46.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:46.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:46.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:46.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:46.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:46.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.451 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:46.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:46.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:46.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:46.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:46.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:46.732 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:46.813 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:46 smithi006 ceph-mon[37501]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s; 10506/145986 objects degraded (7.197%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:57:46.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:46 smithi006 ceph-mon[29446]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s; 10506/145986 objects degraded (7.197%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:57:46.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:46 smithi086 ceph-mon[32965]: pgmap v388: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 3.0 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.12k op/s; 10506/145986 objects degraded (7.197%); 11 MiB/s, 179 objects/s recovering 2024-03-26T03:57:46.910 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:46.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:47.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:47.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:47.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:47.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:47.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:47.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:47.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:47.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:47.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:47.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:47.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:47.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:47.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.751 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:47.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:47.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:47.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:47.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:47.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:47.932 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:48.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:47 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T03:57:47.678+0000 7fd465f8b700 -1 mon.a@0(leader).osd e213 definitely_dead 0 2024-03-26T03:57:48.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:48.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:48.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:48.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:48.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:48.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:48.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:48.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:48.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:48.501 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.3. 2024-03-26T03:57:48.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:48.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.614 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:48.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 10506/143784 objects degraded (7.307%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: osdmap e213: 8 total, 8 up, 6 in 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1711953590' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 10506/143784 objects degraded (7.307%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:57:48.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:57:48.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: osdmap e213: 8 total, 8 up, 6 in 2024-03-26T03:57:48.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1711953590' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:48 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:48.741 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: pgmap v389: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.11k op/s; 10506/143784 objects degraded (7.307%); 11 MiB/s, 175 objects/s recovering 2024-03-26T03:57:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:57:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:57:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: osdmap e213: 8 total, 8 up, 6 in 2024-03-26T03:57:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1711953590' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:48 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T03:57:48.843 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:48.870 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:48.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.963 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1970 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch ping cookie 94232629237248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136393+0000", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 199.824683792, 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25019128699999998, 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1970 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136393+0000", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136393+0000", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136395+0000", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.922e-06 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136401+0000", 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7400000000000001e-06 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136421+0000", 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0676e-05 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136430+0000", 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1869999999999997e-06 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136576+0000", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014601499999999999 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136593+0000", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7632999999999999e-05 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.371858+0000", 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.23526502599999999 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.372211+0000", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035229700000000001 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.382948+0000", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010737516000000001 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.383694+0000", 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00074637800000000002 2024-03-26T03:57:48.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.386584+0000", 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028898970000000002 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1980 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 94232629237248 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136846+0000", 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 199.824231271, 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.28231289199999998, 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1980 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136846+0000", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136846+0000", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136846+0000", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0599999999999999e-07 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136852+0000", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1019999999999996e-06 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136864+0000", 2024-03-26T03:57:48.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2806000000000001e-05 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136872+0000", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1049999999999995e-06 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136930+0000", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7806999999999997e-05 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136942+0000", 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.185e-05 2024-03-26T03:57:48.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.372242+0000", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.235299968 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.372293+0000", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0528999999999997e-05 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.386705+0000", 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014412067000000001 2024-03-26T03:57:48.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.386782+0000", 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7729000000000001e-05 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.419158+0000", 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.032376022999999997 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1981 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94232629231488 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T03:57:48.974 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136890+0000", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 199.82418725400001, 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.34290080899999997, 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1981 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136890+0000", 2024-03-26T03:57:48.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136890+0000", 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136890+0000", 2024-03-26T03:57:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9299999999999998e-07 2024-03-26T03:57:48.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136893+0000", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1690000000000001e-06 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136901+0000", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.469e-06 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136907+0000", 2024-03-26T03:57:48.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3659999999999997e-06 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.137009+0000", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102209 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.137016+0000", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1049999999999997e-06 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.372880+0000", 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.23586351699999999 2024-03-26T03:57:48.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.373144+0000", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026444400000000002 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.415771+0000", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.042626828999999998 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.415858+0000", 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7237999999999994e-05 2024-03-26T03:57:48.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.479790+0000", 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.063931970000000005 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179821 8.b e204/155 8:d635075a:::benchmark_data_smithi006_103169_object49890:head v 204'11473, mlcod=204'11471)", 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.759007+0000", 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 36.202070143999997, 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19189674100000001, 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.981 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759007+0000", 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759007+0000", 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759007+0000", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2500000000000001e-07 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759012+0000", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8300000000000003e-06 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759019+0000", 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0720000000000002e-06 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.759025+0000", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1789999999999996e-06 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768789+0000", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0097639530000000006 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768811+0000", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2311999999999999e-05 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:48.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.775117+0000", 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0063059700000000002 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.950903+0000", 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.175786 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179832 8.3 8:c0756243:::benchmark_data_smithi006_103169_object49901:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.763279+0000", 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 36.197797706000003, 2024-03-26T03:57:48.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19090870300000001, 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179832 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763279+0000", 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763279+0000", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763280+0000", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0100000000000005e-07 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763283+0000", 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.54e-06 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.987 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763291+0000", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4120000000000004e-06 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.763297+0000", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1079999999999998e-06 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.770650+0000", 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0073535470000000002 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.770773+0000", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000123146 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.770901+0000", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127644 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.951406+0000", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18050535000000001 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:48.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.952263+0000", 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00085606199999999999 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.952304+0000", 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1282000000000002e-05 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.952363+0000", 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8936e-05 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.954188+0000", 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001825075 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179836 8.3 8:c4d80d03:::benchmark_data_smithi006_103169_object49905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.768464+0000", 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 36.192612687, 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18734653000000001, 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:48.991 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179836 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768464+0000", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768464+0000", 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768465+0000", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.51e-07 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768468+0000", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3639999999999999e-06 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768475+0000", 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0350000000000004e-06 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.768481+0000", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0270000000000001e-06 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772923+0000", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044415940000000001 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773063+0000", 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000140555 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773204+0000", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000141187 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.952247+0000", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17904267099999999 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.952269+0000", 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1929999999999998e-05 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.954238+0000", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019691829999999998 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.954296+0000", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7394e-05 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.955811+0000", 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015150389999999999 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.996 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179846 8.0 8:04196575:::benchmark_data_smithi006_103169_object49915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.775849+0000", 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 36.185227318999999, 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18435942299999999, 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179846 2024-03-26T03:57:48.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.775849+0000", 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.775849+0000", 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:48.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.775851+0000", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.223e-06 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.775856+0000", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4269999999999999e-06 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943440+0000", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16758377299999999 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:48.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943459+0000", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8640000000000001e-05 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943543+0000", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4930000000000002e-05 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943714+0000", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017066 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,5", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943864+0000", 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014996700000000001 2024-03-26T03:57:49.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953540+0000", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096761409999999992 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953564+0000", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3961e-05 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.955750+0000", 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021854520000000001 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.001 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.955852+0000", 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010234900000000001 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.960209+0000", 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043569000000000004 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.002 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180179 8.3 8:c571072a:::benchmark_data_smithi006_103169_object50248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.219966+0000", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 35.741110353000003, 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20578044600000001, 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180179 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219966+0000", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219966+0000", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219968+0000", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2410000000000001e-06 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219973+0000", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4040000000000003e-06 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219983+0000", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7960000000000007e-06 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.219990+0000", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.418e-06 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.291332+0000", 2024-03-26T03:57:49.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.071341317000000001 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.291528+0000", 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019643800000000001 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.291685+0000", 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000157021 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.296263+0000", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045778490000000002 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.296316+0000", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3526999999999998e-05 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.423985+0000", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.127668804 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.424079+0000", 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3385000000000001e-05 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.425747+0000", 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001668246 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180191 8.7 8:e56fbb99:::benchmark_data_smithi006_103169_object50260:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.228511+0000", 2024-03-26T03:57:49.008 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 35.732565346999998, 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.196837706, 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180191 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228511+0000", 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228511+0000", 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228513+0000", 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5039999999999999e-06 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228518+0000", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9300000000000002e-06 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228527+0000", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4199999999999996e-06 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228535+0000", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2790000000000001e-06 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228583+0000", 2024-03-26T03:57:49.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8065999999999997e-05 2024-03-26T03:57:49.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.289206+0000", 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.060622995999999998 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.289343+0000", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000137563 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.291829+0000", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024854679999999998 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.292522+0000", 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069355600000000001 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.425259+0000", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.132736624 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.425311+0000", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2567e-05 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.425349+0000", 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7732999999999999e-05 2024-03-26T03:57:49.015 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:182156 8.3 8:ccebf960:::benchmark_data_smithi006_103169_object52225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.606322+0000", 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.354754413000002, 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.23014839100000001, 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 182156 2024-03-26T03:57:49.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606322+0000", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606322+0000", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606323+0000", 2024-03-26T03:57:49.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.82e-07 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606326+0000", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7319999999999998e-06 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606332+0000", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9660000000000001e-06 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606337+0000", 2024-03-26T03:57:49.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3009999999999997e-06 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.609628+0000", 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032910919999999998 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.609761+0000", 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000133246 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:49.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.609898+0000", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013674299999999999 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614816+0000", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049175369999999996 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834553+0000", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21973712200000001 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834595+0000", 2024-03-26T03:57:49.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2580000000000002e-05 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834647+0000", 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1545000000000002e-05 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836471+0000", 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018240450000000001 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:182157 8.3 8:c3b10124:::benchmark_data_smithi006_103169_object52226:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.606345+0000", 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.354731649999998, 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.232101421, 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 182157 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606345+0000", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606345+0000", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606345+0000", 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3000000000000002e-07 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606348+0000", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1409999999999999e-06 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606353+0000", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9169999999999997e-06 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.606357+0000", 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8099999999999997e-06 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.612247+0000", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0058895359999999999 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.612462+0000", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00021496199999999999 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.612606+0000", 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014429800000000001 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836523+0000", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22391686299999999 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836544+0000", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0571999999999999e-05 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836699+0000", 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015546700000000001 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836763+0000", 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3895999999999999e-05 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838447+0000", 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016836290000000001 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:182166 8.b e204/155 8:d4535566:::benchmark_data_smithi006_103169_object52235:head v 204'11623, mlcod=204'11621)", 2024-03-26T03:57:49.027 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.613430+0000", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.347646588000003, 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22314531700000001, 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613430+0000", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613430+0000", 2024-03-26T03:57:49.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613431+0000", 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6100000000000001e-07 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613435+0000", 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5739999999999997e-06 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613442+0000", 2024-03-26T03:57:49.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.596e-06 2024-03-26T03:57:49.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.613448+0000", 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7830000000000004e-06 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614279+0000", 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00083113299999999996 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614298+0000", 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9216e-05 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836545+0000", 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22224680599999999 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836576+0000", 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0747999999999998e-05 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:182169 8.3 8:c5b8e7dd:::benchmark_data_smithi006_103169_object52238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.614884+0000", 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.346193006999997, 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.23442723600000001, 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 182169 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614884+0000", 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614884+0000", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614885+0000", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.652e-06 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614891+0000", 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3149999999999999e-06 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614902+0000", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1161999999999999e-05 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.614909+0000", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3479999999999998e-06 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838485+0000", 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.223575406 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838616+0000", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013117200000000001 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838761+0000", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014467899999999999 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.843428+0000", 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046677710000000002 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.847418+0000", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039895180000000001 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.847814+0000", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000396043 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.847957+0000", 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000143187 2024-03-26T03:57:49.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.849311+0000", 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001353983 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:182171 8.1 8:863d27f6:::benchmark_data_smithi006_103169_object52240:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.615974+0000", 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.345102453999999, 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21798314099999999, 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.039 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 182171 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.615974+0000", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.615974+0000", 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.615975+0000", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7700000000000004e-07 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.615992+0000", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6517000000000001e-05 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616010+0000", 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8201000000000001e-05 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616017+0000", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5009999999999998e-06 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616081+0000", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3490000000000004e-05 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616204+0000", 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000123428 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616311+0000", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010644100000000001 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618124+0000", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001813032 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618568+0000", 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00044444800000000001 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833855+0000", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21528642000000001 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833909+0000", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4638000000000001e-05 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833958+0000", 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8248000000000003e-05 2024-03-26T03:57:49.044 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:182172 8.a 8:5f4fb034:::benchmark_data_smithi006_103169_object52241:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.616053+0000", 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.345023945000001, 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22267545, 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 182172 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616053+0000", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616053+0000", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616054+0000", 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0699999999999996e-07 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616057+0000", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8770000000000003e-06 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616065+0000", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.43e-06 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.616071+0000", 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5719999999999999e-06 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617960+0000", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018886109999999999 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618106+0000", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014592399999999999 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618407+0000", 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000300904 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836433+0000", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21802603700000001 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836487+0000", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4413000000000002e-05 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836502+0000", 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4167e-05 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836544+0000", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2781999999999998e-05 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838728+0000", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021840259999999999 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:182156 8.3 e204/181 ondisk, result = 0)", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.617660+0000", 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.343417260999999, 2024-03-26T03:57:49.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21690300600000001, 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617660+0000", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617660+0000", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617660+0000", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7100000000000001e-07 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617664+0000", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0640000000000004e-06 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617674+0000", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2620000000000004e-06 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617680+0000", 2024-03-26T03:57:49.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4409999999999998e-06 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834534+0000", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21685447599999999 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834544+0000", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6299999999999993e-06 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.834563+0000", 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8462e-05 2024-03-26T03:57:49.053 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:182157 8.3 e204/181 ondisk, result = 0)", 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.617841+0000", 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.343236046000001, 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21873689499999999, 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617841+0000", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617841+0000", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617841+0000", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8599999999999998e-07 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617845+0000", 2024-03-26T03:57:49.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.873e-06 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617854+0000", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5529999999999993e-06 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.617860+0000", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0469999999999999e-06 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836506+0000", 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.218646485 2024-03-26T03:57:49.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836515+0000", 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0469999999999999e-06 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836578+0000", 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2403999999999994e-05 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:182157 8.3 e204/181 ondisk, result = 0)", 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.618497+0000", 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.342579952000001, 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21995736199999999, 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618497+0000", 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618497+0000", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618498+0000", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0100000000000004e-07 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618502+0000", 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2230000000000001e-06 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618511+0000", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1519999999999998e-06 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.618518+0000", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1269999999999997e-06 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836684+0000", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21816608800000001 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836693+0000", 2024-03-26T03:57:49.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5329999999999994e-06 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838454+0000", 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017614379999999999 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:182172 8.a e204/181 ondisk, result = 0)", 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.620938+0000", 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.340138383999999, 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21550472300000001, 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.061 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.620938+0000", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.620938+0000", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.620939+0000", 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.51e-07 2024-03-26T03:57:49.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.620942+0000", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.163e-06 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833565+0000", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.212622691 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833582+0000", 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6781000000000001e-05 2024-03-26T03:57:49.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836416+0000", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028347649999999999 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836426+0000", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3449999999999995e-06 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836443+0000", 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7427000000000001e-05 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:182172 8.a e204/181 ondisk, result = 0)", 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:15.622394+0000", 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 33.338682534999997, 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21634303299999999, 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.622394+0000", 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.622394+0000", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.622395+0000", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1199999999999998e-07 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.622398+0000", 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.055e-06 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833585+0000", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.21118683499999999 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.833597+0000", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2317000000000001e-05 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836472+0000", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028753429999999998 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:57:49.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.836480+0000", 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8879999999999997e-06 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:15.838737+0000", 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022571829999999998 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:57:49.068 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:57:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:49.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:49.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:49.196 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:49.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:49.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:49.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:49.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:49.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:49.590 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:49.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:49.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:49.719 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:49.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:49.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:49 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T03:57:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:49 smithi086 ceph-mon[32965]: osdmap e214: 8 total, 8 up, 7 in 2024-03-26T03:57:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 10506/143784 objects degraded (7.307%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:49.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:49 smithi086 ceph-mon[32965]: osdmap e215: 8 total, 8 up, 7 in 2024-03-26T03:57:49.846 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:49.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:49.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T03:57:49.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[37501]: osdmap e214: 8 total, 8 up, 7 in 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 10506/143784 objects degraded (7.307%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[37501]: osdmap e215: 8 total, 8 up, 7 in 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[29446]: osdmap e214: 8 total, 8 up, 7 in 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 10506/143784 objects degraded (7.307%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:49 smithi006 ceph-mon[29446]: osdmap e215: 8 total, 8 up, 7 in 2024-03-26T03:57:49.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:49.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:49.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:49.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:50.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:50.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:50.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:50.224 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:50.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:50.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:50.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:50.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:50.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:50.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:50.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:50.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:50.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:50.745 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:50 smithi086 ceph-mon[32965]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13278 2024-03-26T03:57:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:50 smithi086 ceph-mon[32965]: pgmap v393: 148 pgs: 1 activating+degraded, 10 activating, 4 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 128 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 10538/142053 objects degraded (7.418%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T03:57:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:50 smithi086 ceph-mon[32965]: osdmap e216: 8 total, 8 up, 7 in 2024-03-26T03:57:50.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:50.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:50.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:50.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:50.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[37501]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13278 2024-03-26T03:57:50.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[37501]: pgmap v393: 148 pgs: 1 activating+degraded, 10 activating, 4 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 128 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 10538/142053 objects degraded (7.418%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T03:57:50.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[37501]: osdmap e216: 8 total, 8 up, 7 in 2024-03-26T03:57:50.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[29446]: 8.8 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13278 2024-03-26T03:57:50.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[29446]: pgmap v393: 148 pgs: 1 activating+degraded, 10 activating, 4 peering, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 128 active+clean; 2.9 GiB data, 9.9 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.06k op/s; 10538/142053 objects degraded (7.418%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T03:57:50.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:50 smithi006 ceph-mon[29446]: osdmap e216: 8 total, 8 up, 7 in 2024-03-26T03:57:50.940 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:57:50.977 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-26T03:57:50.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:51.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:51.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:51.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:51.134 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:51.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:51.265 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:51.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:51.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:51.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:51.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:51.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:51.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 213'13376 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13720 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 213'13633 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 213'13759 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13478 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13402 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[37501]: osdmap e217: 8 total, 8 up, 7 in 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 213'13376 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13720 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 213'13633 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 213'13759 2024-03-26T03:57:51.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13478 2024-03-26T03:57:51.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13402 2024-03-26T03:57:51.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:51 smithi006 ceph-mon[29446]: osdmap e217: 8 total, 8 up, 7 in 2024-03-26T03:57:51.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:51.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:51.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:51.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.c starting backfill to osd.3 from (0'0,0'0] MAX to 213'13376 2024-03-26T03:57:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.9 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13720 2024-03-26T03:57:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.f starting backfill to osd.3 from (0'0,0'0] MAX to 213'13633 2024-03-26T03:57:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.b starting backfill to osd.3 from (0'0,0'0] MAX to 213'13759 2024-03-26T03:57:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.0 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13478 2024-03-26T03:57:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: 8.7 starting backfill to osd.3 from (0'0,0'0] MAX to 213'13402 2024-03-26T03:57:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:51 smithi086 ceph-mon[32965]: osdmap e217: 8 total, 8 up, 7 in 2024-03-26T03:57:51.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:51.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:51.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:52.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:52.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:52.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:52.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:52.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:52.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:52.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:52.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:52.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:52.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:52.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:52.672 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:52.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:52.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:52.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:52.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:52 smithi086 ceph-mon[32965]: 8.8 continuing backfill to osd.3 from (197'9277,213'13278] MIN to 213'13278 2024-03-26T03:57:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:52 smithi086 ceph-mon[32965]: pgmap v396: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 5 remapped+peering, 124 active+clean, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 5622/133731 objects degraded (4.204%); 2674/133731 objects misplaced (2.000%); 26 MiB/s, 432 objects/s recovering 2024-03-26T03:57:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:52 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:52 smithi086 ceph-mon[32965]: osdmap e218: 8 total, 8 up, 7 in 2024-03-26T03:57:52.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[29446]: 8.8 continuing backfill to osd.3 from (197'9277,213'13278] MIN to 213'13278 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[29446]: pgmap v396: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 5 remapped+peering, 124 active+clean, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 5622/133731 objects degraded (4.204%); 2674/133731 objects misplaced (2.000%); 26 MiB/s, 432 objects/s recovering 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[29446]: osdmap e218: 8 total, 8 up, 7 in 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[37501]: 8.8 continuing backfill to osd.3 from (197'9277,213'13278] MIN to 213'13278 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[37501]: pgmap v396: 148 pgs: 1 active+undersized+remapped+backfill_toofull, 5 remapped+peering, 124 active+clean, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 5622/133731 objects degraded (4.204%); 2674/133731 objects misplaced (2.000%); 26 MiB/s, 432 objects/s recovering 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:52.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:52 smithi006 ceph-mon[37501]: osdmap e218: 8 total, 8 up, 7 in 2024-03-26T03:57:52.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:52.929 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:52.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:52.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:53.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:53.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:53.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:53.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:53.186 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:53.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:53.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:53.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:53.443 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:53.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:53.545 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:53.574 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:53.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:53.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:53.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:53.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:53.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:53.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:53.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:53 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:57:53.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:53 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:57:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:53 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:57:53.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:53.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:53.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:53.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:54.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:54.083 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:54.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:54.185 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.217 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:54.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:54.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:54.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:54.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:54.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:54.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:54.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:54.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:54.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:54.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[37501]: pgmap v398: 148 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 124 active+clean, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 5148/132315 objects degraded (3.891%); 5926/132315 objects misplaced (4.479%); 25 MiB/s, 418 objects/s recovering 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 5148/132315 objects degraded (3.891%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:57:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[29446]: pgmap v398: 148 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 124 active+clean, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 5148/132315 objects degraded (3.891%); 5926/132315 objects misplaced (4.479%); 25 MiB/s, 418 objects/s recovering 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:54 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 5148/132315 objects degraded (3.891%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:54.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:54.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:54.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:54.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:54.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:57:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:57:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:54 smithi086 ceph-mon[32965]: pgmap v398: 148 pgs: 1 activating+degraded+remapped, 5 remapped+peering, 124 active+clean, 1 active+remapped+backfill_wait, 2 active+recovering+undersized+degraded+remapped, 4 peering, 10 activating, 1 activating+degraded; 2.7 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 0 B/s wr, 945 op/s; 5148/132315 objects degraded (3.891%); 5926/132315 objects misplaced (4.479%); 25 MiB/s, 418 objects/s recovering 2024-03-26T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:57:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:54 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 5148/132315 objects degraded (3.891%), 4 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T03:57:54.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:54.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:54.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:54.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:54.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.000 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:55.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:55.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:55.123 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:55.225 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:55.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:55.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:55.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:55.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:55.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:55.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:55.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:55.649 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:55.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:55.775 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:55.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:57:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:57:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:55 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:55.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:57:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:57:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:57:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:57:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:57:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:55 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:57:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:55.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:55.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:55.978 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:57:55.978 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:57:55.979 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_1 2024-03-26T03:57:55.979 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_1 by 10 min 8 2024-03-26T03:57:55.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:57:56.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:56.031 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:56.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:56.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:56.165 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:56.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:56.292 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:56.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:56.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:56.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.427 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:56.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:56.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:56.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:56.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T03:57:56.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:56.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:56.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:56.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:56.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:56 smithi086 ceph-mon[32965]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 59 MiB/s, 956 objects/s recovering 2024-03-26T03:57:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:56 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:56 smithi006 ceph-mon[29446]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 59 MiB/s, 956 objects/s recovering 2024-03-26T03:57:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:56 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:56 smithi006 ceph-mon[37501]: pgmap v399: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 609 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 59 MiB/s, 956 objects/s recovering 2024-03-26T03:57:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:56 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:57:56.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:56.952 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:56.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:56.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:57.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:57.080 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:57.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:57.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:57.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:57.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:57.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:57.473 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:57.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:57.600 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:57.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:57.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:57.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:57.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:57:57.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:57.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:57.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:57.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:57.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:58.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:58.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:58.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:58.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:58.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:58.377 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:58.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:57:58.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:57:58.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:57:58.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:57:58.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:58.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:58.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:58 smithi086 ceph-mon[32965]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 125 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 36 MiB/s, 575 objects/s recovering 2024-03-26T03:57:58.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:58.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:58 smithi006 ceph-mon[37501]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 125 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 36 MiB/s, 575 objects/s recovering 2024-03-26T03:57:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:58 smithi006 ceph-mon[29446]: pgmap v400: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 activating+degraded+remapped, 1 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.7 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 125 op/s; 5385/131097 objects degraded (4.108%); 15307/131097 objects misplaced (11.676%); 36 MiB/s, 575 objects/s recovering 2024-03-26T03:57:58.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:58.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:58.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:59.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:59.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:59.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:59.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:59.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:57:59.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:57:59.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:57:59.407 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:59.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:57:59.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:57:59.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:57:59.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:57:59.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:57:59 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 5385/131097 objects degraded (4.108%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:57:59.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:57:59 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 5385/131097 objects degraded (4.108%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:57:59.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:57:59.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:57:59.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:57:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:57:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:57:59 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 5385/131097 objects degraded (4.108%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:57:59.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:57:59.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:57:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:57:59.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:00.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:00.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:00.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:00.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:00.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:00.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:00.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:00.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:00.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:00.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:00.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:00.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:00.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:00.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:00.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:00.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:00.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:00 smithi086 ceph-mon[32965]: pgmap v401: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 173 op/s; 4107/129570 objects degraded (3.170%); 13494/129570 objects misplaced (10.414%); 32 MiB/s, 515 objects/s recovering 2024-03-26T03:58:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:00 smithi006 ceph-mon[37501]: pgmap v401: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 173 op/s; 4107/129570 objects degraded (3.170%); 13494/129570 objects misplaced (10.414%); 32 MiB/s, 515 objects/s recovering 2024-03-26T03:58:00.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:00 smithi006 ceph-mon[29446]: pgmap v401: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 3 active+remapped+backfill_wait; 2.6 GiB data, 9.8 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 173 op/s; 4107/129570 objects degraded (3.170%); 13494/129570 objects misplaced (10.414%); 32 MiB/s, 515 objects/s recovering 2024-03-26T03:58:00.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:00.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:00.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:00.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:01.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:01.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:01.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:01.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:01.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:01.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:01.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:01.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:01.475 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:01.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:01.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:01.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:01.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.740 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:01.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:01.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:01.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:01.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[37501]: osdmap e219: 8 total, 8 up, 7 in 2024-03-26T03:58:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:01 smithi006 ceph-mon[29446]: osdmap e219: 8 total, 8 up, 7 in 2024-03-26T03:58:01.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:02.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:02.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:01 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:01 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:02.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:01 smithi086 ceph-mon[32965]: osdmap e219: 8 total, 8 up, 7 in 2024-03-26T03:58:02.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:02.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:02.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:02.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:02.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:02.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:02.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:02.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:02.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[37501]: pgmap v403: 148 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 268 op/s; 3083/126498 objects degraded (2.437%); 11475/126498 objects misplaced (9.071%); 45 MiB/s, 719 objects/s recovering 2024-03-26T03:58:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[37501]: osdmap e220: 8 total, 8 up, 7 in 2024-03-26T03:58:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[29446]: pgmap v403: 148 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 268 op/s; 3083/126498 objects degraded (2.437%); 11475/126498 objects misplaced (9.071%); 45 MiB/s, 719 objects/s recovering 2024-03-26T03:58:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:02.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:02 smithi006 ceph-mon[29446]: osdmap e220: 8 total, 8 up, 7 in 2024-03-26T03:58:03.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:02 smithi086 ceph-mon[32965]: pgmap v403: 148 pgs: 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 2 active+remapped+backfill_wait; 2.6 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 268 op/s; 3083/126498 objects degraded (2.437%); 11475/126498 objects misplaced (9.071%); 45 MiB/s, 719 objects/s recovering 2024-03-26T03:58:03.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:02 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:02 smithi086 ceph-mon[32965]: osdmap e220: 8 total, 8 up, 7 in 2024-03-26T03:58:03.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:03.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:03.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:03.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:03.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:03.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:03.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:04.017 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:03 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:03 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:03 smithi086 ceph-mon[32965]: osdmap e221: 8 total, 8 up, 7 in 2024-03-26T03:58:04.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:04.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:04.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:04.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[37501]: osdmap e221: 8 total, 8 up, 7 in 2024-03-26T03:58:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:03 smithi006 ceph-mon[29446]: osdmap e221: 8 total, 8 up, 7 in 2024-03-26T03:58:04.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:04.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:04.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:04.270 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:04.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:04.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:04.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:04.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:04.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:04.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:04.719 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:04.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:04.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:04.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:04.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:04.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:04.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:04.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:04.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:05.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:05.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:04 smithi086 ceph-mon[32965]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 324 op/s; 2975/125274 objects degraded (2.375%); 10878/125274 objects misplaced (8.683%); 31 MiB/s, 491 objects/s recovering 2024-03-26T03:58:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:04 smithi086 ceph-mon[32965]: osdmap e222: 8 total, 8 up, 7 in 2024-03-26T03:58:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:04 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 2975/125274 objects degraded (2.375%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:58:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:04 smithi086 ceph-mon[32965]: osdmap e223: 8 total, 8 up, 7 in 2024-03-26T03:58:05.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:05.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[29446]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 324 op/s; 2975/125274 objects degraded (2.375%); 10878/125274 objects misplaced (8.683%); 31 MiB/s, 491 objects/s recovering 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[29446]: osdmap e222: 8 total, 8 up, 7 in 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 2975/125274 objects degraded (2.375%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[29446]: osdmap e223: 8 total, 8 up, 7 in 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[37501]: pgmap v406: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+remapped, 2 active+remapped+backfill_toofull, 2 active+recovering+degraded+remapped, 140 active+clean, 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_wait; 2.5 GiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 324 op/s; 2975/125274 objects degraded (2.375%); 10878/125274 objects misplaced (8.683%); 31 MiB/s, 491 objects/s recovering 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[37501]: osdmap e222: 8 total, 8 up, 7 in 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 2975/125274 objects degraded (2.375%), 3 pgs degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T03:58:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:04 smithi006 ceph-mon[37501]: osdmap e223: 8 total, 8 up, 7 in 2024-03-26T03:58:05.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:05.241 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:05.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:05.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:05.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:05.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:05.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:05.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:05.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:05.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:05.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:05.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:05.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:05.786 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:05.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:05.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:05.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:06.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:06.048 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:06.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:06.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:06.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:06.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:06.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:06.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:06.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:06.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:06.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:06.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:06.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:06.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:06.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:06.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: pgmap v409: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 376 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 21 MiB/s, 547 objects/s recovering 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: osdmap e224: 8 total, 8 up, 7 in 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1750754865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: pgmap v409: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 376 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 21 MiB/s, 547 objects/s recovering 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:06.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: osdmap e224: 8 total, 8 up, 7 in 2024-03-26T03:58:06.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1750754865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:06.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:06 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:06.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:06.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:06.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:07.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:07.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:07.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: pgmap v409: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 376 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 21 MiB/s, 547 objects/s recovering 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: osdmap e224: 8 total, 8 up, 7 in 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1750754865' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:06 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:58:07.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:07.208 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:07.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.335 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:07.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:07.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.463 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:07.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:07.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:07.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:07.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:07.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:07.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:07.850 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:07.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:07.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:07.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:07.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:08.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:07 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:07 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:58:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:07 smithi086 ceph-mon[32965]: osdmap e225: 8 total, 8 up, 7 in 2024-03-26T03:58:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.105 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:08.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:08.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:58:08.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[37501]: osdmap e225: 8 total, 8 up, 7 in 2024-03-26T03:58:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:58:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:07 smithi006 ceph-mon[29446]: osdmap e225: 8 total, 8 up, 7 in 2024-03-26T03:58:08.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:08.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:08.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:08.362 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:08.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:08.497 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:08.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:08.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:08.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:08.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:08.930 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:08.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:08.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:09.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:09.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:09.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:09.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:08 smithi086 ceph-mon[32965]: pgmap v412: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 274 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 13 MiB/s, 421 objects/s recovering 2024-03-26T03:58:09.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:08 smithi006 ceph-mon[37501]: pgmap v412: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 274 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 13 MiB/s, 421 objects/s recovering 2024-03-26T03:58:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:08 smithi006 ceph-mon[29446]: pgmap v412: 148 pgs: 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 274 op/s; 1408/121995 objects degraded (1.154%); 9890/121995 objects misplaced (8.107%); 13 MiB/s, 421 objects/s recovering 2024-03-26T03:58:09.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:09.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:09.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:09.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:09.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:09.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:09.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:09.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:09.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:09.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:09.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:09.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:09.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:09.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:09.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:09.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:10.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:10.082 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:10.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:10.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:10.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:10.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:10.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1408/121995 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:10 smithi086 ceph-mon[32965]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 265 op/s; 1292/120735 objects degraded (1.070%); 9303/120735 objects misplaced (7.705%); 15 MiB/s, 384 objects/s recovering 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1408/121995 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[37501]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 265 op/s; 1292/120735 objects degraded (1.070%); 9303/120735 objects misplaced (7.705%); 15 MiB/s, 384 objects/s recovering 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T03:58:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1408/121995 objects degraded (1.154%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:10 smithi006 ceph-mon[29446]: pgmap v413: 148 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_wait, 1 active+recovering+remapped, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 265 op/s; 1292/120735 objects degraded (1.070%); 9303/120735 objects misplaced (7.705%); 15 MiB/s, 384 objects/s recovering 2024-03-26T03:58:10.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:10.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:10.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:10.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:10.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:10.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:10.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:10.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:10.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:10.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:10.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:10.984 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:58:11.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:11.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:11.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:11.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:11.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:11.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:11.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:11.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:11.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:11.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:11.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:11 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T03:58:11.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:11 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T03:58:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:11 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T03:58:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:11 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T03:58:11.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:11.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:11.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:11 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T03:58:11.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:11 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T03:58:11.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:11.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:11.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:11.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:11.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.003 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:12.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:12.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:12.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:12.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:12.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:12.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:12.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:12.524 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:12.600 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[29446]: pgmap v414: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 403 op/s; 1292/116760 objects degraded (1.107%); 11420/116760 objects misplaced (9.781%); 12 MiB/s, 540 objects/s recovering 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T03:58:12.601 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:12 smithi006 ceph-mon[37501]: pgmap v414: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 403 op/s; 1292/116760 objects degraded (1.107%); 11420/116760 objects misplaced (9.781%); 12 MiB/s, 540 objects/s recovering 2024-03-26T03:58:12.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:12.652 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:12.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:12.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:12 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T03:58:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:12 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T03:58:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:12 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T03:58:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:12 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T03:58:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:12 smithi086 ceph-mon[32965]: pgmap v414: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.4 GiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 403 op/s; 1292/116760 objects degraded (1.107%); 11420/116760 objects misplaced (9.781%); 12 MiB/s, 540 objects/s recovering 2024-03-26T03:58:12.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:12.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:12.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:13.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:13.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:13.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:13.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:13.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:13.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:13.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:13.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:13.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:13.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:13.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:13.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:13.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:13.687 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:13.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:13.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:13.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:13.944 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:13.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:14.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:14.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:14.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:14.201 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:14.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:14.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:14.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:14.470 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:14.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:14.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: osdmap e226: 8 total, 8 up, 7 in 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: pgmap v416: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 283 op/s; 1167/115446 objects degraded (1.011%); 11211/115446 objects misplaced (9.711%); 5.5 MiB/s, 293 objects/s recovering 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1292/116760 objects degraded (1.107%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[37501]: osdmap e227: 8 total, 8 up, 7 in 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T03:58:14.604 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: osdmap e226: 8 total, 8 up, 7 in 2024-03-26T03:58:14.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: pgmap v416: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 283 op/s; 1167/115446 objects degraded (1.011%); 11211/115446 objects misplaced (9.711%); 5.5 MiB/s, 293 objects/s recovering 2024-03-26T03:58:14.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1292/116760 objects degraded (1.107%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:14.605 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-mon[29446]: osdmap e227: 8 total, 8 up, 7 in 2024-03-26T03:58:14.605 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:14.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: osdmap e226: 8 total, 8 up, 7 in 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: pgmap v416: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 142 active+clean; 2.3 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 663 B/s rd, 0 B/s wr, 283 op/s; 1167/115446 objects degraded (1.011%); 11211/115446 objects misplaced (9.711%); 5.5 MiB/s, 293 objects/s recovering 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1292/116760 objects degraded (1.107%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:14.717 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-mon[32965]: osdmap e227: 8 total, 8 up, 7 in 2024-03-26T03:58:14.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:14.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:14.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:14.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:14.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:15.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:15.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:15.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:15.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:15.246 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:15.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:15.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:15.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:15.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:58:15.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:15.504 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:15.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:15.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:15.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:15.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:15.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:15.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:15.642 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:58:15.668 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:58:15.694 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:58:15.724 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:58:15.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:15.751 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:58:15.757 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:15.774 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:58:15.796 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:58:15.821 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:58:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:58:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:58:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:58:15.823 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:58:15.824 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:58:15.824 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:58:15.824 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:58:15.824 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:58:15.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:15.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:15.934 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:15.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:16.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:16.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:16.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:16.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:16.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:16.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:16.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:16.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:16.540 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:16.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: pgmap v418: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 450 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 23 MiB/s, 571 objects/s recovering 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1560362069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T03:58:16.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T03:58:16.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T03:58:16.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: pgmap v418: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 450 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 23 MiB/s, 571 objects/s recovering 2024-03-26T03:58:16.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:16 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1560362069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:16.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:16.675 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:16.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:16.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:16.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T03:58:16.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T03:58:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T03:58:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T03:58:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: pgmap v418: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 450 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 23 MiB/s, 571 objects/s recovering 2024-03-26T03:58:16.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:16 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1560362069' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:16.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:16.935 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:16.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:17.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:17.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:17.108 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:58:17.108 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:58:17.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:17.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:17.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:17.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:17.326 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:17.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:17.460 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:17.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:17.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:17.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:17.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T03:58:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:17 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T03:58:17.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:17.719 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:17.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:17.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:17 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T03:58:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:17 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T03:58:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:17 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T03:58:17.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:17 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T03:58:17.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:17.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:17.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:18.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:18.105 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:18.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:18.207 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:18.240 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:18.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:18.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:18.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:18.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:18.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:18.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:18.621 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:18.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:18 smithi006 ceph-mon[37501]: pgmap v419: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 397 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 19 MiB/s, 506 objects/s recovering 2024-03-26T03:58:18.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:18 smithi006 ceph-mon[29446]: pgmap v419: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 397 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 19 MiB/s, 506 objects/s recovering 2024-03-26T03:58:18.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:18.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:18.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:18 smithi086 ceph-mon[32965]: pgmap v419: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 397 op/s; 1167/111210 objects degraded (1.049%); 8525/111210 objects misplaced (7.666%); 19 MiB/s, 506 objects/s recovering 2024-03-26T03:58:18.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:18.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:18.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:18.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.007 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:19.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:19.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:19.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:19.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:19.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:19.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:19.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:19.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:19.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.522 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:19.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:19.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[29446]: osdmap e228: 8 total, 8 up, 7 in 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1167/111210 objects degraded (1.049%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[29446]: osdmap e229: 8 total, 8 up, 7 in 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[37501]: osdmap e228: 8 total, 8 up, 7 in 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1167/111210 objects degraded (1.049%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:19 smithi006 ceph-mon[37501]: osdmap e229: 8 total, 8 up, 7 in 2024-03-26T03:58:19.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:19.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:19.801 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:19 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:58:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:19 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:58:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:19 smithi086 ceph-mon[32965]: osdmap e228: 8 total, 8 up, 7 in 2024-03-26T03:58:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:19 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1167/111210 objects degraded (1.049%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:19 smithi086 ceph-mon[32965]: osdmap e229: 8 total, 8 up, 7 in 2024-03-26T03:58:19.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:19.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:19.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:20.028 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:20.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:20.179 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:20.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:20.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:20.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:20.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:20.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:20.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:20.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:20.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:20.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:20.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:20 smithi006 ceph-mon[37501]: pgmap v422: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 320 op/s; 1036/109710 objects degraded (0.944%); 8006/109710 objects misplaced (7.297%); 29 MiB/s, 459 objects/s recovering 2024-03-26T03:58:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:20 smithi006 ceph-mon[29446]: pgmap v422: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 320 op/s; 1036/109710 objects degraded (0.944%); 8006/109710 objects misplaced (7.297%); 29 MiB/s, 459 objects/s recovering 2024-03-26T03:58:20.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:20.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:20.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:20.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:20.821 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:20.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:20 smithi086 ceph-mon[32965]: pgmap v422: 148 pgs: 2 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 143 active+clean; 2.2 GiB data, 8.9 GiB used, 617 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 320 op/s; 1036/109710 objects degraded (0.944%); 8006/109710 objects misplaced (7.297%); 29 MiB/s, 459 objects/s recovering 2024-03-26T03:58:20.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:20.948 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:20.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:21.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:21.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:21.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:21.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:21.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:21.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.326 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:21.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:21.460 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:21.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:21.590 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:21.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:21.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:21.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:21 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T03:58:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:21 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T03:58:21.859 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:21.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:21 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T03:58:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:21 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T03:58:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:21 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T03:58:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:21 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T03:58:21.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:21.987 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:21.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:22.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:22.115 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:22.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:22.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:22.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:22.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:22.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:22.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:22.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:22.518 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:22.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:22.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:22.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:22.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:22 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T03:58:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:22 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T03:58:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:22 smithi086 ceph-mon[32965]: pgmap v423: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 487 op/s; 1036/105162 objects degraded (0.985%); 6585/105162 objects misplaced (6.262%); 35 MiB/s, 553 objects/s recovering 2024-03-26T03:58:22.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:22.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:22.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T03:58:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T03:58:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[37501]: pgmap v423: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 487 op/s; 1036/105162 objects degraded (0.985%); 6585/105162 objects misplaced (6.262%); 35 MiB/s, 553 objects/s recovering 2024-03-26T03:58:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T03:58:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T03:58:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:22 smithi006 ceph-mon[29446]: pgmap v423: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 726 B/s rd, 0 B/s wr, 487 op/s; 1036/105162 objects degraded (0.985%); 6585/105162 objects misplaced (6.262%); 35 MiB/s, 553 objects/s recovering 2024-03-26T03:58:22.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:23.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:23.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:23.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:23.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:23.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:23.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:23.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:23.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:23.639 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:23.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:23.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:23.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:23.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:23.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:23.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:23.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:23.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:24.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:24.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:24.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:24.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:24.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:24.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:24.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.425 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:24.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:24.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:24.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:24.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:24.587 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:24.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T03:58:24.588 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[29446]: pgmap v424: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 321 op/s; 887/103512 objects degraded (0.857%); 5818/103512 objects misplaced (5.621%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 887/103512 objects degraded (0.857%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[37501]: pgmap v424: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 321 op/s; 887/103512 objects degraded (0.857%); 5818/103512 objects misplaced (5.621%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:24.589 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 887/103512 objects degraded (0.857%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:24.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T03:58:24.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:24.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:24.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:24.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T03:58:24.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T03:58:24.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-mon[32965]: pgmap v424: 148 pgs: 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.1 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 321 op/s; 887/103512 objects degraded (0.857%); 5818/103512 objects misplaced (5.621%); 18 MiB/s, 289 objects/s recovering 2024-03-26T03:58:24.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:24.727 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 887/103512 objects degraded (0.857%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:24.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:24.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:24.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:24.825 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:24.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:24.956 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:24.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:24.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:25.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:25.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:25.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:25.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:25.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:25.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:25.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:25.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:25.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:25.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:25.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:25.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:25.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:25.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:25.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:25.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:25.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:25.841 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:58:25.844 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:58:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:25.872 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:25.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:25.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:26.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:26.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:26.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:26.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:26.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:26.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:26.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:26.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:26.397 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:26.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:26.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:26.606 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T03:58:26.606 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[37501]: pgmap v425: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 566 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 27 MiB/s, 428 objects/s recovering 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[37501]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[29446]: pgmap v425: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 566 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 27 MiB/s, 428 objects/s recovering 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:26.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:26 smithi006 ceph-mon[29446]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:58:26.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:26.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:26.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:26.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:26.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:58:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:26 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T03:58:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:26 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T03:58:26.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:26 smithi086 ceph-mon[32965]: pgmap v425: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 566 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 27 MiB/s, 428 objects/s recovering 2024-03-26T03:58:26.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:26 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:26.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:26 smithi086 ceph-mon[32965]: from='client.25124 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:58:26.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:26.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:26.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:26.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:27.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:27.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:27.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:27.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:27.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:27.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:27.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:27.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:27.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:27.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:27.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:27.491 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-03-26T03:58:25.378752+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2118588864,"num_objects":32549,"num_object_clones":0,"num_object_copies":97647,"num_objects_missing_on_primary":2537,"num_objects_missing":2537,"num_objects_degraded":887,"num_objects_misplaced":4436,"num_objects_unfound":0,"num_objects_dirty":32549,"num_whiteouts":0,"num_read":1788,"num_read_kb":1602,"num_write":228346,"num_write_kb":8317854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36900,"num_bytes_recovered":2243437814,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72786,"ondisk_log_size":72786,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":450,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":8776484,"kb_used_data":6617772,"kb_used_omap":13,"kb_used_meta":2158578,"kb_avail":647439580,"statfs":{"total":671965249536,"available":662978129920,"internally_reserved":0,"allocated":6776598528,"data_stored":6773226694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14011,"internal_metadata":2210384197},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-296288256,"num_objects":-4521,"num_object_clones":0,"num_object_copies":-13563,"num_objects_missing_on_primary":-280,"num_objects_missing":-280,"num_objects_degraded":-280,"num_objects_misplaced":-4089,"num_objects_unfound":0,"num_objects_dirty":-4521,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4521,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3426,"num_bytes_recovered":224526336,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001933"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335803+0000","last_change":"2024-03-26T03:55:59.300239+0000","last_active":"2024-03-26T03:58:19.335803+0000","last_peered":"2024-03-26T03:58:19.335803+0000","last_clean":"2024-03-26T03:58:19.335803+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T03:58:19.335803+0000","last_undegraded":"2024-03-26T03:58:19.335803+0000","last_fullsized":"2024-03-26T03:58:19.335803+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":356,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338207+0000","last_change":"2024-03-26T03:55:59.291583+0000","last_active":"2024-03-26T03:58:19.338207+0000","last_peered":"2024-03-26T03:58:19.338207+0000","last_clean":"2024-03-26T03:58:19.338207+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T03:58:19.338207+0000","last_undegraded":"2024-03-26T03:58:19.338207+0000","last_fullsized":"2024-03-26T03:58:19.338207+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:43.862434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:40.094601+0000","last_clean_scrub_stamp":"2024-03-26T03:52:43.862434+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":374,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404691+0000","last_change":"2024-03-26T03:55:59.307852+0000","last_active":"2024-03-26T03:58:19.404691+0000","last_peered":"2024-03-26T03:58:19.404691+0000","last_clean":"2024-03-26T03:58:19.404691+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T03:58:19.404691+0000","last_undegraded":"2024-03-26T03:58:19.404691+0000","last_fullsized":"2024-03-26T03:58:19.404691+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'15","reported_seq":442,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335593+0000","last_change":"2024-03-26T03:57:49.470002+0000","last_active":"2024-03-26T03:58:19.335593+0000","last_peered":"2024-03-26T03:58:19.335593+0000","last_clean":"2024-03-26T03:58:19.335593+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T03:58:19.335593+0000","last_undegraded":"2024-03-26T03:58:19.335593+0000","last_fullsized":"2024-03-26T03:58:19.335593+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402361+0000","last_change":"2024-03-26T03:57:49.377568+0000","last_active":"2024-03-26T03:58:19.402361+0000","last_peered":"2024-03-26T03:58:19.402361+0000","last_clean":"2024-03-26T03:58:19.402361+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T03:58:19.402361+0000","last_undegraded":"2024-03-26T03:58:19.402361+0000","last_fullsized":"2024-03-26T03:58:19.402361+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":362,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335557+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T03:58:19.335557+0000","last_peered":"2024-03-26T03:58:19.335557+0000","last_clean":"2024-03-26T03:58:19.335557+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:58:19.335557+0000","last_undegraded":"2024-03-26T03:58:19.335557+0000","last_fullsized":"2024-03-26T03:58:19.335557+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405340+0000","last_change":"2024-03-26T03:57:49.382349+0000","last_active":"2024-03-26T03:58:19.405340+0000","last_peered":"2024-03-26T03:58:19.405340+0000","last_clean":"2024-03-26T03:58:19.405340+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T03:58:19.405340+0000","last_undegraded":"2024-03-26T03:58:19.405340+0000","last_fullsized":"2024-03-26T03:58:19.405340+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'39","reported_seq":63,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337943+0000","last_change":"2024-03-26T03:57:49.505699+0000","last_active":"2024-03-26T03:58:19.337943+0000","last_peered":"2024-03-26T03:58:19.337943+0000","last_clean":"2024-03-26T03:58:19.337943+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T03:58:19.337943+0000","last_undegraded":"2024-03-26T03:58:19.337943+0000","last_fullsized":"2024-03-26T03:58:19.337943+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":61,"num_read_kb":48,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":352,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376930+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T03:58:19.376930+0000","last_peered":"2024-03-26T03:58:19.376930+0000","last_clean":"2024-03-26T03:58:19.376930+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:58:19.376930+0000","last_undegraded":"2024-03-26T03:58:19.376930+0000","last_fullsized":"2024-03-26T03:58:19.376930+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":346,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339438+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T03:58:19.339438+0000","last_peered":"2024-03-26T03:58:19.339438+0000","last_clean":"2024-03-26T03:58:19.339438+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T03:58:19.339438+0000","last_undegraded":"2024-03-26T03:58:19.339438+0000","last_fullsized":"2024-03-26T03:58:19.339438+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":296,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339644+0000","last_change":"2024-03-26T03:55:59.304086+0000","last_active":"2024-03-26T03:58:19.339644+0000","last_peered":"2024-03-26T03:58:19.339644+0000","last_clean":"2024-03-26T03:58:19.339644+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T03:58:19.339644+0000","last_undegraded":"2024-03-26T03:58:19.339644+0000","last_fullsized":"2024-03-26T03:58:19.339644+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'22","reported_seq":441,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336787+0000","last_change":"2024-03-26T03:57:49.407755+0000","last_active":"2024-03-26T03:58:19.336787+0000","last_peered":"2024-03-26T03:58:19.336787+0000","last_clean":"2024-03-26T03:58:19.336787+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T03:58:19.336787+0000","last_undegraded":"2024-03-26T03:58:19.336787+0000","last_fullsized":"2024-03-26T03:58:19.336787+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":321,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.345795+0000","last_change":"2024-03-26T03:57:49.381007+0000","last_active":"2024-03-26T03:58:19.345795+0000","last_peered":"2024-03-26T03:58:19.345795+0000","last_clean":"2024-03-26T03:58:19.345795+0000","last_became_active":"2024-03-26T03:57:49.380178+0000","last_became_peered":"2024-03-26T03:57:49.380178+0000","last_unstale":"2024-03-26T03:58:19.345795+0000","last_undegraded":"2024-03-26T03:58:19.345795+0000","last_fullsized":"2024-03-26T03:58:19.345795+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":356,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446094+0000","last_change":"2024-03-26T03:54:11.544712+0000","last_active":"2024-03-26T03:58:19.446094+0000","last_peered":"2024-03-26T03:58:19.446094+0000","last_clean":"2024-03-26T03:58:19.446094+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:58:19.446094+0000","last_undegraded":"2024-03-26T03:58:19.446094+0000","last_fullsized":"2024-03-26T03:58:19.446094+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.544637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.544637+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":321,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339285+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T03:58:19.339285+0000","last_peered":"2024-03-26T03:58:19.339285+0000","last_clean":"2024-03-26T03:58:19.339285+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T03:58:19.339285+0000","last_undegraded":"2024-03-26T03:58:19.339285+0000","last_fullsized":"2024-03-26T03:58:19.339285+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"131'14","reported_seq":413,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336001+0000","last_change":"2024-03-26T03:57:49.485971+0000","last_active":"2024-03-26T03:58:19.336001+0000","last_peered":"2024-03-26T03:58:19.336001+0000","last_clean":"2024-03-26T03:58:19.336001+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T03:58:19.336001+0000","last_undegraded":"2024-03-26T03:58:19.336001+0000","last_fullsized":"2024-03-26T03:58:19.336001+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":372,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336222+0000","last_change":"2024-03-26T03:57:49.361032+0000","last_active":"2024-03-26T03:58:19.336222+0000","last_peered":"2024-03-26T03:58:19.336222+0000","last_clean":"2024-03-26T03:58:19.336222+0000","last_became_active":"2024-03-26T03:57:49.360508+0000","last_became_peered":"2024-03-26T03:57:49.360508+0000","last_unstale":"2024-03-26T03:58:19.336222+0000","last_undegraded":"2024-03-26T03:58:19.336222+0000","last_fullsized":"2024-03-26T03:58:19.336222+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":355,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.403209+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T03:58:19.403209+0000","last_peered":"2024-03-26T03:58:19.403209+0000","last_clean":"2024-03-26T03:58:19.403209+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:58:19.403209+0000","last_undegraded":"2024-03-26T03:58:19.403209+0000","last_fullsized":"2024-03-26T03:58:19.403209+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":372,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404527+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T03:58:19.404527+0000","last_peered":"2024-03-26T03:58:19.404527+0000","last_clean":"2024-03-26T03:58:19.404527+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:58:19.404527+0000","last_undegraded":"2024-03-26T03:58:19.404527+0000","last_fullsized":"2024-03-26T03:58:19.404527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":445,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401204+0000","last_change":"2024-03-26T03:57:49.417488+0000","last_active":"2024-03-26T03:58:19.401204+0000","last_peered":"2024-03-26T03:58:19.401204+0000","last_clean":"2024-03-26T03:58:19.401204+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T03:58:19.401204+0000","last_undegraded":"2024-03-26T03:58:19.401204+0000","last_fullsized":"2024-03-26T03:58:19.401204+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:30.771573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.771573+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405109+0000","last_change":"2024-03-26T03:57:49.377389+0000","last_active":"2024-03-26T03:58:19.405109+0000","last_peered":"2024-03-26T03:58:19.405109+0000","last_clean":"2024-03-26T03:58:19.405109+0000","last_became_active":"2024-03-26T03:57:49.376975+0000","last_became_peered":"2024-03-26T03:57:49.376975+0000","last_unstale":"2024-03-26T03:58:19.405109+0000","last_undegraded":"2024-03-26T03:58:19.405109+0000","last_fullsized":"2024-03-26T03:58:19.405109+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":368,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336832+0000","last_change":"2024-03-26T03:57:49.360798+0000","last_active":"2024-03-26T03:58:19.336832+0000","last_peered":"2024-03-26T03:58:19.336832+0000","last_clean":"2024-03-26T03:58:19.336832+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T03:58:19.336832+0000","last_undegraded":"2024-03-26T03:58:19.336832+0000","last_fullsized":"2024-03-26T03:58:19.336832+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":364,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336894+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:58:19.336894+0000","last_peered":"2024-03-26T03:58:19.336894+0000","last_clean":"2024-03-26T03:58:19.336894+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:58:19.336894+0000","last_undegraded":"2024-03-26T03:58:19.336894+0000","last_fullsized":"2024-03-26T03:58:19.336894+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'18","reported_seq":53,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401392+0000","last_change":"2024-03-26T03:57:49.469098+0000","last_active":"2024-03-26T03:58:19.401392+0000","last_peered":"2024-03-26T03:58:19.401392+0000","last_clean":"2024-03-26T03:58:19.401392+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T03:58:19.401392+0000","last_undegraded":"2024-03-26T03:58:19.401392+0000","last_fullsized":"2024-03-26T03:58:19.401392+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":33,"num_read_kb":23,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":363,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498358+0000","last_change":"2024-03-26T03:55:59.300737+0000","last_active":"2024-03-26T03:58:19.498358+0000","last_peered":"2024-03-26T03:58:19.498358+0000","last_clean":"2024-03-26T03:58:19.498358+0000","last_became_active":"2024-03-26T03:55:59.299335+0000","last_became_peered":"2024-03-26T03:55:59.299335+0000","last_unstale":"2024-03-26T03:58:19.498358+0000","last_undegraded":"2024-03-26T03:58:19.498358+0000","last_fullsized":"2024-03-26T03:58:19.498358+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:54:15.688518+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.688518+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":422,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336481+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T03:58:19.336481+0000","last_peered":"2024-03-26T03:58:19.336481+0000","last_clean":"2024-03-26T03:58:19.336481+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:58:19.336481+0000","last_undegraded":"2024-03-26T03:58:19.336481+0000","last_fullsized":"2024-03-26T03:58:19.336481+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":384,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338108+0000","last_change":"2024-03-26T03:57:49.493948+0000","last_active":"2024-03-26T03:58:19.338108+0000","last_peered":"2024-03-26T03:58:19.338108+0000","last_clean":"2024-03-26T03:58:19.338108+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T03:58:19.338108+0000","last_undegraded":"2024-03-26T03:58:19.338108+0000","last_fullsized":"2024-03-26T03:58:19.338108+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:55:20.749583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.749583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'16","reported_seq":428,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402094+0000","last_change":"2024-03-26T03:57:49.470100+0000","last_active":"2024-03-26T03:58:19.402094+0000","last_peered":"2024-03-26T03:58:19.402094+0000","last_clean":"2024-03-26T03:58:19.402094+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T03:58:19.402094+0000","last_undegraded":"2024-03-26T03:58:19.402094+0000","last_fullsized":"2024-03-26T03:58:19.402094+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":352,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402891+0000","last_change":"2024-03-26T03:55:59.343699+0000","last_active":"2024-03-26T03:58:19.402891+0000","last_peered":"2024-03-26T03:58:19.402891+0000","last_clean":"2024-03-26T03:58:19.402891+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T03:58:19.402891+0000","last_undegraded":"2024-03-26T03:58:19.402891+0000","last_fullsized":"2024-03-26T03:58:19.402891+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:42.867780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:42.867780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":357,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404893+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:58:19.404893+0000","last_peered":"2024-03-26T03:58:19.404893+0000","last_clean":"2024-03-26T03:58:19.404893+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:58:19.404893+0000","last_undegraded":"2024-03-26T03:58:19.404893+0000","last_fullsized":"2024-03-26T03:58:19.404893+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402633+0000","last_change":"2024-03-26T03:57:49.399354+0000","last_active":"2024-03-26T03:58:19.402633+0000","last_peered":"2024-03-26T03:58:19.402633+0000","last_clean":"2024-03-26T03:58:19.402633+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T03:58:19.402633+0000","last_undegraded":"2024-03-26T03:58:19.402633+0000","last_fullsized":"2024-03-26T03:58:19.402633+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'19","reported_seq":351,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339830+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T03:58:19.339830+0000","last_peered":"2024-03-26T03:58:19.339830+0000","last_clean":"2024-03-26T03:58:19.339830+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T03:58:19.339830+0000","last_undegraded":"2024-03-26T03:58:19.339830+0000","last_fullsized":"2024-03-26T03:58:19.339830+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":20,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446174+0000","last_change":"2024-03-26T03:54:19.645849+0000","last_active":"2024-03-26T03:58:19.446174+0000","last_peered":"2024-03-26T03:58:19.446174+0000","last_clean":"2024-03-26T03:58:19.446174+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:58:19.446174+0000","last_undegraded":"2024-03-26T03:58:19.446174+0000","last_fullsized":"2024-03-26T03:58:19.446174+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:19.645746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.645746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":295,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335492+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:58:19.335492+0000","last_peered":"2024-03-26T03:58:19.335492+0000","last_clean":"2024-03-26T03:58:19.335492+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:58:19.335492+0000","last_undegraded":"2024-03-26T03:58:19.335492+0000","last_fullsized":"2024-03-26T03:58:19.335492+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":386,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405102+0000","last_change":"2024-03-26T03:57:49.385528+0000","last_active":"2024-03-26T03:58:19.405102+0000","last_peered":"2024-03-26T03:58:19.405102+0000","last_clean":"2024-03-26T03:58:19.405102+0000","last_became_active":"2024-03-26T03:57:49.384847+0000","last_became_peered":"2024-03-26T03:57:49.384847+0000","last_unstale":"2024-03-26T03:58:19.405102+0000","last_undegraded":"2024-03-26T03:58:19.405102+0000","last_fullsized":"2024-03-26T03:58:19.405102+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"140'6","reported_seq":45,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402525+0000","last_change":"2024-03-26T03:57:49.492568+0000","last_active":"2024-03-26T03:58:19.402525+0000","last_peered":"2024-03-26T03:58:19.402525+0000","last_clean":"2024-03-26T03:58:19.402525+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T03:58:19.402525+0000","last_undegraded":"2024-03-26T03:58:19.402525+0000","last_fullsized":"2024-03-26T03:58:19.402525+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402281+0000","last_change":"2024-03-26T03:57:49.383008+0000","last_active":"2024-03-26T03:58:19.402281+0000","last_peered":"2024-03-26T03:58:19.402281+0000","last_clean":"2024-03-26T03:58:19.402281+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T03:58:19.402281+0000","last_undegraded":"2024-03-26T03:58:19.402281+0000","last_fullsized":"2024-03-26T03:58:19.402281+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":363,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446128+0000","last_change":"2024-03-26T03:55:25.766015+0000","last_active":"2024-03-26T03:58:19.446128+0000","last_peered":"2024-03-26T03:58:19.446128+0000","last_clean":"2024-03-26T03:58:19.446128+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:58:19.446128+0000","last_undegraded":"2024-03-26T03:58:19.446128+0000","last_fullsized":"2024-03-26T03:58:19.446128+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.765904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.765904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":330,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339609+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T03:58:19.339609+0000","last_peered":"2024-03-26T03:58:19.339609+0000","last_clean":"2024-03-26T03:58:19.339609+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T03:58:19.339609+0000","last_undegraded":"2024-03-26T03:58:19.339609+0000","last_fullsized":"2024-03-26T03:58:19.339609+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"89'23","reported_seq":53,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338109+0000","last_change":"2024-03-26T03:57:49.481780+0000","last_active":"2024-03-26T03:58:19.338109+0000","last_peered":"2024-03-26T03:58:19.338109+0000","last_clean":"2024-03-26T03:58:19.338109+0000","last_became_active":"2024-03-26T03:57:49.373634+0000","last_became_peered":"2024-03-26T03:57:49.373634+0000","last_unstale":"2024-03-26T03:58:19.338109+0000","last_undegraded":"2024-03-26T03:58:19.338109+0000","last_fullsized":"2024-03-26T03:58:19.338109+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":362,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376658+0000","last_change":"2024-03-26T03:55:59.324307+0000","last_active":"2024-03-26T03:58:19.376658+0000","last_peered":"2024-03-26T03:58:19.376658+0000","last_clean":"2024-03-26T03:58:19.376658+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T03:58:19.376658+0000","last_undegraded":"2024-03-26T03:58:19.376658+0000","last_fullsized":"2024-03-26T03:58:19.376658+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402202+0000","last_change":"2024-03-26T03:57:49.377475+0000","last_active":"2024-03-26T03:58:19.402202+0000","last_peered":"2024-03-26T03:58:19.402202+0000","last_clean":"2024-03-26T03:58:19.402202+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T03:58:19.402202+0000","last_undegraded":"2024-03-26T03:58:19.402202+0000","last_fullsized":"2024-03-26T03:58:19.402202+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":385,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498526+0000","last_change":"2024-03-26T03:58:11.047294+0000","last_active":"2024-03-26T03:58:19.498526+0000","last_peered":"2024-03-26T03:58:19.498526+0000","last_clean":"2024-03-26T03:58:19.498526+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T03:58:19.498526+0000","last_undegraded":"2024-03-26T03:58:19.498526+0000","last_fullsized":"2024-03-26T03:58:19.498526+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.047184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.047184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:39.616278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"152'13","reported_seq":404,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401637+0000","last_change":"2024-03-26T03:55:30.839253+0000","last_active":"2024-03-26T03:58:19.401637+0000","last_peered":"2024-03-26T03:58:19.401637+0000","last_clean":"2024-03-26T03:58:19.401637+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:58:19.401637+0000","last_undegraded":"2024-03-26T03:58:19.401637+0000","last_fullsized":"2024-03-26T03:58:19.401637+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:55:30.839058+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.839058+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337839+0000","last_change":"2024-03-26T03:57:49.373069+0000","last_active":"2024-03-26T03:58:19.337839+0000","last_peered":"2024-03-26T03:58:19.337839+0000","last_clean":"2024-03-26T03:58:19.337839+0000","last_became_active":"2024-03-26T03:57:49.370042+0000","last_became_peered":"2024-03-26T03:57:49.370042+0000","last_unstale":"2024-03-26T03:58:19.337839+0000","last_undegraded":"2024-03-26T03:58:19.337839+0000","last_fullsized":"2024-03-26T03:58:19.337839+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.804652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":377,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:20.461745+0000","last_change":"2024-03-26T03:58:20.461781+0000","last_active":"2024-03-26T03:58:20.461745+0000","last_peered":"2024-03-26T03:58:20.461745+0000","last_clean":"2024-03-26T03:58:20.461745+0000","last_became_active":"2024-03-26T03:57:49.379985+0000","last_became_peered":"2024-03-26T03:57:49.379985+0000","last_unstale":"2024-03-26T03:58:20.461745+0000","last_undegraded":"2024-03-26T03:58:20.461745+0000","last_fullsized":"2024-03-26T03:58:20.461745+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:20.461665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:20.461665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:46.093646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":371,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335961+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:58:19.335961+0000","last_peered":"2024-03-26T03:58:19.335961+0000","last_clean":"2024-03-26T03:58:19.335961+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:58:19.335961+0000","last_undegraded":"2024-03-26T03:58:19.335961+0000","last_fullsized":"2024-03-26T03:58:19.335961+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":399,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336637+0000","last_change":"2024-03-26T03:55:59.300136+0000","last_active":"2024-03-26T03:58:19.336637+0000","last_peered":"2024-03-26T03:58:19.336637+0000","last_clean":"2024-03-26T03:58:19.336637+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T03:58:19.336637+0000","last_undegraded":"2024-03-26T03:58:19.336637+0000","last_fullsized":"2024-03-26T03:58:19.336637+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":356,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401793+0000","last_change":"2024-03-26T03:57:49.360446+0000","last_active":"2024-03-26T03:58:19.401793+0000","last_peered":"2024-03-26T03:58:19.401793+0000","last_clean":"2024-03-26T03:58:19.401793+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T03:58:19.401793+0000","last_undegraded":"2024-03-26T03:58:19.401793+0000","last_fullsized":"2024-03-26T03:58:19.401793+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":462,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339441+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T03:58:19.339441+0000","last_peered":"2024-03-26T03:58:19.339441+0000","last_clean":"2024-03-26T03:58:19.339441+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T03:58:19.339441+0000","last_undegraded":"2024-03-26T03:58:19.339441+0000","last_fullsized":"2024-03-26T03:58:19.339441+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":373,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446319+0000","last_change":"2024-03-26T03:55:57.916905+0000","last_active":"2024-03-26T03:58:19.446319+0000","last_peered":"2024-03-26T03:58:19.446319+0000","last_clean":"2024-03-26T03:58:19.446319+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:58:19.446319+0000","last_undegraded":"2024-03-26T03:58:19.446319+0000","last_fullsized":"2024-03-26T03:58:19.446319+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.916786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.916786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"73'22","reported_seq":55,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401823+0000","last_change":"2024-03-26T03:57:49.435763+0000","last_active":"2024-03-26T03:58:19.401823+0000","last_peered":"2024-03-26T03:58:19.401823+0000","last_clean":"2024-03-26T03:58:19.401823+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T03:58:19.401823+0000","last_undegraded":"2024-03-26T03:58:19.401823+0000","last_fullsized":"2024-03-26T03:58:19.401823+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":361,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336547+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T03:58:19.336547+0000","last_peered":"2024-03-26T03:58:19.336547+0000","last_clean":"2024-03-26T03:58:19.336547+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T03:58:19.336547+0000","last_undegraded":"2024-03-26T03:58:19.336547+0000","last_fullsized":"2024-03-26T03:58:19.336547+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":51,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401690+0000","last_change":"2024-03-26T03:57:49.512060+0000","last_active":"2024-03-26T03:58:19.401690+0000","last_peered":"2024-03-26T03:58:19.401690+0000","last_clean":"2024-03-26T03:58:19.401690+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T03:58:19.401690+0000","last_undegraded":"2024-03-26T03:58:19.401690+0000","last_fullsized":"2024-03-26T03:58:19.401690+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":403,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.377015+0000","last_change":"2024-03-26T03:57:49.480593+0000","last_active":"2024-03-26T03:58:19.377015+0000","last_peered":"2024-03-26T03:58:19.377015+0000","last_clean":"2024-03-26T03:58:19.377015+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T03:58:19.377015+0000","last_undegraded":"2024-03-26T03:58:19.377015+0000","last_fullsized":"2024-03-26T03:58:19.377015+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"89'20","reported_seq":388,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339517+0000","last_change":"2024-03-26T03:57:49.431007+0000","last_active":"2024-03-26T03:58:19.339517+0000","last_peered":"2024-03-26T03:58:19.339517+0000","last_clean":"2024-03-26T03:58:19.339517+0000","last_became_active":"2024-03-26T03:57:49.382991+0000","last_became_peered":"2024-03-26T03:57:49.382991+0000","last_unstale":"2024-03-26T03:58:19.339517+0000","last_undegraded":"2024-03-26T03:58:19.339517+0000","last_fullsized":"2024-03-26T03:58:19.339517+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":371,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401548+0000","last_change":"2024-03-26T03:58:15.056185+0000","last_active":"2024-03-26T03:58:19.401548+0000","last_peered":"2024-03-26T03:58:19.401548+0000","last_clean":"2024-03-26T03:58:19.401548+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:58:19.401548+0000","last_undegraded":"2024-03-26T03:58:19.401548+0000","last_fullsized":"2024-03-26T03:58:19.401548+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:58:15.056047+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.056047+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:16.858445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405425+0000","last_change":"2024-03-26T03:57:49.368813+0000","last_active":"2024-03-26T03:58:19.405425+0000","last_peered":"2024-03-26T03:58:19.405425+0000","last_clean":"2024-03-26T03:58:19.405425+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T03:58:19.405425+0000","last_undegraded":"2024-03-26T03:58:19.405425+0000","last_fullsized":"2024-03-26T03:58:19.405425+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":379,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446185+0000","last_change":"2024-03-26T03:58:15.134862+0000","last_active":"2024-03-26T03:58:19.446185+0000","last_peered":"2024-03-26T03:58:19.446185+0000","last_clean":"2024-03-26T03:58:19.446185+0000","last_became_active":"2024-03-26T03:50:45.593850+0000","last_became_peered":"2024-03-26T03:50:45.593850+0000","last_unstale":"2024-03-26T03:58:19.446185+0000","last_undegraded":"2024-03-26T03:58:19.446185+0000","last_fullsized":"2024-03-26T03:58:19.446185+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:15.134741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.134741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:35.717482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'21","reported_seq":440,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402190+0000","last_change":"2024-03-26T03:57:49.445142+0000","last_active":"2024-03-26T03:58:19.402190+0000","last_peered":"2024-03-26T03:58:19.402190+0000","last_clean":"2024-03-26T03:58:19.402190+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T03:58:19.402190+0000","last_undegraded":"2024-03-26T03:58:19.402190+0000","last_fullsized":"2024-03-26T03:58:19.402190+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":367,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498043+0000","last_change":"2024-03-26T03:58:16.141306+0000","last_active":"2024-03-26T03:58:19.498043+0000","last_peered":"2024-03-26T03:58:19.498043+0000","last_clean":"2024-03-26T03:58:19.498043+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T03:58:19.498043+0000","last_undegraded":"2024-03-26T03:58:19.498043+0000","last_fullsized":"2024-03-26T03:58:19.498043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.141216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.141216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:27.650832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":327,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339868+0000","last_change":"2024-03-26T03:57:49.381012+0000","last_active":"2024-03-26T03:58:19.339868+0000","last_peered":"2024-03-26T03:58:19.339868+0000","last_clean":"2024-03-26T03:58:19.339868+0000","last_became_active":"2024-03-26T03:57:49.380167+0000","last_became_peered":"2024-03-26T03:57:49.380167+0000","last_unstale":"2024-03-26T03:58:19.339868+0000","last_undegraded":"2024-03-26T03:58:19.339868+0000","last_fullsized":"2024-03-26T03:58:19.339868+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":316,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376850+0000","last_change":"2024-03-26T03:57:49.378443+0000","last_active":"2024-03-26T03:58:19.376850+0000","last_peered":"2024-03-26T03:58:19.376850+0000","last_clean":"2024-03-26T03:58:19.376850+0000","last_became_active":"2024-03-26T03:57:49.377869+0000","last_became_peered":"2024-03-26T03:57:49.377869+0000","last_unstale":"2024-03-26T03:58:19.376850+0000","last_undegraded":"2024-03-26T03:58:19.376850+0000","last_fullsized":"2024-03-26T03:58:19.376850+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'13","reported_seq":49,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405257+0000","last_change":"2024-03-26T03:57:49.511256+0000","last_active":"2024-03-26T03:58:19.405257+0000","last_peered":"2024-03-26T03:58:19.405257+0000","last_clean":"2024-03-26T03:58:19.405257+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T03:58:19.405257+0000","last_undegraded":"2024-03-26T03:58:19.405257+0000","last_fullsized":"2024-03-26T03:58:19.405257+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"229'14236","reported_seq":20823,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:23.014195+0000","last_change":"2024-03-26T03:57:00.336002+0000","last_active":"2024-03-26T03:58:23.014195+0000","last_peered":"2024-03-26T03:58:23.014195+0000","last_clean":"2024-03-26T03:58:23.014195+0000","last_became_active":"2024-03-26T03:57:00.322652+0000","last_became_peered":"2024-03-26T03:57:00.322652+0000","last_unstale":"2024-03-26T03:58:23.014195+0000","last_undegraded":"2024-03-26T03:58:23.014195+0000","last_fullsized":"2024-03-26T03:58:23.014195+0000","mapping_epoch":201,"log_start":"197'9384","ondisk_log_start":"197'9384","created":155,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4852,"log_dups_size":0,"ondisk_log_size":4852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134610944,"num_objects":2054,"num_object_clones":0,"num_object_copies":6162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2054,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14236,"num_write_kb":521280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3207,"num_bytes_recovered":210173952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":314,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339541+0000","last_change":"2024-03-26T03:55:59.325043+0000","last_active":"2024-03-26T03:58:19.339541+0000","last_peered":"2024-03-26T03:58:19.339541+0000","last_clean":"2024-03-26T03:58:19.339541+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T03:58:19.339541+0000","last_undegraded":"2024-03-26T03:58:19.339541+0000","last_fullsized":"2024-03-26T03:58:19.339541+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":377,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498315+0000","last_change":"2024-03-26T03:54:16.696733+0000","last_active":"2024-03-26T03:58:19.498315+0000","last_peered":"2024-03-26T03:58:19.498315+0000","last_clean":"2024-03-26T03:58:19.498315+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:58:19.498315+0000","last_undegraded":"2024-03-26T03:58:19.498315+0000","last_fullsized":"2024-03-26T03:58:19.498315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:54:16.696602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.696602+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"73'27","reported_seq":387,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339326+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T03:58:19.339326+0000","last_peered":"2024-03-26T03:58:19.339326+0000","last_clean":"2024-03-26T03:58:19.339326+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T03:58:19.339326+0000","last_undegraded":"2024-03-26T03:58:19.339326+0000","last_fullsized":"2024-03-26T03:58:19.339326+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":500,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.446404+0000","last_change":"2024-03-26T03:55:52.900155+0000","last_active":"2024-03-26T03:58:19.446404+0000","last_peered":"2024-03-26T03:58:19.446404+0000","last_clean":"2024-03-26T03:58:19.446404+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:58:19.446404+0000","last_undegraded":"2024-03-26T03:58:19.446404+0000","last_fullsized":"2024-03-26T03:58:19.446404+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:52.900040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:52.900040+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"229'14143","reported_seq":14304,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:24.302568+0000","last_change":"2024-03-26T03:54:38.786252+0000","last_active":"2024-03-26T03:58:24.302568+0000","last_peered":"2024-03-26T03:58:24.302568+0000","last_clean":"2024-03-26T03:58:24.302568+0000","last_became_active":"2024-03-26T03:54:38.785773+0000","last_became_peered":"2024-03-26T03:54:38.785773+0000","last_unstale":"2024-03-26T03:58:24.302568+0000","last_undegraded":"2024-03-26T03:58:24.302568+0000","last_fullsized":"2024-03-26T03:58:24.302568+0000","mapping_epoch":155,"log_start":"198'9800","ondisk_log_start":"198'9800","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4343,"log_dups_size":0,"ondisk_log_size":4343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131792896,"num_objects":2011,"num_object_clones":0,"num_object_copies":6033,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2011,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14143,"num_write_kb":516928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402122+0000","last_change":"2024-03-26T03:57:49.372738+0000","last_active":"2024-03-26T03:58:19.402122+0000","last_peered":"2024-03-26T03:58:19.402122+0000","last_clean":"2024-03-26T03:58:19.402122+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T03:58:19.402122+0000","last_undegraded":"2024-03-26T03:58:19.402122+0000","last_fullsized":"2024-03-26T03:58:19.402122+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":365,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498243+0000","last_change":"2024-03-26T03:54:18.653671+0000","last_active":"2024-03-26T03:58:19.498243+0000","last_peered":"2024-03-26T03:58:19.498243+0000","last_clean":"2024-03-26T03:58:19.498243+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:58:19.498243+0000","last_undegraded":"2024-03-26T03:58:19.498243+0000","last_fullsized":"2024-03-26T03:58:19.498243+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.653555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.653555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'24","reported_seq":452,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337028+0000","last_change":"2024-03-26T03:57:49.408864+0000","last_active":"2024-03-26T03:58:19.337028+0000","last_peered":"2024-03-26T03:58:19.337028+0000","last_clean":"2024-03-26T03:58:19.337028+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T03:58:19.337028+0000","last_undegraded":"2024-03-26T03:58:19.337028+0000","last_fullsized":"2024-03-26T03:58:19.337028+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":373,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338085+0000","last_change":"2024-03-26T03:58:18.180129+0000","last_active":"2024-03-26T03:58:19.338085+0000","last_peered":"2024-03-26T03:58:19.338085+0000","last_clean":"2024-03-26T03:58:19.338085+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:58:19.338085+0000","last_undegraded":"2024-03-26T03:58:19.338085+0000","last_fullsized":"2024-03-26T03:58:19.338085+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:18.179988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:18.179988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:22.302751+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"229'14019","reported_seq":14180,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.016600+0000","last_change":"2024-03-26T03:54:38.785847+0000","last_active":"2024-03-26T03:58:25.016600+0000","last_peered":"2024-03-26T03:58:25.016600+0000","last_clean":"2024-03-26T03:58:25.016600+0000","last_became_active":"2024-03-26T03:54:38.785420+0000","last_became_peered":"2024-03-26T03:54:38.785420+0000","last_unstale":"2024-03-26T03:58:25.016600+0000","last_undegraded":"2024-03-26T03:58:25.016600+0000","last_fullsized":"2024-03-26T03:58:25.016600+0000","mapping_epoch":155,"log_start":"197'9300","ondisk_log_start":"197'9300","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4719,"log_dups_size":0,"ondisk_log_size":4719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130482176,"num_objects":1991,"num_object_clones":0,"num_object_copies":5973,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1991,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14019,"num_write_kb":512320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":373,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:21.112197+0000","last_change":"2024-03-26T03:58:21.112228+0000","last_active":"2024-03-26T03:58:21.112197+0000","last_peered":"2024-03-26T03:58:21.112197+0000","last_clean":"2024-03-26T03:58:21.112197+0000","last_became_active":"2024-03-26T03:57:49.354993+0000","last_became_peered":"2024-03-26T03:57:49.354993+0000","last_unstale":"2024-03-26T03:58:21.112197+0000","last_undegraded":"2024-03-26T03:58:21.112197+0000","last_fullsized":"2024-03-26T03:58:21.112197+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:21.112131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:21.112131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:35.978284+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":382,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338699+0000","last_change":"2024-03-26T03:58:16.032385+0000","last_active":"2024-03-26T03:58:19.338699+0000","last_peered":"2024-03-26T03:58:19.338699+0000","last_clean":"2024-03-26T03:58:19.338699+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T03:58:19.338699+0000","last_undegraded":"2024-03-26T03:58:19.338699+0000","last_fullsized":"2024-03-26T03:58:19.338699+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.032069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.032069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:41.260369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":392,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498732+0000","last_change":"2024-03-26T03:57:49.349093+0000","last_active":"2024-03-26T03:58:19.498732+0000","last_peered":"2024-03-26T03:58:19.498732+0000","last_clean":"2024-03-26T03:58:19.498732+0000","last_became_active":"2024-03-26T03:57:49.348729+0000","last_became_peered":"2024-03-26T03:57:49.348729+0000","last_unstale":"2024-03-26T03:58:19.498732+0000","last_undegraded":"2024-03-26T03:58:19.498732+0000","last_fullsized":"2024-03-26T03:58:19.498732+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.612556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.612556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"73'25","reported_seq":388,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401294+0000","last_change":"2024-03-26T03:55:59.302041+0000","last_active":"2024-03-26T03:58:19.401294+0000","last_peered":"2024-03-26T03:58:19.401294+0000","last_clean":"2024-03-26T03:58:19.401294+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T03:58:19.401294+0000","last_undegraded":"2024-03-26T03:58:19.401294+0000","last_fullsized":"2024-03-26T03:58:19.401294+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:55:14.773103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773103+0000","objects_scrubbed":10,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":40,"num_read_kb":28,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.7","version":"229'14153","reported_seq":14323,"reported_epoch":229,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T03:58:25.153442+0000","last_change":"2024-03-26T03:57:59.729899+0000","last_active":"2024-03-26T03:58:25.153442+0000","last_peered":"2024-03-26T03:58:25.153442+0000","last_clean":"2024-03-26T03:57:48.458318+0000","last_became_active":"2024-03-26T03:57:50.448280+0000","last_became_peered":"2024-03-26T03:57:50.448280+0000","last_unstale":"2024-03-26T03:58:25.153442+0000","last_undegraded":"2024-03-26T03:58:25.153442+0000","last_fullsized":"2024-03-26T03:58:25.153442+0000","mapping_epoch":215,"log_start":"198'9900","ondisk_log_start":"198'9900","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4253,"log_dups_size":0,"ondisk_log_size":4253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128385024,"num_objects":1959,"num_object_clones":0,"num_object_copies":5877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1959,"num_objects_unfound":0,"num_objects_dirty":1959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14153,"num_write_kb":515584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,2,4],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1959}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.377132+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T03:58:19.377132+0000","last_peered":"2024-03-26T03:58:19.377132+0000","last_clean":"2024-03-26T03:58:19.377132+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:58:19.377132+0000","last_undegraded":"2024-03-26T03:58:19.377132+0000","last_fullsized":"2024-03-26T03:58:19.377132+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":385,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338485+0000","last_change":"2024-03-26T03:58:10.028917+0000","last_active":"2024-03-26T03:58:19.338485+0000","last_peered":"2024-03-26T03:58:19.338485+0000","last_clean":"2024-03-26T03:58:19.338485+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T03:58:19.338485+0000","last_undegraded":"2024-03-26T03:58:19.338485+0000","last_fullsized":"2024-03-26T03:58:19.338485+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:10.028829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:58:10.028829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:15.966391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":349,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339175+0000","last_change":"2024-03-26T03:57:49.368242+0000","last_active":"2024-03-26T03:58:19.339175+0000","last_peered":"2024-03-26T03:58:19.339175+0000","last_clean":"2024-03-26T03:58:19.339175+0000","last_became_active":"2024-03-26T03:57:49.367885+0000","last_became_peered":"2024-03-26T03:57:49.367885+0000","last_unstale":"2024-03-26T03:58:19.339175+0000","last_undegraded":"2024-03-26T03:58:19.339175+0000","last_fullsized":"2024-03-26T03:58:19.339175+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"73'15","reported_seq":416,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404789+0000","last_change":"2024-03-26T03:55:59.309207+0000","last_active":"2024-03-26T03:58:19.404789+0000","last_peered":"2024-03-26T03:58:19.404789+0000","last_clean":"2024-03-26T03:58:19.404789+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T03:58:19.404789+0000","last_undegraded":"2024-03-26T03:58:19.404789+0000","last_fullsized":"2024-03-26T03:58:19.404789+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"229'13931","reported_seq":23529,"reported_epoch":229,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:58:23.226776+0000","last_change":"2024-03-26T03:57:59.729027+0000","last_active":"2024-03-26T03:58:23.226776+0000","last_peered":"2024-03-26T03:58:23.226776+0000","last_clean":"2024-03-26T03:55:19.234210+0000","last_became_active":"2024-03-26T03:57:51.346040+0000","last_became_peered":"2024-03-26T03:57:51.346040+0000","last_unstale":"2024-03-26T03:58:23.226776+0000","last_undegraded":"2024-03-26T03:58:23.226776+0000","last_fullsized":"2024-03-26T03:58:23.226776+0000","mapping_epoch":216,"log_start":"197'9277","ondisk_log_start":"197'9277","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132448256,"num_objects":2021,"num_object_clones":0,"num_object_copies":6063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":536,"num_objects_unfound":0,"num_objects_dirty":2021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13931,"num_write_kb":510464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4876,"num_bytes_recovered":319553536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,7,0],"avail_no_missing":["5","0","7"],"object_location_counts":[{"shards":"0,5,7","objects":2021}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405235+0000","last_change":"2024-03-26T03:57:49.361707+0000","last_active":"2024-03-26T03:58:19.405235+0000","last_peered":"2024-03-26T03:58:19.405235+0000","last_clean":"2024-03-26T03:58:19.405235+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T03:58:19.405235+0000","last_undegraded":"2024-03-26T03:58:19.405235+0000","last_fullsized":"2024-03-26T03:58:19.405235+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":317,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339378+0000","last_change":"2024-03-26T03:55:59.316898+0000","last_active":"2024-03-26T03:58:19.339378+0000","last_peered":"2024-03-26T03:58:19.339378+0000","last_clean":"2024-03-26T03:58:19.339378+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T03:58:19.339378+0000","last_undegraded":"2024-03-26T03:58:19.339378+0000","last_fullsized":"2024-03-26T03:58:19.339378+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":371,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335863+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T03:58:19.335863+0000","last_peered":"2024-03-26T03:58:19.335863+0000","last_clean":"2024-03-26T03:58:19.335863+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:58:19.335863+0000","last_undegraded":"2024-03-26T03:58:19.335863+0000","last_fullsized":"2024-03-26T03:58:19.335863+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"215'36","reported_seq":597,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.403764+0000","last_change":"2024-03-26T03:57:49.488773+0000","last_active":"2024-03-26T03:58:19.403764+0000","last_peered":"2024-03-26T03:58:19.403764+0000","last_clean":"2024-03-26T03:58:19.403764+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T03:58:19.403764+0000","last_undegraded":"2024-03-26T03:58:19.403764+0000","last_fullsized":"2024-03-26T03:58:19.403764+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":357,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404924+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:58:19.404924+0000","last_peered":"2024-03-26T03:58:19.404924+0000","last_clean":"2024-03-26T03:58:19.404924+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:58:19.404924+0000","last_undegraded":"2024-03-26T03:58:19.404924+0000","last_fullsized":"2024-03-26T03:58:19.404924+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"229'14506","reported_seq":16887,"reported_epoch":229,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T03:58:25.268519+0000","last_change":"2024-03-26T03:58:21.279927+0000","last_active":"2024-03-26T03:58:25.268519+0000","last_peered":"2024-03-26T03:58:25.268519+0000","last_clean":"2024-03-26T03:57:48.478090+0000","last_became_active":"2024-03-26T03:57:50.438472+0000","last_became_peered":"2024-03-26T03:57:50.438472+0000","last_unstale":"2024-03-26T03:58:25.268519+0000","last_undegraded":"2024-03-26T03:58:25.268519+0000","last_fullsized":"2024-03-26T03:58:25.268519+0000","mapping_epoch":215,"log_start":"198'10200","ondisk_log_start":"198'10200","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4306,"log_dups_size":0,"ondisk_log_size":4306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1043,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14506,"num_write_kb":531904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1108,"num_bytes_recovered":72613888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":2116}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":380,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498946+0000","last_change":"2024-03-26T03:57:49.346969+0000","last_active":"2024-03-26T03:58:19.498946+0000","last_peered":"2024-03-26T03:58:19.498946+0000","last_clean":"2024-03-26T03:58:19.498946+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T03:58:19.498946+0000","last_undegraded":"2024-03-26T03:58:19.498946+0000","last_fullsized":"2024-03-26T03:58:19.498946+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.773109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":378,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337737+0000","last_change":"2024-03-26T03:55:29.863335+0000","last_active":"2024-03-26T03:58:19.337737+0000","last_peered":"2024-03-26T03:58:19.337737+0000","last_clean":"2024-03-26T03:58:19.337737+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:58:19.337737+0000","last_undegraded":"2024-03-26T03:58:19.337737+0000","last_fullsized":"2024-03-26T03:58:19.337737+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.863201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.863201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":402,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404575+0000","last_change":"2024-03-26T03:58:11.071645+0000","last_active":"2024-03-26T03:58:19.404575+0000","last_peered":"2024-03-26T03:58:19.404575+0000","last_clean":"2024-03-26T03:58:19.404575+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T03:58:19.404575+0000","last_undegraded":"2024-03-26T03:58:19.404575+0000","last_fullsized":"2024-03-26T03:58:19.404575+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.071522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.071522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:31.854810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"73'19","reported_seq":400,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404197+0000","last_change":"2024-03-26T03:55:59.300086+0000","last_active":"2024-03-26T03:58:19.404197+0000","last_peered":"2024-03-26T03:58:19.404197+0000","last_clean":"2024-03-26T03:58:19.404197+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T03:58:19.404197+0000","last_undegraded":"2024-03-26T03:58:19.404197+0000","last_fullsized":"2024-03-26T03:58:19.404197+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:52:32.926441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:32.926441+0000","objects_scrubbed":6,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":374,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401928+0000","last_change":"2024-03-26T03:58:08.985333+0000","last_active":"2024-03-26T03:58:19.401928+0000","last_peered":"2024-03-26T03:58:19.401928+0000","last_clean":"2024-03-26T03:58:19.401928+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T03:58:19.401928+0000","last_undegraded":"2024-03-26T03:58:19.401928+0000","last_fullsized":"2024-03-26T03:58:19.401928+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:08.985207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:08.985207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:34.567248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"229'14440","reported_seq":16066,"reported_epoch":229,"state":"active+recovering+degraded+remapped","last_fresh":"2024-03-26T03:58:23.218369+0000","last_change":"2024-03-26T03:57:53.925881+0000","last_active":"2024-03-26T03:58:23.218369+0000","last_peered":"2024-03-26T03:58:23.218369+0000","last_clean":"2024-03-26T03:55:19.247499+0000","last_became_active":"2024-03-26T03:57:53.119894+0000","last_became_peered":"2024-03-26T03:57:53.119894+0000","last_unstale":"2024-03-26T03:58:23.218369+0000","last_undegraded":"2024-03-26T03:57:53.069743+0000","last_fullsized":"2024-03-26T03:58:23.218369+0000","mapping_epoch":215,"log_start":"197'9700","ondisk_log_start":"197'9700","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135921664,"num_objects":2074,"num_object_clones":0,"num_object_copies":6222,"num_objects_missing_on_primary":2537,"num_objects_missing":2537,"num_objects_degraded":887,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":2074,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14440,"num_write_kb":528448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":47251456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":887},{"shards":"0,4,5","objects":1176}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":323,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339931+0000","last_change":"2024-03-26T03:57:49.368565+0000","last_active":"2024-03-26T03:58:19.339931+0000","last_peered":"2024-03-26T03:58:19.339931+0000","last_clean":"2024-03-26T03:58:19.339931+0000","last_became_active":"2024-03-26T03:57:49.367895+0000","last_became_peered":"2024-03-26T03:57:49.367895+0000","last_unstale":"2024-03-26T03:58:19.339931+0000","last_undegraded":"2024-03-26T03:58:19.339931+0000","last_fullsized":"2024-03-26T03:58:19.339931+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":629,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.445918+0000","last_change":"2024-03-26T03:55:17.691688+0000","last_active":"2024-03-26T03:58:19.445918+0000","last_peered":"2024-03-26T03:58:19.445918+0000","last_clean":"2024-03-26T03:58:19.445918+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:58:19.445918+0000","last_undegraded":"2024-03-26T03:58:19.445918+0000","last_fullsized":"2024-03-26T03:58:19.445918+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:55:17.691502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.691502+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405560+0000","last_change":"2024-03-26T03:57:49.369789+0000","last_active":"2024-03-26T03:58:19.405560+0000","last_peered":"2024-03-26T03:58:19.405560+0000","last_clean":"2024-03-26T03:58:19.405560+0000","last_became_active":"2024-03-26T03:57:49.369463+0000","last_became_peered":"2024-03-26T03:57:49.369463+0000","last_unstale":"2024-03-26T03:58:19.405560+0000","last_undegraded":"2024-03-26T03:58:19.405560+0000","last_fullsized":"2024-03-26T03:58:19.405560+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"100'17","reported_seq":448,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337074+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T03:58:19.337074+0000","last_peered":"2024-03-26T03:58:19.337074+0000","last_clean":"2024-03-26T03:58:19.337074+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:58:19.337074+0000","last_undegraded":"2024-03-26T03:58:19.337074+0000","last_fullsized":"2024-03-26T03:58:19.337074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":374,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.498464+0000","last_change":"2024-03-26T03:58:13.093942+0000","last_active":"2024-03-26T03:58:19.498464+0000","last_peered":"2024-03-26T03:58:19.498464+0000","last_clean":"2024-03-26T03:58:19.498464+0000","last_became_active":"2024-03-26T03:55:59.298950+0000","last_became_peered":"2024-03-26T03:55:59.298950+0000","last_unstale":"2024-03-26T03:58:19.498464+0000","last_undegraded":"2024-03-26T03:58:19.498464+0000","last_fullsized":"2024-03-26T03:58:19.498464+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:13.093867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:58:13.093867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:37.325120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"229'14064","reported_seq":20656,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.256465+0000","last_change":"2024-03-26T03:58:04.403791+0000","last_active":"2024-03-26T03:58:25.256465+0000","last_peered":"2024-03-26T03:58:25.256465+0000","last_clean":"2024-03-26T03:58:25.256465+0000","last_became_active":"2024-03-26T03:58:04.337655+0000","last_became_peered":"2024-03-26T03:58:04.337655+0000","last_unstale":"2024-03-26T03:58:25.256465+0000","last_undegraded":"2024-03-26T03:58:25.256465+0000","last_fullsized":"2024-03-26T03:58:25.256465+0000","mapping_epoch":222,"log_start":"198'9800","ondisk_log_start":"198'9800","created":155,"last_epoch_clean":223,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4264,"log_dups_size":0,"ondisk_log_size":4264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131727360,"num_objects":2010,"num_object_clones":0,"num_object_copies":6030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2010,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14064,"num_write_kb":514368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3209,"num_bytes_recovered":163381248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":365,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.337951+0000","last_change":"2024-03-26T03:57:49.379889+0000","last_active":"2024-03-26T03:58:19.337951+0000","last_peered":"2024-03-26T03:58:19.337951+0000","last_clean":"2024-03-26T03:58:19.337951+0000","last_became_active":"2024-03-26T03:57:49.379569+0000","last_became_peered":"2024-03-26T03:57:49.379569+0000","last_unstale":"2024-03-26T03:58:19.337951+0000","last_undegraded":"2024-03-26T03:58:19.337951+0000","last_fullsized":"2024-03-26T03:58:19.337951+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:20.691483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:20.691483+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":394,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376890+0000","last_change":"2024-03-26T03:57:49.364364+0000","last_active":"2024-03-26T03:58:19.376890+0000","last_peered":"2024-03-26T03:58:19.376890+0000","last_clean":"2024-03-26T03:58:19.376890+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T03:58:19.376890+0000","last_undegraded":"2024-03-26T03:58:19.376890+0000","last_fullsized":"2024-03-26T03:58:19.376890+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":957,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:22.504155+0000","last_change":"2024-03-26T03:55:58.942740+0000","last_active":"2024-03-26T03:58:22.504155+0000","last_peered":"2024-03-26T03:58:22.504155+0000","last_clean":"2024-03-26T03:58:22.504155+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:58:22.504155+0000","last_undegraded":"2024-03-26T03:58:22.504155+0000","last_fullsized":"2024-03-26T03:58:22.504155+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:55:58.942635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.942635+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":572,"num_read_kb":567,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":379,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401894+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T03:58:19.401894+0000","last_peered":"2024-03-26T03:58:19.401894+0000","last_clean":"2024-03-26T03:58:19.401894+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:58:19.401894+0000","last_undegraded":"2024-03-26T03:58:19.401894+0000","last_fullsized":"2024-03-26T03:58:19.401894+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":304,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339936+0000","last_change":"2024-03-26T03:55:59.311162+0000","last_active":"2024-03-26T03:58:19.339936+0000","last_peered":"2024-03-26T03:58:19.339936+0000","last_clean":"2024-03-26T03:58:19.339936+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T03:58:19.339936+0000","last_undegraded":"2024-03-26T03:58:19.339936+0000","last_fullsized":"2024-03-26T03:58:19.339936+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"229'14397","reported_seq":19960,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:24.729441+0000","last_change":"2024-03-26T03:58:01.712492+0000","last_active":"2024-03-26T03:58:24.729441+0000","last_peered":"2024-03-26T03:58:24.729441+0000","last_clean":"2024-03-26T03:58:24.729441+0000","last_became_active":"2024-03-26T03:58:01.665790+0000","last_became_peered":"2024-03-26T03:58:01.665790+0000","last_unstale":"2024-03-26T03:58:24.729441+0000","last_undegraded":"2024-03-26T03:58:24.729441+0000","last_fullsized":"2024-03-26T03:58:24.729441+0000","mapping_epoch":219,"log_start":"198'10108","ondisk_log_start":"198'10108","created":155,"last_epoch_clean":220,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4289,"log_dups_size":0,"ondisk_log_size":4289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130088960,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14397,"num_write_kb":524224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2694,"num_bytes_recovered":176553984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":358,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338258+0000","last_change":"2024-03-26T03:55:15.751265+0000","last_active":"2024-03-26T03:58:19.338258+0000","last_peered":"2024-03-26T03:58:19.338258+0000","last_clean":"2024-03-26T03:58:19.338258+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:58:19.338258+0000","last_undegraded":"2024-03-26T03:58:19.338258+0000","last_fullsized":"2024-03-26T03:58:19.338258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:15.751136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:15.751136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":373,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338171+0000","last_change":"2024-03-26T03:55:59.301777+0000","last_active":"2024-03-26T03:58:19.338171+0000","last_peered":"2024-03-26T03:58:19.338171+0000","last_clean":"2024-03-26T03:58:19.338171+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T03:58:19.338171+0000","last_undegraded":"2024-03-26T03:58:19.338171+0000","last_fullsized":"2024-03-26T03:58:19.338171+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:31.824648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:31.824648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"73'25","reported_seq":418,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404622+0000","last_change":"2024-03-26T03:55:59.309535+0000","last_active":"2024-03-26T03:58:19.404622+0000","last_peered":"2024-03-26T03:58:19.404622+0000","last_clean":"2024-03-26T03:58:19.404622+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T03:58:19.404622+0000","last_undegraded":"2024-03-26T03:58:19.404622+0000","last_fullsized":"2024-03-26T03:58:19.404622+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":304,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376872+0000","last_change":"2024-03-26T03:55:59.308367+0000","last_active":"2024-03-26T03:58:19.376872+0000","last_peered":"2024-03-26T03:58:19.376872+0000","last_clean":"2024-03-26T03:58:19.376872+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T03:58:19.376872+0000","last_undegraded":"2024-03-26T03:58:19.376872+0000","last_fullsized":"2024-03-26T03:58:19.376872+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.e","version":"229'14170","reported_seq":14331,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.030088+0000","last_change":"2024-03-26T03:54:38.379391+0000","last_active":"2024-03-26T03:58:25.030088+0000","last_peered":"2024-03-26T03:58:25.030088+0000","last_clean":"2024-03-26T03:58:25.030088+0000","last_became_active":"2024-03-26T03:54:38.379010+0000","last_became_peered":"2024-03-26T03:54:38.379010+0000","last_unstale":"2024-03-26T03:58:25.030088+0000","last_undegraded":"2024-03-26T03:58:25.030088+0000","last_fullsized":"2024-03-26T03:58:25.030088+0000","mapping_epoch":155,"log_start":"197'9400","ondisk_log_start":"197'9400","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4770,"log_dups_size":0,"ondisk_log_size":4770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132251648,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14170,"num_write_kb":518016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":366,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.376692+0000","last_change":"2024-03-26T03:57:49.378513+0000","last_active":"2024-03-26T03:58:19.376692+0000","last_peered":"2024-03-26T03:58:19.376692+0000","last_clean":"2024-03-26T03:58:19.376692+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T03:58:19.376692+0000","last_undegraded":"2024-03-26T03:58:19.376692+0000","last_fullsized":"2024-03-26T03:58:19.376692+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":335,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339771+0000","last_change":"2024-03-26T03:57:49.368358+0000","last_active":"2024-03-26T03:58:19.339771+0000","last_peered":"2024-03-26T03:58:19.339771+0000","last_clean":"2024-03-26T03:58:19.339771+0000","last_became_active":"2024-03-26T03:57:49.367990+0000","last_became_peered":"2024-03-26T03:57:49.367990+0000","last_unstale":"2024-03-26T03:58:19.339771+0000","last_undegraded":"2024-03-26T03:58:19.339771+0000","last_fullsized":"2024-03-26T03:58:19.339771+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"73'16","reported_seq":396,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401539+0000","last_change":"2024-03-26T03:55:59.299304+0000","last_active":"2024-03-26T03:58:19.401539+0000","last_peered":"2024-03-26T03:58:19.401539+0000","last_clean":"2024-03-26T03:58:19.401539+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T03:58:19.401539+0000","last_undegraded":"2024-03-26T03:58:19.401539+0000","last_fullsized":"2024-03-26T03:58:19.401539+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"215'13","reported_seq":515,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404462+0000","last_change":"2024-03-26T03:57:49.447118+0000","last_active":"2024-03-26T03:58:19.404462+0000","last_peered":"2024-03-26T03:58:19.404462+0000","last_clean":"2024-03-26T03:58:19.404462+0000","last_became_active":"2024-03-26T03:57:49.361026+0000","last_became_peered":"2024-03-26T03:57:49.361026+0000","last_unstale":"2024-03-26T03:58:19.404462+0000","last_undegraded":"2024-03-26T03:58:19.404462+0000","last_fullsized":"2024-03-26T03:58:19.404462+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"229'14123","reported_seq":20641,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:23.067603+0000","last_change":"2024-03-26T03:57:28.557742+0000","last_active":"2024-03-26T03:58:23.067603+0000","last_peered":"2024-03-26T03:58:23.067603+0000","last_clean":"2024-03-26T03:58:23.067603+0000","last_became_active":"2024-03-26T03:57:28.533151+0000","last_became_peered":"2024-03-26T03:57:28.533151+0000","last_unstale":"2024-03-26T03:58:23.067603+0000","last_undegraded":"2024-03-26T03:58:23.067603+0000","last_fullsized":"2024-03-26T03:58:23.067603+0000","mapping_epoch":208,"log_start":"196'9235","ondisk_log_start":"196'9235","created":155,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4888,"log_dups_size":0,"ondisk_log_size":4888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133955608,"num_objects":2045,"num_object_clones":0,"num_object_copies":6135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2045,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14123,"num_write_kb":517313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3174,"num_bytes_recovered":208011264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":367,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:23.160451+0000","last_change":"2024-03-26T03:58:23.160475+0000","last_active":"2024-03-26T03:58:23.160451+0000","last_peered":"2024-03-26T03:58:23.160451+0000","last_clean":"2024-03-26T03:58:23.160451+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:58:23.160451+0000","last_undegraded":"2024-03-26T03:58:23.160451+0000","last_fullsized":"2024-03-26T03:58:23.160451+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:23.160383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:58:23.160383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:31.341937+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":366,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401991+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:58:19.401991+0000","last_peered":"2024-03-26T03:58:19.401991+0000","last_clean":"2024-03-26T03:58:19.401991+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:58:19.401991+0000","last_undegraded":"2024-03-26T03:58:19.401991+0000","last_fullsized":"2024-03-26T03:58:19.401991+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"73'20","reported_seq":427,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336754+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T03:58:19.336754+0000","last_peered":"2024-03-26T03:58:19.336754+0000","last_clean":"2024-03-26T03:58:19.336754+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T03:58:19.336754+0000","last_undegraded":"2024-03-26T03:58:19.336754+0000","last_fullsized":"2024-03-26T03:58:19.336754+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":51,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401242+0000","last_change":"2024-03-26T03:57:49.461921+0000","last_active":"2024-03-26T03:58:19.401242+0000","last_peered":"2024-03-26T03:58:19.401242+0000","last_clean":"2024-03-26T03:58:19.401242+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T03:58:19.401242+0000","last_undegraded":"2024-03-26T03:58:19.401242+0000","last_fullsized":"2024-03-26T03:58:19.401242+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"229'14109","reported_seq":25550,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.016708+0000","last_change":"2024-03-26T03:58:14.376535+0000","last_active":"2024-03-26T03:58:25.016708+0000","last_peered":"2024-03-26T03:58:25.016708+0000","last_clean":"2024-03-26T03:58:25.016708+0000","last_became_active":"2024-03-26T03:58:14.344585+0000","last_became_peered":"2024-03-26T03:58:14.344585+0000","last_unstale":"2024-03-26T03:58:25.016708+0000","last_undegraded":"2024-03-26T03:58:25.016708+0000","last_fullsized":"2024-03-26T03:58:25.016708+0000","mapping_epoch":226,"log_start":"197'9400","ondisk_log_start":"197'9400","created":155,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130088960,"num_objects":1985,"num_object_clones":0,"num_object_copies":5955,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1985,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14109,"num_write_kb":515008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5624,"num_bytes_recovered":368574464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":357,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335907+0000","last_change":"2024-03-26T03:57:49.360270+0000","last_active":"2024-03-26T03:58:19.335907+0000","last_peered":"2024-03-26T03:58:19.335907+0000","last_clean":"2024-03-26T03:58:19.335907+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T03:58:19.335907+0000","last_undegraded":"2024-03-26T03:58:19.335907+0000","last_fullsized":"2024-03-26T03:58:19.335907+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.405517+0000","last_change":"2024-03-26T03:57:49.362068+0000","last_active":"2024-03-26T03:58:19.405517+0000","last_peered":"2024-03-26T03:58:19.405517+0000","last_clean":"2024-03-26T03:58:19.405517+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T03:58:19.405517+0000","last_undegraded":"2024-03-26T03:58:19.405517+0000","last_fullsized":"2024-03-26T03:58:19.405517+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"85'40","reported_seq":61,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402726+0000","last_change":"2024-03-26T03:57:49.531398+0000","last_active":"2024-03-26T03:58:19.402726+0000","last_peered":"2024-03-26T03:58:19.402726+0000","last_clean":"2024-03-26T03:58:19.402726+0000","last_became_active":"2024-03-26T03:57:49.398824+0000","last_became_peered":"2024-03-26T03:57:49.398824+0000","last_unstale":"2024-03-26T03:58:19.402726+0000","last_undegraded":"2024-03-26T03:58:19.402726+0000","last_fullsized":"2024-03-26T03:58:19.402726+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":369,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335617+0000","last_change":"2024-03-26T03:57:49.360193+0000","last_active":"2024-03-26T03:58:19.335617+0000","last_peered":"2024-03-26T03:58:19.335617+0000","last_clean":"2024-03-26T03:58:19.335617+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T03:58:19.335617+0000","last_undegraded":"2024-03-26T03:58:19.335617+0000","last_fullsized":"2024-03-26T03:58:19.335617+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"229'14540","reported_seq":21237,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.189901+0000","last_change":"2024-03-26T03:57:52.385514+0000","last_active":"2024-03-26T03:58:25.189901+0000","last_peered":"2024-03-26T03:58:25.189901+0000","last_clean":"2024-03-26T03:58:25.189901+0000","last_became_active":"2024-03-26T03:57:52.355530+0000","last_became_peered":"2024-03-26T03:57:52.355530+0000","last_unstale":"2024-03-26T03:58:25.189901+0000","last_undegraded":"2024-03-26T03:58:25.189901+0000","last_fullsized":"2024-03-26T03:58:25.189901+0000","mapping_epoch":217,"log_start":"198'10300","ondisk_log_start":"198'10300","created":155,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4240,"log_dups_size":0,"ondisk_log_size":4240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134873088,"num_objects":2058,"num_object_clones":0,"num_object_copies":6174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14540,"num_write_kb":531136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":213712896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":282,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339677+0000","last_change":"2024-03-26T03:55:59.341468+0000","last_active":"2024-03-26T03:58:19.339677+0000","last_peered":"2024-03-26T03:58:19.339677+0000","last_clean":"2024-03-26T03:58:19.339677+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T03:58:19.339677+0000","last_undegraded":"2024-03-26T03:58:19.339677+0000","last_fullsized":"2024-03-26T03:58:19.339677+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":493,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336830+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T03:58:19.336830+0000","last_peered":"2024-03-26T03:58:19.336830+0000","last_clean":"2024-03-26T03:58:19.336830+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T03:58:19.336830+0000","last_undegraded":"2024-03-26T03:58:19.336830+0000","last_fullsized":"2024-03-26T03:58:19.336830+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402876+0000","last_change":"2024-03-26T03:57:49.373904+0000","last_active":"2024-03-26T03:58:19.402876+0000","last_peered":"2024-03-26T03:58:19.402876+0000","last_clean":"2024-03-26T03:58:19.402876+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T03:58:19.402876+0000","last_undegraded":"2024-03-26T03:58:19.402876+0000","last_fullsized":"2024-03-26T03:58:19.402876+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":429,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.404290+0000","last_change":"2024-03-26T03:57:49.438962+0000","last_active":"2024-03-26T03:58:19.404290+0000","last_peered":"2024-03-26T03:58:19.404290+0000","last_clean":"2024-03-26T03:58:19.404290+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T03:58:19.404290+0000","last_undegraded":"2024-03-26T03:58:19.404290+0000","last_fullsized":"2024-03-26T03:58:19.404290+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:24.835587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.835587+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2","version":"229'14061","reported_seq":20590,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:24.315722+0000","last_change":"2024-03-26T03:56:40.032243+0000","last_active":"2024-03-26T03:58:24.315722+0000","last_peered":"2024-03-26T03:58:24.315722+0000","last_clean":"2024-03-26T03:58:24.315722+0000","last_became_active":"2024-03-26T03:56:40.023931+0000","last_became_peered":"2024-03-26T03:56:40.023931+0000","last_unstale":"2024-03-26T03:58:24.315722+0000","last_undegraded":"2024-03-26T03:58:24.315722+0000","last_fullsized":"2024-03-26T03:58:24.315722+0000","mapping_epoch":196,"log_start":"198'9700","ondisk_log_start":"198'9700","created":155,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4361,"log_dups_size":0,"ondisk_log_size":4361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127860736,"num_objects":1951,"num_object_clones":0,"num_object_copies":5853,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1951,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14061,"num_write_kb":512384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3178,"num_bytes_recovered":208273408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.401740+0000","last_change":"2024-03-26T03:57:49.373101+0000","last_active":"2024-03-26T03:58:19.401740+0000","last_peered":"2024-03-26T03:58:19.401740+0000","last_clean":"2024-03-26T03:58:19.401740+0000","last_became_active":"2024-03-26T03:57:49.372744+0000","last_became_peered":"2024-03-26T03:57:49.372744+0000","last_unstale":"2024-03-26T03:58:19.401740+0000","last_undegraded":"2024-03-26T03:58:19.401740+0000","last_fullsized":"2024-03-26T03:58:19.401740+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":324,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.339837+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T03:58:19.339837+0000","last_peered":"2024-03-26T03:58:19.339837+0000","last_clean":"2024-03-26T03:58:19.339837+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T03:58:19.339837+0000","last_undegraded":"2024-03-26T03:58:19.339837+0000","last_fullsized":"2024-03-26T03:58:19.339837+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":365,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.335642+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:58:19.335642+0000","last_peered":"2024-03-26T03:58:19.335642+0000","last_clean":"2024-03-26T03:58:19.335642+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:58:19.335642+0000","last_undegraded":"2024-03-26T03:58:19.335642+0000","last_fullsized":"2024-03-26T03:58:19.335642+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"73'17","reported_seq":461,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336181+0000","last_change":"2024-03-26T03:57:49.495410+0000","last_active":"2024-03-26T03:58:19.336181+0000","last_peered":"2024-03-26T03:58:19.336181+0000","last_clean":"2024-03-26T03:58:19.336181+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T03:58:19.336181+0000","last_undegraded":"2024-03-26T03:58:19.336181+0000","last_fullsized":"2024-03-26T03:58:19.336181+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"229'14436","reported_seq":14597,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:25.179555+0000","last_change":"2024-03-26T03:54:38.382794+0000","last_active":"2024-03-26T03:58:25.179555+0000","last_peered":"2024-03-26T03:58:25.179555+0000","last_clean":"2024-03-26T03:58:25.179555+0000","last_became_active":"2024-03-26T03:54:38.382452+0000","last_became_peered":"2024-03-26T03:54:38.382452+0000","last_unstale":"2024-03-26T03:58:25.179555+0000","last_undegraded":"2024-03-26T03:58:25.179555+0000","last_fullsized":"2024-03-26T03:58:25.179555+0000","mapping_epoch":155,"log_start":"198'10200","ondisk_log_start":"198'10200","created":155,"last_epoch_clean":156,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4236,"log_dups_size":0,"ondisk_log_size":4236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133431296,"num_objects":2036,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14436,"num_write_kb":527104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":381,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402265+0000","last_change":"2024-03-26T03:57:49.361380+0000","last_active":"2024-03-26T03:58:19.402265+0000","last_peered":"2024-03-26T03:58:19.402265+0000","last_clean":"2024-03-26T03:58:19.402265+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T03:58:19.402265+0000","last_undegraded":"2024-03-26T03:58:19.402265+0000","last_fullsized":"2024-03-26T03:58:19.402265+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:55.887078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:55.887078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":356,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338366+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:58:19.338366+0000","last_peered":"2024-03-26T03:58:19.338366+0000","last_clean":"2024-03-26T03:58:19.338366+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:58:19.338366+0000","last_undegraded":"2024-03-26T03:58:19.338366+0000","last_fullsized":"2024-03-26T03:58:19.338366+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":316,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336636+0000","last_change":"2024-03-26T03:57:49.360677+0000","last_active":"2024-03-26T03:58:19.336636+0000","last_peered":"2024-03-26T03:58:19.336636+0000","last_clean":"2024-03-26T03:58:19.336636+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T03:58:19.336636+0000","last_undegraded":"2024-03-26T03:58:19.336636+0000","last_fullsized":"2024-03-26T03:58:19.336636+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":389,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336869+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:58:19.336869+0000","last_peered":"2024-03-26T03:58:19.336869+0000","last_clean":"2024-03-26T03:58:19.336869+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:58:19.336869+0000","last_undegraded":"2024-03-26T03:58:19.336869+0000","last_fullsized":"2024-03-26T03:58:19.336869+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"229'14211","reported_seq":25521,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:24.682238+0000","last_change":"2024-03-26T03:58:19.434464+0000","last_active":"2024-03-26T03:58:24.682238+0000","last_peered":"2024-03-26T03:58:24.682238+0000","last_clean":"2024-03-26T03:58:24.682238+0000","last_became_active":"2024-03-26T03:58:19.406819+0000","last_became_peered":"2024-03-26T03:58:19.406819+0000","last_unstale":"2024-03-26T03:58:24.682238+0000","last_undegraded":"2024-03-26T03:58:24.682238+0000","last_fullsized":"2024-03-26T03:58:24.682238+0000","mapping_epoch":228,"log_start":"198'9900","ondisk_log_start":"198'9900","created":155,"last_epoch_clean":229,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:37.359520+0000","last_clean_scrub_stamp":"2024-03-26T03:54:37.359520+0000","objects_scrubbed":0,"log_size":4311,"log_dups_size":0,"ondisk_log_size":4311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131399680,"num_objects":2005,"num_object_clones":0,"num_object_copies":6015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14211,"num_write_kb":518912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5564,"num_bytes_recovered":252968960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":353,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336892+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:58:19.336892+0000","last_peered":"2024-03-26T03:58:19.336892+0000","last_clean":"2024-03-26T03:58:19.336892+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:58:19.336892+0000","last_undegraded":"2024-03-26T03:58:19.336892+0000","last_fullsized":"2024-03-26T03:58:19.336892+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":501,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.338605+0000","last_change":"2024-03-26T03:55:59.796875+0000","last_active":"2024-03-26T03:58:19.338605+0000","last_peered":"2024-03-26T03:58:19.338605+0000","last_clean":"2024-03-26T03:58:19.338605+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T03:58:19.338605+0000","last_undegraded":"2024-03-26T03:58:19.338605+0000","last_fullsized":"2024-03-26T03:58:19.338605+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:55:59.796779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:59.796779+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":39,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.402042+0000","last_change":"2024-03-26T03:57:49.360595+0000","last_active":"2024-03-26T03:58:19.402042+0000","last_peered":"2024-03-26T03:58:19.402042+0000","last_clean":"2024-03-26T03:58:19.402042+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T03:58:19.402042+0000","last_undegraded":"2024-03-26T03:58:19.402042+0000","last_fullsized":"2024-03-26T03:58:19.402042+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":430,"reported_epoch":229,"state":"active+clean","last_fresh":"2024-03-26T03:58:19.336767+0000","last_change":"2024-03-26T03:57:49.507032+0000","last_active":"2024-03-26T03:58:19.336767+0000","last_peered":"2024-03-26T03:58:19.336767+0000","last_clean":"2024-03-26T03:58:19.336767+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T03:58:19.336767+0000","last_undegraded":"2024-03-26T03:58:19.336767+0000","last_fullsized":"2024-03-26T03:58:19.336767+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2117992472,"num_objects":32319,"num_object_clones":0,"num_object_copies":96957,"num_objects_missing_on_primary":2537,"num_objects_missing":2537,"num_objects_degraded":887,"num_objects_misplaced":4436,"num_objects_unfound":0,"num_objects_dirty":32319,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":227539,"num_write_kb":8315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36616,"num_bytes_recovered":2241069056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6717648896,"data_stored":6717636680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71935,"ondisk_log_size":71935,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":572,"num_read_kb":567,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":979,"num_read_kb":704,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070584,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876724,"kb_used_data":647020,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92868428,"statfs":{"total":95995035648,"available":95097270272,"internally_reserved":0,"allocated":662548480,"data_stored":662066827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574999,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1244040,"kb_used_data":919872,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92501112,"statfs":{"total":95995035648,"available":94721138688,"internally_reserved":0,"allocated":941948928,"data_stored":941445924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738524,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1370304,"kb_used_data":1032312,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92374848,"statfs":{"total":95995035648,"available":94591844352,"internally_reserved":0,"allocated":1057087488,"data_stored":1056593645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411474,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1695636,"kb_used_data":1313228,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92049516,"statfs":{"total":95995035648,"available":94258704384,"internally_reserved":0,"allocated":1344745472,"data_stored":1344293869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182540,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":799104,"kb_used_data":529144,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92946048,"statfs":{"total":95995035648,"available":95176753152,"internally_reserved":0,"allocated":541843456,"data_stored":541348993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":276429255},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953607,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1523952,"kb_used_data":1177640,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92221200,"statfs":{"total":95995035648,"available":94434508800,"internally_reserved":0,"allocated":1205903360,"data_stored":1205423551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113310,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266724,"kb_used_data":998556,"kb_used_omap":4,"kb_used_meta":268155,"kb_avail":92478428,"statfs":{"total":95995035648,"available":94697910272,"internally_reserved":0,"allocated":1022521344,"data_stored":1022053885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":274591392},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1044119552,"data_stored":1044119552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":926941184,"data_stored":926941184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1325469696,"data_stored":1325465624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":552730624,"data_stored":552730624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1188888576,"data_stored":1188888576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1020923904,"data_stored":1020919832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":658575360,"data_stored":658571288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:58:27.497 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-26T03:58:27.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:27.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:27.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:27.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.698 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:27.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:27.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:27.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:27.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:27.949 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:27.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:28.050 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:28.053 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:28.056 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.090 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:28.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:28.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:28.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:28.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:28.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:28.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:28.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:28.433 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:58:28.450 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[29446]: pgmap v426: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 447 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 21 MiB/s, 330 objects/s recovering 2024-03-26T03:58:28.450 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3422544259' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.450 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.450 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[37501]: pgmap v426: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 447 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 21 MiB/s, 330 objects/s recovering 2024-03-26T03:58:28.450 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3422544259' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.451 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:28 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:28.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:28.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:28.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:28.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:28.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:28.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:28.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:28 smithi086 ceph-mon[32965]: pgmap v426: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 2.0 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 447 op/s; 887/97647 objects degraded (0.908%); 4436/97647 objects misplaced (4.543%); 21 MiB/s, 330 objects/s recovering 2024-03-26T03:58:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3422544259' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:28 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:28.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:28.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:28.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:28.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:28.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:29.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:29.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:29.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:29.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:29.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:29.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:29.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:29.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:29.508 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:29.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:29.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: osdmap e230: 8 total, 8 up, 7 in 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 887/97647 objects degraded (0.908%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[29446]: osdmap e231: 8 total, 8 up, 7 in 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T03:58:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: osdmap e230: 8 total, 8 up, 7 in 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 887/97647 objects degraded (0.908%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:29.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:29 smithi006 ceph-mon[37501]: osdmap e231: 8 total, 8 up, 7 in 2024-03-26T03:58:29.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:29.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T03:58:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: osdmap e230: 8 total, 8 up, 7 in 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 887/97647 objects degraded (0.908%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:29 smithi086 ceph-mon[32965]: osdmap e231: 8 total, 8 up, 7 in 2024-03-26T03:58:29.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:29.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:29.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:30.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:30.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:30.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:30.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:30.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:30.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:30.396 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:30.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:30.537 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:30.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:30.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:30.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:30.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T03:58:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T03:58:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: pgmap v429: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 398 op/s; 726/95619 objects degraded (0.759%); 4112/95619 objects misplaced (4.300%); 17 MiB/s, 264 objects/s recovering 2024-03-26T03:58:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T03:58:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T03:58:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:30 smithi086 ceph-mon[32965]: osdmap e232: 8 total, 8 up, 7 in 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: pgmap v429: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 398 op/s; 726/95619 objects degraded (0.759%); 4112/95619 objects misplaced (4.300%); 17 MiB/s, 264 objects/s recovering 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[37501]: osdmap e232: 8 total, 8 up, 7 in 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T03:58:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T03:58:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: pgmap v429: 148 pgs: 2 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering+degraded+remapped, 144 active+clean; 1.9 GiB data, 8.2 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 398 op/s; 726/95619 objects degraded (0.759%); 4112/95619 objects misplaced (4.300%); 17 MiB/s, 264 objects/s recovering 2024-03-26T03:58:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T03:58:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T03:58:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:30 smithi006 ceph-mon[29446]: osdmap e232: 8 total, 8 up, 7 in 2024-03-26T03:58:30.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:30.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:30.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:31.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:31.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:31.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:31.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:31.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:31.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:31.347 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:31.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:31.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:31.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:31.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:31.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:31 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T03:58:31.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:31 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T03:58:31.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:31 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T03:58:31.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:31 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T03:58:31.645 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:31.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:31.781 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:31.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:31 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T03:58:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:31 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T03:58:31.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:31.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:31.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:32.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:32.055 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:32.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:32.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:32.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:32.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:32.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:32.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:32.474 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:32.498 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:58:32.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:58:32.499 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-26T03:58:32.499 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.758657 2024-03-26T03:58:32.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 7 0.7586567763386856 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[37501]: pgmap v431: 148 pgs: 2 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 145 active+clean; 1.8 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 379 op/s; 726/90825 objects degraded (0.799%); 2672/90825 objects misplaced (2.942%); 16 MiB/s, 261 objects/s recovering 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[29446]: pgmap v431: 148 pgs: 2 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 145 active+clean; 1.8 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 379 op/s; 726/90825 objects degraded (0.799%); 2672/90825 objects misplaced (2.942%); 16 MiB/s, 261 objects/s recovering 2024-03-26T03:58:32.514 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:32 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:58:32.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:32.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:32.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:32.718 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:58:32.743 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:32.756 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:58:32.799 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:58:32.834 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:58:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:32 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T03:58:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:32 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T03:58:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:32 smithi086 ceph-mon[32965]: pgmap v431: 148 pgs: 2 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 145 active+clean; 1.8 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 379 op/s; 726/90825 objects degraded (0.799%); 2672/90825 objects misplaced (2.942%); 16 MiB/s, 261 objects/s recovering 2024-03-26T03:58:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:32 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:58:32.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:32.874 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:58:32.887 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:32.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:32.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:32.914 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:58:32.954 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:58:32.984 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:58:32.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:33.059 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:58:33.059 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:58:33.059 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:58:33.059 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:58:33.059 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:58:33.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:33.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:33.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:33.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:33.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:33.405 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:33.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:33.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:33.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:33.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:33.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:33.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:33 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:33 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:33 smithi086 ceph-mon[32965]: osdmap e233: 8 total, 8 up, 7 in 2024-03-26T03:58:33.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/366689161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[37501]: osdmap e233: 8 total, 8 up, 7 in 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/366689161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[29446]: osdmap e233: 8 total, 8 up, 7 in 2024-03-26T03:58:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/366689161' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:33.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:33.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:33.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:33.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:33.946 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:33.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T03:58:34.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:58:34.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:34.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:34.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:34.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:34.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:34.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:34.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:34.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:34.465 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:34.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:34.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:34.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T03:58:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T03:58:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[37501]: pgmap v433: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 537/88296 objects degraded (0.608%); 2108/88296 objects misplaced (2.387%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:58:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 537/88296 objects degraded (0.608%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T03:58:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T03:58:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[29446]: pgmap v433: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 537/88296 objects degraded (0.608%); 2108/88296 objects misplaced (2.387%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:58:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 537/88296 objects degraded (0.608%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:34.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:34.728 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:34.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:34 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T03:58:34.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:34 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T03:58:34.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:34 smithi086 ceph-mon[32965]: pgmap v433: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.8 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 492 op/s; 537/88296 objects degraded (0.608%); 2108/88296 objects misplaced (2.387%); 23 MiB/s, 371 objects/s recovering 2024-03-26T03:58:34.737 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 537/88296 objects degraded (0.608%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:34.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:34.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:34.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:34.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:34.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:34.992 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:35.072 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:35.073 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:35.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:35.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:35.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:35.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:35.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:35.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:35.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:35.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:35.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:35.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:35.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.508 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:35.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:35.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:35.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.647 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:35.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:35.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:35.779 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:35.780 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.838 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:35.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:35.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:35.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:35.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:36.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:36.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:36.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:36.219 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:36.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:36.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:36.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:36.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:58:36.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:36.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:36.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:36.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:36.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:36.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:36 smithi006 ceph-mon[37501]: pgmap v434: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 917 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 31 MiB/s, 503 objects/s recovering 2024-03-26T03:58:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:36 smithi006 ceph-mon[29446]: pgmap v434: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 917 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 31 MiB/s, 503 objects/s recovering 2024-03-26T03:58:36.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:36.743 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:36.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:36 smithi086 ceph-mon[32965]: pgmap v434: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 917 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 31 MiB/s, 503 objects/s recovering 2024-03-26T03:58:36.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:36.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:36.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:36.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:36.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:36.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:36.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:37.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:37.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:37.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:37.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:37.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:37.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:37.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:37.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:37.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:37.479 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:37.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:37.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:37.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:37.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:37.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:37.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:37.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:37.817 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:37 smithi006 ceph-mon[37501]: osdmap e234: 8 total, 8 up, 7 in 2024-03-26T03:58:37.817 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:37 smithi006 ceph-mon[29446]: osdmap e234: 8 total, 8 up, 7 in 2024-03-26T03:58:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:37 smithi086 ceph-mon[32965]: osdmap e234: 8 total, 8 up, 7 in 2024-03-26T03:58:37.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:37.952 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:37.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:38.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:38.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:38.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:38.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:38.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:38.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.366 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:38.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:38.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:38.513 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:38.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:38.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:38.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:38.769 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T03:58:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T03:58:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: pgmap v436: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 561 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 16 MiB/s, 249 objects/s recovering 2024-03-26T03:58:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: 5.19 deep-scrub starts 2024-03-26T03:58:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: 5.19 deep-scrub ok 2024-03-26T03:58:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:38 smithi086 ceph-mon[32965]: osdmap e235: 8 total, 8 up, 7 in 2024-03-26T03:58:38.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: pgmap v436: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 561 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 16 MiB/s, 249 objects/s recovering 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: 5.19 deep-scrub starts 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: 5.19 deep-scrub ok 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[37501]: osdmap e235: 8 total, 8 up, 7 in 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: pgmap v436: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.6 GiB data, 7.2 GiB used, 619 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 561 op/s; 537/78993 objects degraded (0.680%); 744/78993 objects misplaced (0.942%); 16 MiB/s, 249 objects/s recovering 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: 5.19 deep-scrub starts 2024-03-26T03:58:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: 5.19 deep-scrub ok 2024-03-26T03:58:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:38 smithi006 ceph-mon[29446]: osdmap e235: 8 total, 8 up, 7 in 2024-03-26T03:58:38.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:38.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:38.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:38.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:39.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:39.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:39.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:39.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:39.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:39.430 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:39.496 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.7 primary-affinity to 0.758657 (8497192) 2024-03-26T03:58:39.508 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]: dispatch 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 537/78993 objects degraded (0.680%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]: dispatch 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:39.509 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 537/78993 objects degraded (0.680%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:39.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:39.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:39.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:39.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:39.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:39.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T03:58:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T03:58:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]: dispatch 2024-03-26T03:58:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 537/78993 objects degraded (0.680%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:39.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:39.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:39.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:40.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:40.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:40.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:40.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179824 8.d 8:ba5e2828:::benchmark_data_smithi006_103169_object49893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.757884+0000", 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.926684442999999, 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18686103100000001, 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.282 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179824 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757884+0000", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757884+0000", 2024-03-26T03:58:40.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757885+0000", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3969999999999999e-06 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757891+0000", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757903+0000", 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1813e-05 2024-03-26T03:58:40.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757910+0000", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7330000000000003e-06 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764628+0000", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067178130000000004 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764765+0000", 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013682399999999999 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764906+0000", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000140879 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773926+0000", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090205979999999995 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773974+0000", 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7148999999999999e-05 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.286 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.777029+0000", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0030555439999999999 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.777123+0000", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3794000000000004e-05 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944745+0000", 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16762194899999999 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179841 8.d 8:b003b8a9:::benchmark_data_smithi006_103169_object49910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.772355+0000", 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.912212866000004, 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17670936200000001, 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179841 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772355+0000", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772355+0000", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772357+0000", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.133e-06 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772361+0000", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8389999999999997e-06 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772371+0000", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3880000000000006e-06 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772378+0000", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3629999999999996e-06 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774017+0000", 2024-03-26T03:58:40.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016384349999999999 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774142+0000", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012499599999999999 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774277+0000", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013563299999999999 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948148+0000", 2024-03-26T03:58:40.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17387116 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948164+0000", 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5752000000000001e-05 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948200+0000", 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5728000000000001e-05 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948305+0000", 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105346 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.949065+0000", 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00075958899999999997 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179843 8.d 8:b91817d9:::benchmark_data_smithi006_103169_object49912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.772830+0000", 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.911738017000005, 2024-03-26T03:58:40.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18273748100000001, 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179843 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772830+0000", 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772830+0000", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772831+0000", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0500000000000002e-07 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772836+0000", 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5460000000000002e-06 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772845+0000", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7270000000000002e-06 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772852+0000", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1219999999999997e-06 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944785+0000", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.171933004 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944910+0000", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012439600000000001 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.945040+0000", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012977600000000001 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953269+0000", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082293650000000006 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953284+0000", 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4982e-05 2024-03-26T03:58:40.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953319+0000", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5049999999999998e-05 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953626+0000", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030677900000000003 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.955568+0000", 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001941929 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.298 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179839 8.6 e204/155 8:6fb0a9c5:::benchmark_data_smithi006_103169_object49908:head v 204'11031, mlcod=204'11028)", 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.773759+0000", 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.910809190999998, 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.173202892, 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.299 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.03e-07 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773764+0000", 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2799999999999997e-06 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.300 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773770+0000", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7120000000000003e-06 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773777+0000", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5910000000000001e-06 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776407+0000", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026304940000000001 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.301 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776428+0000", 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0061e-05 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.946907+0000", 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17047985500000001 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.946962+0000", 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4595999999999997e-05 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.302 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179845 8.e e204/155 8:759074b4:::benchmark_data_smithi006_103169_object49914:head v 204'11148, mlcod=204'11146)", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.776039+0000", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.908529388000005, 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18142430700000001, 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776039+0000", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.303 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776039+0000", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776040+0000", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7199999999999997e-07 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776046+0000", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3069999999999999e-06 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.304 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776055+0000", 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3300000000000005e-06 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776062+0000", 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9879999999999998e-06 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776187+0000", 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012491299999999999 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.305 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776200+0000", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2371000000000001e-05 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957379+0000", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18117967400000001 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957463+0000", 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3851999999999996e-05 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.306 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179840 8.6 e204/155 8:695aac83:::benchmark_data_smithi006_103169_object49909:head v 204'11032, mlcod=204'11028)", 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.776299+0000", 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.908269421, 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18122993900000001, 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9200000000000001e-07 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776305+0000", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3569999999999997e-06 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776313+0000", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2709999999999992e-06 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776321+0000", 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4109999999999999e-06 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943968+0000", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.167647087 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944005+0000", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6952000000000001e-05 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957469+0000", 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013464511 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957529+0000", 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9757999999999999e-05 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179849 8.e e204/155 8:76272e1c:::benchmark_data_smithi006_103169_object49918:head v 204'11149, mlcod=204'11146)", 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.779800+0000", 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.904768583000006, 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17778460700000001, 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.311 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9299999999999997e-07 2024-03-26T03:58:40.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779806+0000", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943759+0000", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16395269500000001 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943772+0000", 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3074000000000001e-05 2024-03-26T03:58:40.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.947163+0000", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033908800000000002 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.947181+0000", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8664999999999998e-05 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957531+0000", 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010350114000000001 2024-03-26T03:58:40.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957584+0000", 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2948000000000002e-05 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180185 8.f 8:fadf5a1b:::benchmark_data_smithi006_103169_object50254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.224359+0000", 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.460209065000001, 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19796323499999999, 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.315 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180185 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224359+0000", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224359+0000", 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224361+0000", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.813e-06 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224368+0000", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2470000000000002e-06 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224383+0000", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4498000000000001e-05 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224392+0000", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3219999999999999e-06 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224458+0000", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5902999999999996e-05 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224792+0000", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033372499999999999 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:58:40.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224918+0000", 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012586699999999999 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228128+0000", 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032105810000000001 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229340+0000", 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001211953 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.420194+0000", 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19085339700000001 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422132+0000", 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019380059999999999 2024-03-26T03:58:40.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422322+0000", 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019092299999999999 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180178 8.b e204/155 8:de4dc8ce:::benchmark_data_smithi006_103169_object50247:head v 204'11490, mlcod=204'11487)", 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.224599+0000", 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.459969025999996, 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19294374, 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.322 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224599+0000", 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224599+0000", 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224600+0000", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5799999999999998e-07 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224608+0000", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2930000000000001e-06 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224617+0000", 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0399999999999998e-06 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224625+0000", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3769999999999998e-06 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224750+0000", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012485799999999999 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224769+0000", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.897e-05 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417449+0000", 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19268063999999999 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417543+0000", 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3703999999999996e-05 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180180 8.b e204/155 8:d7b21ae2:::benchmark_data_smithi006_103169_object50249:head v 204'11491, mlcod=204'11487)", 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.225396+0000", 2024-03-26T03:58:40.326 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.459171901999994, 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19221780899999999, 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225396+0000", 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225396+0000", 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225397+0000", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1699999999999998e-07 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225402+0000", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3460000000000001e-06 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225410+0000", 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5270000000000001e-06 2024-03-26T03:58:40.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225416+0000", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8919999999999997e-06 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.226972+0000", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001556125 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.226991+0000", 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8947e-05 2024-03-26T03:58:40.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417551+0000", 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19056042300000001 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417614+0000", 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3032000000000002e-05 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDPGPushReply(8.d 204/181 [PushReplyOp(8:b7097072:::benchmark_data_smithi006_103169_object91940:head)])", 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227013+0000", 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.457555524, 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19041166800000001, 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.331 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.58e-07 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227017+0000", 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.087e-06 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227025+0000", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9410000000000007e-06 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227033+0000", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4279999999999999e-06 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231756+0000", 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047229999999999998 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417424+0000", 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18566885399999999 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180187 8.6 e204/155 8:624ff609:::benchmark_data_smithi006_103169_object50256:head v 204'11050, mlcod=204'11047)", 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227200+0000", 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.457368368000004, 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.197095462, 2024-03-26T03:58:40.334 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227200+0000", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227200+0000", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227201+0000", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5099999999999999e-07 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227207+0000", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3110000000000002e-06 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227214+0000", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4479999999999997e-06 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227221+0000", 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6159999999999998e-06 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229327+0000", 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002106101 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229346+0000", 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8715999999999998e-05 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.337 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.424258+0000", 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19491228199999999 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.424295+0000", 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7336999999999998e-05 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180182 8.b e204/155 8:dbfc3080:::benchmark_data_smithi006_103169_object50251:head v 204'11492, mlcod=204'11487)", 2024-03-26T03:58:40.338 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227311+0000", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.457257071000001, 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19036519299999999, 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227311+0000", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227311+0000", 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227312+0000", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5400000000000001e-07 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227317+0000", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.434e-06 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227325+0000", 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0299999999999994e-06 2024-03-26T03:58:40.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227331+0000", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1940000000000003e-06 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229191+0000", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018592000000000001 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229211+0000", 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0041e-05 2024-03-26T03:58:40.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417618+0000", 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18840707200000001 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417676+0000", 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8668000000000001e-05 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.342 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180189 8.d 8:ba34b65e:::benchmark_data_smithi006_103169_object50258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227480+0000", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.457088099999993, 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19577961699999999, 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180189 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227480+0000", 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227480+0000", 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227482+0000", 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5290000000000001e-06 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227488+0000", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9059999999999999e-06 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227497+0000", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.132e-06 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227504+0000", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5789999999999998e-06 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228870+0000", 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00136516 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229009+0000", 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013946799999999999 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229150+0000", 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014095899999999999 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422212+0000", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19306240099999999 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422258+0000", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5677000000000003e-05 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422312+0000", 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4200999999999998e-05 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422404+0000", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1336999999999996e-05 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.423260+0000", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00085626799999999998 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180192 8.b e204/155 8:d0d6cac1:::benchmark_data_smithi006_103169_object50261:head v 204'11493, mlcod=204'11488)", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229244+0000", 2024-03-26T03:58:40.348 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.455324114000007, 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18849374599999999, 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229244+0000", 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229244+0000", 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229245+0000", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.68e-07 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229250+0000", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4960000000000004e-06 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229258+0000", 2024-03-26T03:58:40.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4420000000000001e-06 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229264+0000", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6510000000000004e-06 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231808+0000", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025432200000000001 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231827+0000", 2024-03-26T03:58:40.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9038e-05 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417679+0000", 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.185852506 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417738+0000", 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8825e-05 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180188 8.e e204/155 8:76527e2e:::benchmark_data_smithi006_103169_object50257:head v 204'11169, mlcod=204'11167)", 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229725+0000", 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.454843351999997, 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20416622300000001, 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.44e-07 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229731+0000", 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3970000000000002e-06 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229739+0000", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9780000000000005e-06 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229745+0000", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3080000000000004e-06 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.230445+0000", 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069970999999999998 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.230464+0000", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8643000000000001e-05 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433736+0000", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20327236200000001 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433891+0000", 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015528100000000001 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180193 8.6 e204/155 8:64b9ce04:::benchmark_data_smithi006_103169_object50262:head v 204'11051, mlcod=204'11047)", 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229761+0000", 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.454807381999998, 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.204196553, 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7000000000000001e-07 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229765+0000", 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.472e-06 2024-03-26T03:58:40.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229770+0000", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9180000000000002e-06 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229774+0000", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5499999999999996e-06 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232272+0000", 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024981080000000002 2024-03-26T03:58:40.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232291+0000", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8461000000000002e-05 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433897+0000", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.201606272 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433957+0000", 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0501999999999998e-05 2024-03-26T03:58:40.360 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180194 8.f 8:fd5e33ac:::benchmark_data_smithi006_103169_object50263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.231221+0000", 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.453347198000003, 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19125515100000001, 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180194 2024-03-26T03:58:40.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231221+0000", 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231221+0000", 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231222+0000", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3960000000000001e-06 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231229+0000", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2829999999999998e-06 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231240+0000", 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1445e-05 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231248+0000", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0299999999999994e-06 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231306+0000", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7861e-05 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231423+0000", 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116883 2024-03-26T03:58:40.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231541+0000", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118492 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417331+0000", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18578977699999999 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417374+0000", 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2775999999999998e-05 2024-03-26T03:58:40.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422378+0000", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050036409999999996 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422433+0000", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5676e-05 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422476+0000", 2024-03-26T03:58:40.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2891e-05 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:180189 8.d e204/181 ondisk, result = 0)", 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.232503+0000", 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.452065337999997, 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18976958399999999, 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232503+0000", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232503+0000", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232504+0000", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0100000000000004e-07 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232508+0000", 2024-03-26T03:58:40.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0980000000000004e-06 2024-03-26T03:58:40.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417440+0000", 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18493225599999999 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417450+0000", 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0380999999999999e-05 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422242+0000", 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047916160000000003 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422251+0000", 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8219999999999996e-06 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422273+0000", 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1710000000000001e-05 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.371 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:180194 8.f e204/155 ondisk, result = 0)", 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.234408+0000", 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 86.450160054999998, 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18297554299999999, 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234408+0000", 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234408+0000", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234409+0000", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0500000000000003e-07 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234413+0000", 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8739999999999997e-06 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.416658+0000", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18224552899999999 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.416672+0000", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3723000000000001e-05 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417358+0000", 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068599199999999996 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417367+0000", 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4159999999999993e-06 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417384+0000", 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6303999999999998e-05 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:40.375 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:58:40.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:40.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:40.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:40.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:40.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:40.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: pgmap v438: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 315/76374 objects degraded (0.412%); 516/76374 objects misplaced (0.676%); 18 MiB/s, 286 objects/s recovering 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]': finished 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: osdmap e236: 8 total, 8 up, 7 in 2024-03-26T03:58:40.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[29446]: osdmap e237: 8 total, 8 up, 7 in 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: pgmap v438: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 315/76374 objects degraded (0.412%); 516/76374 objects misplaced (0.676%); 18 MiB/s, 286 objects/s recovering 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]': finished 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: osdmap e236: 8 total, 8 up, 7 in 2024-03-26T03:58:40.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:40 smithi006 ceph-mon[37501]: osdmap e237: 8 total, 8 up, 7 in 2024-03-26T03:58:40.668 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:40.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:40.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T03:58:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T03:58:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: pgmap v438: 148 pgs: 1 active+remapped+backfilling, 1 active+recovering+degraded+remapped, 146 active+clean; 1.5 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 696 op/s; 315/76374 objects degraded (0.412%); 516/76374 objects misplaced (0.676%); 18 MiB/s, 286 objects/s recovering 2024-03-26T03:58:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T03:58:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T03:58:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1447835022' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.7586567763386856}]': finished 2024-03-26T03:58:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: osdmap e236: 8 total, 8 up, 7 in 2024-03-26T03:58:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:40 smithi086 ceph-mon[32965]: osdmap e237: 8 total, 8 up, 7 in 2024-03-26T03:58:40.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:40.928 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:40.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:41.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:41.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:41.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:41.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:41.198 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:41.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:41.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:41.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:41.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:41.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:41.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T03:58:41.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:41.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:41.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:41.651 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:41 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T03:58:41.651 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:41 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T03:58:41.651 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:41 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T03:58:41.651 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:41 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T03:58:41.661 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:41.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:41.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.738 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:41.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:41.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:41 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T03:58:41.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:41 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T03:58:41.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:41.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:41.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:41.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:41.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:42.037 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:42.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:42.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.186 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:42.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:42.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:42.322 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:42.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:42.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:42.464 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:42.531 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:58:42.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:42.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:42.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:42.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:42.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T03:58:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T03:58:42.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: pgmap v441: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 315/68286 objects degraded (0.461%); 318/68286 objects misplaced (0.466%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:58:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T03:58:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T03:58:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:58:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1959555858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:42 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:42.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:42.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:42.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:42.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T03:58:42.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T03:58:42.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: pgmap v441: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 315/68286 objects degraded (0.461%); 318/68286 objects misplaced (0.466%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:58:42.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T03:58:42.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1959555858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: pgmap v441: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.4 GiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 315/68286 objects degraded (0.461%); 318/68286 objects misplaced (0.466%); 5.4 MiB/s, 86 objects/s recovering 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1959555858' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:42 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:58:42.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:43.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:43.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:43.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:43.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:43.309 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:43.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:43.450 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:43.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:43.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:43.631 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:43.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:43.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.757 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:43.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:58:43.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:43 smithi086 ceph-mon[32965]: osdmap e238: 8 total, 8 up, 7 in 2024-03-26T03:58:43.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:43.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[37501]: osdmap e238: 8 total, 8 up, 7 in 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:58:43.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:43 smithi006 ceph-mon[29446]: osdmap e238: 8 total, 8 up, 7 in 2024-03-26T03:58:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:43.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:43.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:44.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:44.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:44.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:44.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:44.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:44.332 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:44.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:44.443 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:44.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:44.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:44.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:44.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[29446]: pgmap v443: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.3 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 797 op/s; 14/64887 objects degraded (0.022%); 15/64887 objects misplaced (0.023%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 14/64887 objects degraded (0.022%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[29446]: osdmap e239: 8 total, 8 up, 7 in 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[37501]: pgmap v443: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.3 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 797 op/s; 14/64887 objects degraded (0.022%); 15/64887 objects misplaced (0.023%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 14/64887 objects degraded (0.022%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:44.592 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:44 smithi006 ceph-mon[37501]: osdmap e239: 8 total, 8 up, 7 in 2024-03-26T03:58:44.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:44.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:44.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:44 smithi086 ceph-mon[32965]: pgmap v443: 148 pgs: 2 peering, 1 active+recovering+degraded+remapped, 145 active+clean; 1.3 GiB data, 6.4 GiB used, 619 GiB / 626 GiB avail; 867 B/s rd, 0 B/s wr, 797 op/s; 14/64887 objects degraded (0.022%); 15/64887 objects misplaced (0.023%); 7.7 MiB/s, 123 objects/s recovering 2024-03-26T03:58:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 14/64887 objects degraded (0.022%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:44 smithi086 ceph-mon[32965]: osdmap e239: 8 total, 8 up, 7 in 2024-03-26T03:58:44.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:44.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:44.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:44.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:44.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:44.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:45.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:45.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:45.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:45.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:45.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:45.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:45.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:45.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:45.537 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:45.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:45.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:58:45.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:45.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:45.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:45.800 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:45.800 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:45.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:45.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:45.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:45.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:45.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:46.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:46.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:46.210 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:46.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:46.357 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:46.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:46.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:46.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:46.661 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:46.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:46.820 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:46 smithi086 ceph-mon[32965]: pgmap v445: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 1.24k op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 5.3 MiB/s, 84 objects/s recovering 2024-03-26T03:58:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:46 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:58:46.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:46 smithi006 ceph-mon[37501]: pgmap v445: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 1.24k op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 5.3 MiB/s, 84 objects/s recovering 2024-03-26T03:58:46.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:46 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:58:46.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:46 smithi006 ceph-mon[29446]: pgmap v445: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 868 B/s rd, 0 B/s wr, 1.24k op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 5.3 MiB/s, 84 objects/s recovering 2024-03-26T03:58:46.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:46 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-03-26T03:58:46.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:46.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:46.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:47.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:47.162 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:58:47.163 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:58:47.163 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T03:58:47.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:58:47.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:47.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.263 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:47.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:47.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:47.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:47.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:47.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:47.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:47.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:47.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:47.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:47.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.840 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:47.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:47.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:47.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:47.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:47.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:48.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:48.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:48.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:48.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:48.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:48.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:48.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:48.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.520 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:48.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:48.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:48.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:48.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[37501]: pgmap v446: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 737 op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 2.7 MiB/s, 43 objects/s recovering 2024-03-26T03:58:48.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[37501]: osdmap e240: 8 total, 8 up, 7 in 2024-03-26T03:58:48.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1422387485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:58:48.575 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:58:48.576 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:58:48.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:48.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:48.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:48.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:48.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:48 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:48 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:48 smithi086 ceph-mon[32965]: pgmap v446: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 737 op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 2.7 MiB/s, 43 objects/s recovering 2024-03-26T03:58:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:48 smithi086 ceph-mon[32965]: osdmap e240: 8 total, 8 up, 7 in 2024-03-26T03:58:48.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:48 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1422387485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:48.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:48.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:48.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[29446]: pgmap v446: 148 pgs: 1 active+recovering+degraded+remapped, 147 active+clean; 1.1 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 742 B/s rd, 0 B/s wr, 737 op/s; 14/53037 objects degraded (0.026%); 15/53037 objects misplaced (0.028%); 2.7 MiB/s, 43 objects/s recovering 2024-03-26T03:58:48.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[29446]: osdmap e240: 8 total, 8 up, 7 in 2024-03-26T03:58:48.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:48 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1422387485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:58:48.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:48.969 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:48.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:49.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:49.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:49.197 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:49.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.273 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:49.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:49.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:49.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:49.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:49.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:49.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:49.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:49.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:49.720 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:49.721 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:49 smithi006 ceph-mon[37501]: osdmap e241: 8 total, 8 up, 7 in 2024-03-26T03:58:49.721 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:49.721 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:49 smithi006 ceph-mon[29446]: osdmap e241: 8 total, 8 up, 7 in 2024-03-26T03:58:49.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:49.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:49.829 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T03:58:49.829 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:58:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded (PG_DEGRADED) 2024-03-26T03:58:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:49 smithi086 ceph-mon[32965]: osdmap e241: 8 total, 8 up, 7 in 2024-03-26T03:58:49.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:49.969 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:49.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:50.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:50.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:50.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:50.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:50.278 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:50.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:50.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:50.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:50.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:50.539 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:50.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:50.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:50.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:50.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.811 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:50.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:50 smithi086 ceph-mon[32965]: pgmap v449: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1006 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 887 op/s; 160 KiB/s, 136 objects/s recovering 2024-03-26T03:58:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:50 smithi086 ceph-mon[32965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded) 2024-03-26T03:58:50.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:50 smithi006 ceph-mon[29446]: pgmap v449: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1006 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 887 op/s; 160 KiB/s, 136 objects/s recovering 2024-03-26T03:58:50.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:50 smithi006 ceph-mon[29446]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded) 2024-03-26T03:58:50.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:50 smithi006 ceph-mon[37501]: pgmap v449: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 1006 MiB data, 5.4 GiB used, 620 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 887 op/s; 160 KiB/s, 136 objects/s recovering 2024-03-26T03:58:50.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:50 smithi006 ceph-mon[37501]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/53037 objects degraded (0.026%), 1 pg degraded) 2024-03-26T03:58:50.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:50.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:50.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:51.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:51.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:51.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:51.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:51.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:58:51.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:51.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:51.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:51.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:51.473 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:51.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:51.612 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:51.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:51.759 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:51.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:51.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:51.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:51.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:51.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:52.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:52.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:52.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:52.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:52.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:52.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:52.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:52.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T03:58:52.472 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:52.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:52.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:52.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:52.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:52.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:52 smithi006 ceph-mon[29446]: pgmap v450: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 747 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 1.34k op/s; 136 KiB/s, 116 objects/s recovering 2024-03-26T03:58:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:52 smithi006 ceph-mon[37501]: pgmap v450: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 747 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 1.34k op/s; 136 KiB/s, 116 objects/s recovering 2024-03-26T03:58:52.704 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:52.753 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:52 smithi086 ceph-mon[32965]: pgmap v450: 148 pgs: 1 active+recovering+remapped, 147 active+clean; 747 MiB data, 4.7 GiB used, 621 GiB / 626 GiB avail; 727 B/s rd, 0 B/s wr, 1.34k op/s; 136 KiB/s, 116 objects/s recovering 2024-03-26T03:58:52.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:52.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:52.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:53.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:53.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:53.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:53.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:53.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:53.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:53.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:53.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:53.447 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:53.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:53.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:53.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:53.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:53.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:53.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[29446]: osdmap e242: 8 total, 8 up, 7 in 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:53 smithi006 ceph-mon[37501]: osdmap e242: 8 total, 8 up, 7 in 2024-03-26T03:58:53.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:53.988 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:53.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:53.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:53 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:58:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:53 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:58:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:53 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T03:58:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:53 smithi086 ceph-mon[32965]: osdmap e242: 8 total, 8 up, 7 in 2024-03-26T03:58:54.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:54.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:54.271 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:54.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:54.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:54.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:54.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:54.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:54.622 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:58:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:58:54.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:54.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:54.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:54.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:54.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[29446]: pgmap v451: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 654 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 881 op/s; 451/32061 objects misplaced (1.407%); 120 KiB/s, 208 objects/s recovering 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[29446]: osdmap e243: 8 total, 8 up, 7 in 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[37501]: pgmap v451: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 654 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 881 op/s; 451/32061 objects misplaced (1.407%); 120 KiB/s, 208 objects/s recovering 2024-03-26T03:58:54.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:54.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:54.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:54 smithi006 ceph-mon[37501]: osdmap e243: 8 total, 8 up, 7 in 2024-03-26T03:58:54.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:54.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:54.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:55.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:55.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:55.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:58:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:58:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:54 smithi086 ceph-mon[32965]: pgmap v451: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 654 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 4.7 KiB/s rd, 0 B/s wr, 881 op/s; 451/32061 objects misplaced (1.407%); 120 KiB/s, 208 objects/s recovering 2024-03-26T03:58:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:58:55.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:54 smithi086 ceph-mon[32965]: osdmap e243: 8 total, 8 up, 7 in 2024-03-26T03:58:55.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:55.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:55.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:55.365 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:55.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:55.467 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:55.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:55.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:55.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:55.653 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:58:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.663 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:55.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:55.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.811 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:55.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:58:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3872411481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:55.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:58:55.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3872411481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:55.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:55 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:55.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:58:55.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:55.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:58:56.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.087 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:56.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:58:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:58:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3872411481' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:55 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:58:56.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:56.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:56.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:56.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:56.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:56.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:56.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:56.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:56.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:56.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:56.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:56.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:56.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:56.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:56.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:56.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[37501]: pgmap v454: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.76k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 139 objects/s recovering 2024-03-26T03:58:56.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[37501]: osdmap e244: 8 total, 8 up, 7 in 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[29446]: pgmap v454: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.76k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 139 objects/s recovering 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:56 smithi006 ceph-mon[29446]: osdmap e244: 8 total, 8 up, 7 in 2024-03-26T03:58:56.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:57.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:57.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:56 smithi086 ceph-mon[32965]: pgmap v454: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.76k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 139 objects/s recovering 2024-03-26T03:58:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:56 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:58:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:56 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:58:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:56 smithi086 ceph-mon[32965]: osdmap e244: 8 total, 8 up, 7 in 2024-03-26T03:58:57.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:57.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.180 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:57.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:57.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:57.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:57.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:57.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.472 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:57.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:57.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:57.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:57.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:57.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:57.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:58:57.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:58:57.728 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:57.728 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:58:57.728 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:58:57.729 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:57 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:57.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:57.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:57.859 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:58:57.860 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:57.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:57.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:58.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:57 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:58:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:57 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:58:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:57 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:58:58.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:58.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:58.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:58.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:58.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:58.338 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 244, 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-03-26T03:43:14.534401+0000", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-03-26T03:58:55.623388+0000", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-03-26T03:55:58.386605+0000", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-03-26T03:57:48.463474+0000", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:58:58.339 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5806080, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "nodeep-scrub", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "noscrub", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 19, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 8, 2024-03-26T03:58:58.340 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 8, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "luminous", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:46:30.183658+0000", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.341 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 1, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 1, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 1, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 1, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 1, 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.342 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "165", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.343 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.344 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.345 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 7.8899998664855957, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 7.8899998664855957, 2024-03-26T03:58:58.346 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 3, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 3, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "datapool", 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:41.622497+0000", 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:58:58.347 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 3, 2024-03-26T03:58:58.348 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.349 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 69, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.350 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.351 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.75, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.352 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".rgw.root", 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:42.191818+0000", 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.353 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.354 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.355 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.357 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "67", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.358 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.359 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:58:58.360 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.361 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 4, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:44.079106+0000", 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.362 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:58:58.363 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.364 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.365 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.366 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 5, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:46.035261+0000", 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.367 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.368 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "71", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.369 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.370 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.371 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 6, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:48.086810+0000", 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.373 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.374 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "73", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.375 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.376 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_bias": 4 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.377 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 8, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_1", 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:54:37.148544+0000", 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:58:58.379 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 16, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 16, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 16, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 16, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 16, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "210", 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:58:58.380 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:58:58.381 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:58:58.382 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "rados": {} 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:58:58.383 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.75, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.75, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e", 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.384 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 8, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 226, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.385 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.386 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.387 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/1246039834", 2024-03-26T03:58:58.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:58.388 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6802/1246039834", 2024-03-26T03:58:58.388 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6804/1246039834", 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6803/1246039834", 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "753f7459-62ec-447b-a0da-64d06e813f16", 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.389 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 13, 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 234, 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.391 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6805/260311411", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6806/260311411", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6808/260311411", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6807/260311411", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b7d7409f-341a-4711-8570-92fe9082a528", 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 18, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 236, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.393 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:58:58.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.395 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/3103276794", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6810/3103276794", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6812/3103276794", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6811/3103276794", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.396 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6", 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 25, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 230, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-03-26T03:58:58.397 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.398 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6813/2795883342", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6814/2795883342", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6816/2795883342", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6815/2795883342", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8", 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.400 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 32, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 236, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6800", 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.401 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6801", 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.402 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6803", 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6802", 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:58:58.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6800/931538411", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6801/931538411", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6803/931538411", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6802/931538411", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc", 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.404 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 41, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 167, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 180, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 236, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 168, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6804", 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6805", 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.406 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6807", 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6806", 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6804/909197223", 2024-03-26T03:58:58.407 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6805/909197223", 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6807/909197223", 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6806/909197223", 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "58b58555-59a3-4620-a15f-024f7041713f", 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.408 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 50, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 95, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6808", 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6809", 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6811", 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6810", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6808/1580133777", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6809/1580133777", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6811/1580133777", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6810/1580133777", 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.411 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7", 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 0.7586517333984375, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 59, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 214, 2024-03-26T03:58:58.412 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6812", 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.413 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6813", 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6815", 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6814", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6812/3905426306", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6813/3905426306", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6815/3905426306", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6814/3905426306", 2024-03-26T03:58:58.415 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:00.485830+0000", 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:57:40.315263+0000", 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.417 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:05.377743+0000", 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:57:47.388775+0000", 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.418 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:47.161748+0000", 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.419 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:32.173892+0000", 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 167 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:02.396934+0000", 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:53.055285+0000", 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [], 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [ 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "3.1f", 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:58:58.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 0, 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.b", 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 5 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.17", 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.1f", 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:58:58.423 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [ 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "8.9", 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: 5, 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: 0, 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: 4 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.424 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": { 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1978459317": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1771670410": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/3964053112": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/14180211": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/2547319348": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/601470125": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/3870917014": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1892511221": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/435737617": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/93567168": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/2150730999": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:58:58.425 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/1166209788": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1248481299": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1832156537": "2024-03-27T03:44:16.095969+0000" 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [], 2024-03-26T03:58:58.426 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:58:58.427 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:58:58.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:58.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:58.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:58.594 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:58:58.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:58.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:58.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:58:58.868 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:58.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:58.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:58:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:58 smithi006 ceph-mon[37501]: pgmap v456: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.09k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 140 objects/s recovering 2024-03-26T03:58:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:58:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1160511301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:58:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:58 smithi006 ceph-mon[29446]: pgmap v456: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.09k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 140 objects/s recovering 2024-03-26T03:58:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:58:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1160511301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:58:58.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:58:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:58 smithi086 ceph-mon[32965]: pgmap v456: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 384 MiB data, 3.6 GiB used, 622 GiB / 626 GiB avail; 71 KiB/s rd, 0 B/s wr, 1.09k op/s; 451/19089 objects misplaced (2.363%); 0 B/s, 140 objects/s recovering 2024-03-26T03:58:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:58:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1160511301' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:58:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:59.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:58:59.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:58:59.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:58:59.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:58:59.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:59.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:59.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:59.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:59.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:59.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:59.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:58:59.561 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:58:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:58:59.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:58:59.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:58:59.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:58:59.847 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:58:59.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:58:59.984 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:58:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:58:59.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:00.075 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Removed 129928 objects 2024-03-26T03:59:00.075 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Clean up completed and total clean up time :159.465 2024-03-26T03:59:00.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:00.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:00.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:00.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:00.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:00.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:00.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:00.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:00.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:00.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:00.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:00.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:00 smithi006 ceph-mon[37501]: osdmap e245: 8 total, 8 up, 7 in 2024-03-26T03:59:00.586 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:00 smithi006 ceph-mon[37501]: pgmap v458: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 302 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 70 KiB/s rd, 0 B/s wr, 1.08k op/s; 125/15156 objects misplaced (0.825%) 2024-03-26T03:59:00.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:00 smithi006 ceph-mon[29446]: osdmap e245: 8 total, 8 up, 7 in 2024-03-26T03:59:00.586 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:00 smithi006 ceph-mon[29446]: pgmap v458: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 302 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 70 KiB/s rd, 0 B/s wr, 1.08k op/s; 125/15156 objects misplaced (0.825%) 2024-03-26T03:59:00.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:00.611 INFO:tasks.thrashosds.thrasher:j is {'epoch': 244, 'fsid': 'de980954-eb22-11ee-95d0-87774f69a715', 'created': '2024-03-26T03:43:14.534401+0000', 'modified': '2024-03-26T03:58:55.623388+0000', 'last_up_change': '2024-03-26T03:55:58.386605+0000', 'last_in_change': '2024-03-26T03:57:48.463474+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T03:46:30.183658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T03:48:41.622497+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T03:48:42.191818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T03:48:44.079106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T03:48:46.035261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T03:48:48.086810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-26T03:54:37.148544+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '1d1c84cf-5fa7-437e-8004-3b7aca831d6e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 226, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6801', 'nonce': 1246039834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 1246039834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 1246039834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6803', 'nonce': 1246039834}]}, 'public_addr': '172.21.15.6:6801/1246039834', 'cluster_addr': '172.21.15.6:6802/1246039834', 'heartbeat_back_addr': '172.21.15.6:6804/1246039834', 'heartbeat_front_addr': '172.21.15.6:6803/1246039834', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '753f7459-62ec-447b-a0da-64d06e813f16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 234, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6805', 'nonce': 260311411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 260311411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 260311411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6807', 'nonce': 260311411}]}, 'public_addr': '172.21.15.6:6805/260311411', 'cluster_addr': '172.21.15.6:6806/260311411', 'heartbeat_back_addr': '172.21.15.6:6808/260311411', 'heartbeat_front_addr': '172.21.15.6:6807/260311411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b7d7409f-341a-4711-8570-92fe9082a528', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 236, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6809', 'nonce': 3103276794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 3103276794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 3103276794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6811', 'nonce': 3103276794}]}, 'public_addr': '172.21.15.6:6809/3103276794', 'cluster_addr': '172.21.15.6:6810/3103276794', 'heartbeat_back_addr': '172.21.15.6:6812/3103276794', 'heartbeat_front_addr': '172.21.15.6:6811/3103276794', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a91459a0-dee4-420c-ae2f-940756c985a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 230, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6813', 'nonce': 2795883342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 2795883342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2795883342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6815', 'nonce': 2795883342}]}, 'public_addr': '172.21.15.6:6813/2795883342', 'cluster_addr': '172.21.15.6:6814/2795883342', 'heartbeat_back_addr': '172.21.15.6:6816/2795883342', 'heartbeat_front_addr': '172.21.15.6:6815/2795883342', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '087da212-7bf7-4982-b7cc-11f3f66c7ff8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 236, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 931538411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 931538411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6803', 'nonce': 931538411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6802', 'nonce': 931538411}]}, 'public_addr': '172.21.15.86:6800/931538411', 'cluster_addr': '172.21.15.86:6801/931538411', 'heartbeat_back_addr': '172.21.15.86:6803/931538411', 'heartbeat_front_addr': '172.21.15.86:6802/931538411', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7a688745-b213-4bbc-a735-42ffd2a5bfdc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 167, 'up_from': 180, 'up_thru': 236, 'down_at': 168, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 909197223}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 909197223}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6807', 'nonce': 909197223}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6806', 'nonce': 909197223}]}, 'public_addr': '172.21.15.86:6804/909197223', 'cluster_addr': '172.21.15.86:6805/909197223', 'heartbeat_back_addr': '172.21.15.86:6807/909197223', 'heartbeat_front_addr': '172.21.15.86:6806/909197223', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '58b58555-59a3-4620-a15f-024f7041713f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 1580133777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 1580133777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6811', 'nonce': 1580133777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6810', 'nonce': 1580133777}]}, 'public_addr': '172.21.15.86:6808/1580133777', 'cluster_addr': '172.21.15.86:6809/1580133777', 'heartbeat_back_addr': '172.21.15.86:6811/1580133777', 'heartbeat_front_addr': '172.21.15.86:6810/1580133777', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '49b1833d-0a24-4168-a195-47b8ac1d9bc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7586517333984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 214, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 3905426306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 3905426306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6815', 'nonce': 3905426306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6814', 'nonce': 3905426306}]}, 'public_addr': '172.21.15.86:6812/3905426306', 'cluster_addr': '172.21.15.86:6813/3905426306', 'heartbeat_back_addr': '172.21.15.86:6815/3905426306', 'heartbeat_front_addr': '172.21.15.86:6814/3905426306', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:00.485830+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:57:40.315263+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:05.377743+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:57:47.388775+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:47.161748+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:32.173892+0000', 'dead_epoch': 167}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:58:02.396934+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:53.055285+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.1f', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.9', 'osds': [5, 0, 4]}], 'primary_temp': [], 'blocklist': {'172.21.15.6:0/1978459317': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/1771670410': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/3964053112': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/14180211': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:6800/2547319348': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/601470125': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:6800/3870917014': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1892511221': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/435737617': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/93567168': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/2150730999': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:6800/1166209788': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/1248481299': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1832156537': '2024-03-27T03:44:16.095969+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:59:00.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:59:00.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:00.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:00.679 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-26T03:59:00.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-26T03:59:00.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.748 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:00.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:00 smithi086 ceph-mon[32965]: osdmap e245: 8 total, 8 up, 7 in 2024-03-26T03:59:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:00 smithi086 ceph-mon[32965]: pgmap v458: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 302 MiB data, 3.3 GiB used, 622 GiB / 626 GiB avail; 70 KiB/s rd, 0 B/s wr, 1.08k op/s; 125/15156 objects misplaced (0.825%) 2024-03-26T03:59:00.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:00.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:00.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:00.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:01.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:01.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:01.108 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:59:01.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:01.139 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:59:01.158 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:59:01.178 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:59:01.200 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:59:01.221 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:59:01.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:01.242 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:59:01.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:01.263 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:59:01.264 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:59:01.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:01.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[29446]: osdmap e246: 8 total, 8 up, 7 in 2024-03-26T03:59:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1316924740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:01.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:01.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[37501]: osdmap e246: 8 total, 8 up, 7 in 2024-03-26T03:59:01.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1316924740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:01.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:01.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:01.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:01.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:01.646 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:01.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:01.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:01 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:01 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:01 smithi086 ceph-mon[32965]: osdmap e246: 8 total, 8 up, 7 in 2024-03-26T03:59:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1316924740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:01.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:01.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:01.903 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:01.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:02.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:02.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:02.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:02.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:02.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:02.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:02.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:02.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:02.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:02.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:02.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:02.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:02 smithi006 ceph-mon[37501]: pgmap v460: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 40 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 917 op/s; 125/2607 objects misplaced (4.795%) 2024-03-26T03:59:02.610 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:02 smithi006 ceph-mon[29446]: pgmap v460: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 40 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 917 op/s; 125/2607 objects misplaced (4.795%) 2024-03-26T03:59:02.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:02.666 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:02.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:02.792 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:02 smithi086 ceph-mon[32965]: pgmap v460: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 40 MiB data, 2.6 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 917 op/s; 125/2607 objects misplaced (4.795%) 2024-03-26T03:59:02.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:02.922 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:02.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:03.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:03.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:03.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:03.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:03.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:03.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:03.333 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:03.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:03.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:03.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:03.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:03.758 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:03.775 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:59:03.775 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T03:59:03.775 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T03:59:03.775 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T03:59:03.775 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:59:03.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:03.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:03.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:03.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:03.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:04.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:04.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:04.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:04.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:04.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:04.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:04.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:04.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:04.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:04.364 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:04.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:04.490 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:04.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:04.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:04.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:59:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[37501]: osdmap e247: 8 total, 8 up, 7 in 2024-03-26T03:59:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[37501]: pgmap v462: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 9.1 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T03:59:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[29446]: osdmap e247: 8 total, 8 up, 7 in 2024-03-26T03:59:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:04 smithi006 ceph-mon[29446]: pgmap v462: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 9.1 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T03:59:04.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:04.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:04.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:04.768 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:04 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:04.768 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:04 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:04.768 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:04 smithi086 ceph-mon[32965]: osdmap e247: 8 total, 8 up, 7 in 2024-03-26T03:59:04.768 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:04 smithi086 ceph-mon[32965]: pgmap v462: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 9.1 MiB data, 2.5 GiB used, 623 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 1.00k op/s 2024-03-26T03:59:04.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:04.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:04.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:04.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:04.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:05.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:05.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:05.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:05.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:05.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:05.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:05.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:05.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:05.388 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:05.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:05.516 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:05.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:05.638 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:05.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:05.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T03:59:05.643 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:05.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:05.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:05.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:05.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:05.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:05.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:05.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:06.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:06.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:06.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:06.262 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:06.285 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:06.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:06.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:06.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:06.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:06.446 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:06 smithi006 ceph-mon[29446]: pgmap v463: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 799 op/s 2024-03-26T03:59:06.447 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:06 smithi006 ceph-mon[37501]: pgmap v463: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 799 op/s 2024-03-26T03:59:06.508 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:06.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:06.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:06.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:06.732 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:06.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:06.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:06 smithi086 ceph-mon[32965]: pgmap v463: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 847 B/s rd, 0 B/s wr, 799 op/s 2024-03-26T03:59:06.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:06.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:06.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:06.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:07.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:07.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:07.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:07.182 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:07.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:07.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.310 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:07.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:07.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:07.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:07.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:07.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:07.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:07.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:07.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:07.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:07.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:07.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:08.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:08.067 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:08.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:08.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:08.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:08.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:08.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:08.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:08.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:08.453 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T03:59:08.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:08.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:08.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:08.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:08.805 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:08.829 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:08.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:08 smithi086 ceph-mon[32965]: pgmap v464: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 603 op/s 2024-03-26T03:59:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:08 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/120025977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:08 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[29446]: pgmap v464: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 603 op/s 2024-03-26T03:59:08.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/120025977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[37501]: pgmap v464: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 603 op/s 2024-03-26T03:59:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/120025977' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:08 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T03:59:08.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:08.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:08.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:08.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:09.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:09.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:09.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:09.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:09.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:09.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:09.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:09.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:09.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:09.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:09.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:09.598 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:09.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:09.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:09.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:09.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:59:09.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[37501]: osdmap e248: 8 total, 8 up, 7 in 2024-03-26T03:59:09.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:09.793 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[37501]: osdmap e249: 8 total, 8 up, 7 in 2024-03-26T03:59:09.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:09.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:59:09.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[29446]: osdmap e248: 8 total, 8 up, 7 in 2024-03-26T03:59:09.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:09.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:09 smithi006 ceph-mon[29446]: osdmap e249: 8 total, 8 up, 7 in 2024-03-26T03:59:09.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:09 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T03:59:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:09 smithi086 ceph-mon[32965]: osdmap e248: 8 total, 8 up, 7 in 2024-03-26T03:59:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:09 smithi086 ceph-mon[32965]: osdmap e249: 8 total, 8 up, 7 in 2024-03-26T03:59:09.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:09.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:09.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:09.985 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:09.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:09.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:10.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:10.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:10.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:10.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:10.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:10.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:10.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:10.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:10.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:10.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:10.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:10.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.623 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:10.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:10.704 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:10 smithi006 ceph-mon[29446]: pgmap v467: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:59:10.704 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:10 smithi006 ceph-mon[37501]: pgmap v467: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:59:10.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:10.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:10.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:10 smithi086 ceph-mon[32965]: pgmap v467: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T03:59:10.853 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:10.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:10.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:10.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:10.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:11.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:11.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:11.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:11.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:11.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:11.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:11.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:11.304 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:59:11.304 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215910 8.b 8.e1f86c5b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.521718+0000", 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.780912946000001, 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81234027399999997, 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.305 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215910 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521718+0000", 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521718+0000", 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.306 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521720+0000", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.666e-06 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521724+0000", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8350000000000003e-06 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521733+0000", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2889999999999998e-06 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.307 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521741+0000", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6759999999999996e-06 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521871+0000", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012913900000000001 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.521879+0000", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9899999999999997e-06 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.308 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334048+0000", 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81216919899999995 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334058+0000", 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.048e-05 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215912 8.b 8.33b50e1b (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.522012+0000", 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.780618935999996, 2024-03-26T03:59:11.309 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81207836, 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215912 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522012+0000", 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.310 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522012+0000", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522013+0000", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1699999999999998e-07 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522016+0000", 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8039999999999999e-06 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.311 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522025+0000", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4419999999999998e-06 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522032+0000", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4540000000000001e-06 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522181+0000", 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014888600000000001 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.312 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522188+0000", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9569999999999996e-06 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334082+0000", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81189413600000004 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334090+0000", 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0639999999999994e-06 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.313 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215918 8.b 8.2467ddab (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.522390+0000", 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.780240489999997, 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81172792900000001, 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215918 2024-03-26T03:59:11.314 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522390+0000", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522390+0000", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522391+0000", 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3e-07 2024-03-26T03:59:11.315 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522395+0000", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.765e-06 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522402+0000", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.725e-06 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522409+0000", 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3479999999999998e-06 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.316 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522525+0000", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116154 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522532+0000", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7050000000000002e-06 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334111+0000", 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81157887100000003 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.317 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334118+0000", 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7309999999999993e-06 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215923 8.b 8.334b0bcb (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.522544+0000", 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.780086783000002, 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81160226999999996, 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.318 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215923 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522544+0000", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522544+0000", 2024-03-26T03:59:11.319 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522545+0000", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5199999999999997e-07 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522548+0000", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5379999999999998e-06 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522554+0000", 2024-03-26T03:59:11.320 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6930000000000004e-06 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522560+0000", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9499999999999998e-06 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522878+0000", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00031817999999999999 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522885+0000", 2024-03-26T03:59:11.321 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8079999999999999e-06 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334139+0000", 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81125376900000001 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334146+0000", 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7800000000000001e-06 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.322 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215928 8.f 8.9cfc66af (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.522902+0000", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.779728706, 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82001081899999995, 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215928 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.323 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522902+0000", 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522902+0000", 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522903+0000", 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7299999999999995e-07 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.324 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522907+0000", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.771e-06 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522913+0000", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3849999999999999e-06 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.522920+0000", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.477e-06 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.325 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.523232+0000", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00031223200000000002 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.523239+0000", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.764e-06 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.342903+0000", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81966474300000003 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.326 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.342913+0000", 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7739999999999999e-06 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215941 8.b 8.2a5e570b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.543466+0000", 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.759164631000004, 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79070805899999996, 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215941 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543466+0000", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543466+0000", 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.328 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543468+0000", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.95e-06 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543478+0000", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5610000000000004e-06 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543500+0000", 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.243e-05 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.329 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543511+0000", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1191e-05 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543581+0000", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9280000000000001e-05 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.543590+0000", 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9199999999999993e-06 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.330 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334167+0000", 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79057719199999998 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.334174+0000", 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5349999999999999e-06 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.332 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215946 8.f 8.d2c6034f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.549694+0000", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.752937255999996, 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.793249435, 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215946 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549694+0000", 2024-03-26T03:59:11.333 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549694+0000", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549696+0000", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1500000000000002e-06 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549704+0000", 2024-03-26T03:59:11.334 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6060000000000003e-06 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549724+0000", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9385000000000002e-05 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549736+0000", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1771e-05 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549811+0000", 2024-03-26T03:59:11.335 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5597000000000001e-05 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.549822+0000", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1204e-05 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.342935+0000", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79311306400000003 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.342943+0000", 2024-03-26T03:59:11.336 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6580000000000002e-06 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:2399 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94232629231488 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.177720+0000", 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 82.124911028, 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.28297981799999999, 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.337 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 2399 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177720+0000", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177720+0000", 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.338 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177722+0000", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8819999999999999e-06 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177731+0000", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7150000000000004e-06 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177753+0000", 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2982000000000001e-05 2024-03-26T03:59:11.339 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177765+0000", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.113e-05 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177970+0000", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00020536 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177977+0000", 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4089999999999997e-06 2024-03-26T03:59:11.340 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.374099+0000", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19612127500000001 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.374232+0000", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013335800000000001 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.383012+0000", 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0087797299999999995 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.341 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for rw locks", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.383102+0000", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0451000000000005e-05 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.387037+0000", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039349720000000001 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.387129+0000", 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2120000000000006e-05 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.342 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.460700+0000", 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.073570434000000004 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218240 8.f 8:ff705fb5:::benchmark_data_smithi006_103169_object88309:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.629728+0000", 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.672902679999993, 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.09094242, 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.343 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218240 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629728+0000", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629728+0000", 2024-03-26T03:59:11.344 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629729+0000", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.226e-06 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629735+0000", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3539999999999999e-06 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629745+0000", 2024-03-26T03:59:11.345 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9930000000000007e-06 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629753+0000", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8830000000000006e-06 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629809+0000", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5939000000000001e-05 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.629815+0000", 2024-03-26T03:59:11.346 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3450000000000002e-06 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.710465+0000", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.080650358 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.710622+0000", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000156362 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.710739+0000", 2024-03-26T03:59:11.347 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117349 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720437+0000", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096975870000000006 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720525+0000", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8181000000000002e-05 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720536+0000", 2024-03-26T03:59:11.348 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0818999999999999e-05 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720621+0000", 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5654000000000004e-05 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720671+0000", 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9370000000000003e-05 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.349 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218259 8.f 8:f36c2d7c:::benchmark_data_smithi006_103169_object88328:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.710820+0000", 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.591810593000005, 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.010280308, 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218259 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.350 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710820+0000", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710820+0000", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710822+0000", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3740000000000001e-06 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.351 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710827+0000", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9910000000000002e-06 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710837+0000", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0540000000000001e-05 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710844+0000", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2389999999999996e-06 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.352 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710894+0000", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9400000000000001e-05 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.710901+0000", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8109999999999997e-06 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.712663+0000", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.001762713 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.353 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.712780+0000", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116153 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.712914+0000", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000134005 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720483+0000", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075693590000000003 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.354 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720933+0000", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000449911 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720990+0000", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6777999999999997e-05 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721061+0000", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1045999999999997e-05 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721101+0000", 2024-03-26T03:59:11.355 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9987999999999997e-05 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218277 8.f 8:f0df8837:::benchmark_data_smithi006_103169_object88346:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.783292+0000", 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.519339197999997, 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.94766722999999997, 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.356 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218277 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783292+0000", 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783292+0000", 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.357 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783293+0000", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.31e-06 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783299+0000", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7899999999999996e-06 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783311+0000", 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2561000000000001e-05 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.358 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783320+0000", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3599999999999996e-06 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783368+0000", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8421999999999998e-05 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.783378+0000", 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6889999999999999e-06 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.359 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.714582+0000", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.93120455000000002 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.714701+0000", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118517 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.714819+0000", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118299 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.360 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.720938+0000", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061191359999999998 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.729045+0000", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081067940000000005 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.730827+0000", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017818529999999999 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.361 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.730895+0000", 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8343999999999995e-05 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.730959+0000", 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3604999999999998e-05 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218287 8.f 8:fc35f4b9:::benchmark_data_smithi006_103169_object88356:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.812496+0000", 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.490134808999997, 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.928276345, 2024-03-26T03:59:11.362 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218287 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812496+0000", 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.363 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812496+0000", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812499+0000", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5299999999999999e-06 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812512+0000", 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.293e-05 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.364 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812530+0000", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8372999999999998e-05 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812541+0000", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.066e-05 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812616+0000", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5557999999999997e-05 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.812626+0000", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0281e-05 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.716367+0000", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.90374029199999995 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.716479+0000", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011226900000000001 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.366 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.716589+0000", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110012 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721115+0000", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045260650000000001 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.729086+0000", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079705209999999995 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740673+0000", 2024-03-26T03:59:11.367 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011587879000000001 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740724+0000", 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0056999999999999e-05 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740772+0000", 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8918000000000002e-05 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.368 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218314 8.f 8:fc8c408e:::benchmark_data_smithi006_103169_object88383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.905057+0000", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.397574390000003, 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.83583754799999999, 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218314 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.369 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905057+0000", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905057+0000", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905058+0000", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3239999999999999e-06 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.370 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905063+0000", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5960000000000003e-06 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905078+0000", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4273999999999999e-05 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905084+0000", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3790000000000003e-06 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.371 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905144+0000", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0173000000000001e-05 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.905154+0000", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.798e-06 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.718102+0000", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81294758700000003 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.372 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.718229+0000", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012725400000000001 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.718345+0000", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011642300000000001 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721120+0000", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002774636 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.373 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.730790+0000", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096697410000000008 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740818+0000", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010027781 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740859+0000", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1261999999999999e-05 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740894+0000", 2024-03-26T03:59:11.374 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5320000000000001e-05 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218321 8.f 8:fc69d2fb:::benchmark_data_smithi006_103169_object88390:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.913015+0000", 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.389615939999999, 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.83167136100000005, 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.375 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218321 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913015+0000", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913015+0000", 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.376 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913017+0000", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5850000000000001e-06 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913022+0000", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4840000000000003e-06 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913032+0000", 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0022e-05 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.377 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913038+0000", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.241e-06 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913086+0000", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7309999999999999e-05 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.913092+0000", 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0460000000000002e-06 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.378 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721150+0000", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80805831699999997 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721271+0000", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120689 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.721353+0000", 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2583000000000005e-05 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.379 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.736919+0000", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015565536 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740930+0000", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040114349999999998 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.744453+0000", 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035227829999999998 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.380 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.744626+0000", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017340000000000001 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.744686+0000", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9929999999999997e-05 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218327 8.f 8:ffb46c9c:::benchmark_data_smithi006_103169_object88396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:00.925106+0000", 2024-03-26T03:59:11.381 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.37752476, 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82044407200000002, 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218327 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925106+0000", 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.382 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925106+0000", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925108+0000", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3430000000000001e-06 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925112+0000", 2024-03-26T03:59:11.383 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4410000000000003e-06 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925121+0000", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4960000000000002e-06 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925128+0000", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2519999999999996e-06 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925180+0000", 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.223e-05 2024-03-26T03:59:11.384 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:00.925190+0000", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0084e-05 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.722804+0000", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79761351300000005 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.722909+0000", 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010535900000000001 2024-03-26T03:59:11.385 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.723022+0000", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113355 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.736928+0000", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013905367 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.742127+0000", 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0051995369999999997 2024-03-26T03:59:11.386 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745419+0000", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032918779999999998 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745508+0000", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9009000000000001e-05 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745550+0000", 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2208e-05 2024-03-26T03:59:11.387 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218373 8.f 8:f7709088:::benchmark_data_smithi006_103169_object88442:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:01.045633+0000", 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.256997849000001, 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.70008051100000002, 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.388 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218373 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045633+0000", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045633+0000", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045635+0000", 2024-03-26T03:59:11.389 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4810000000000001e-06 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045647+0000", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2319e-05 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045657+0000", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0249e-05 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045665+0000", 2024-03-26T03:59:11.390 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8429999999999993e-06 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045726+0000", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0786999999999999e-05 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.045734+0000", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3690000000000007e-06 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.724763+0000", 2024-03-26T03:59:11.391 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.67902891899999995 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.724868+0000", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000104783 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.724973+0000", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000104676 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.738859+0000", 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013886928 2024-03-26T03:59:11.392 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.742182+0000", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003322228 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745588+0000", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034059400000000001 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745673+0000", 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5006999999999994e-05 2024-03-26T03:59:11.393 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.745714+0000", 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0982000000000002e-05 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218389 8.f 8:fe3507db:::benchmark_data_smithi006_103169_object88458:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:01.155645+0000", 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 70.146985551, 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.59328298499999998, 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.394 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.395 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.395 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.395 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218389 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155645+0000", 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.396 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155645+0000", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155647+0000", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.517e-06 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155653+0000", 2024-03-26T03:59:11.397 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7989999999999999e-06 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155666+0000", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3165e-05 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155674+0000", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2970000000000004e-06 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155737+0000", 2024-03-26T03:59:11.398 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3300999999999996e-05 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.155746+0000", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0159999999999997e-06 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.726590+0000", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.57084333700000001 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.726705+0000", 2024-03-26T03:59:11.399 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011469200000000001 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.726819+0000", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011466100000000001 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.738865+0000", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012046148 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.742215+0000", 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033499409999999999 2024-03-26T03:59:11.400 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.748786+0000", 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0065710589999999998 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.748877+0000", 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1133999999999999e-05 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.401 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.748928+0000", 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0918000000000003e-05 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218419 8.f 8:fadec2c4:::benchmark_data_smithi006_103169_object88488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:01.456866+0000", 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.845765177000004, 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.29232641100000001, 2024-03-26T03:59:11.403 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218419 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456866+0000", 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.404 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456866+0000", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456867+0000", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6139999999999999e-06 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456873+0000", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8449999999999999e-06 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.405 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456886+0000", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2957e-05 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.456894+0000", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2379999999999997e-06 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.457195+0000", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030085300000000001 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.406 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.457205+0000", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5950000000000005e-06 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.729112+0000", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.27190667600000001 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.729233+0000", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121457 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.407 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.729363+0000", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000130345 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.738870+0000", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.009506904 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.742248+0000", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033775290000000002 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.748973+0000", 2024-03-26T03:59:11.408 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067249689999999999 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.749151+0000", 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017807399999999999 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.749192+0000", 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1355e-05 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.409 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218433 8.f 8:f92365a7:::benchmark_data_smithi006_103169_object88502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:01.474522+0000", 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.828109178000005, 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.27883419599999998, 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218433 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.410 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474522+0000", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474522+0000", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474524+0000", 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7799999999999999e-06 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.411 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474530+0000", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0390000000000001e-06 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474539+0000", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3200000000000006e-06 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474549+0000", 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6520000000000001e-06 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.412 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474618+0000", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9201000000000004e-05 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.474627+0000", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4639999999999995e-06 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.730989+0000", 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25636186700000002 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.413 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.731118+0000", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012925999999999999 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.731264+0000", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014562399999999999 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740622+0000", 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0093576879999999994 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.414 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.750174+0000", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0095520750000000001 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753256+0000", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0030820539999999999 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753311+0000", 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5461999999999999e-05 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.415 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753356+0000", 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4709999999999997e-05 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:218442 8.f 8:f241910b:::benchmark_data_smithi006_103169_object88511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:01.497756+0000", 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 69.804874574999999, 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.25572953799999998, 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T03:59:11.416 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 218442 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497756+0000", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497756+0000", 2024-03-26T03:59:11.417 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497758+0000", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5850000000000001e-06 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497765+0000", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6660000000000002e-06 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497779+0000", 2024-03-26T03:59:11.418 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4311000000000001e-05 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497788+0000", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1300000000000007e-06 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497849+0000", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0733999999999998e-05 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.497860+0000", 2024-03-26T03:59:11.419 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0941000000000001e-05 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.733560+0000", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.235699783 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.733661+0000", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010165700000000001 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.733773+0000", 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000112039 2024-03-26T03:59:11.420 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.740627+0000", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068540559999999999 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.750924+0000", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010297219999999999 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753402+0000", 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002477295 2024-03-26T03:59:11.421 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753451+0000", 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9098999999999999e-05 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:01.753486+0000", 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5021999999999998e-05 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:11.422 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:59:11.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:11.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:11.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:11.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:11 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T03:59:11.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:11 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T03:59:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:11 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T03:59:11.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:11 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T03:59:11.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:11.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:11.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:11.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:11.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:11.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:11 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T03:59:11.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:11 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T03:59:11.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:11.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:12.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:12.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:12.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:12.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:12.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:12.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:12.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:12.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:12.484 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_1' removed 2024-03-26T03:59:12.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:12.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:12.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:12.735 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:12.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: pgmap v468: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: osdmap e250: 8 total, 8 up, 7 in 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/260800531' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.792 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: pgmap v468: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: osdmap e250: 8 total, 8 up, 7 in 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/260800531' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.793 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:12 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.806 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:12.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: pgmap v468: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 18 op/s 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: osdmap e250: 8 total, 8 up, 7 in 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/260800531' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:12 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T03:59:12.862 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:12.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:12.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:12.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:12.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:13.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:13.114 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:13.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:13.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:13.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:13.365 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:13.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:13.490 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:13.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:13.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:13.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T03:59:13.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T03:59:13.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[37501]: osdmap e251: 8 total, 8 up, 7 in 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T03:59:13.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:59:13.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:13 smithi006 ceph-mon[29446]: osdmap e251: 8 total, 8 up, 7 in 2024-03-26T03:59:13.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T03:59:13.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T03:59:13.726 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:13 smithi086 ceph-mon[32965]: osdmap e251: 8 total, 8 up, 7 in 2024-03-26T03:59:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:13.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:13.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:13.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:13.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:13.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:13.942 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:13.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:14.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:14.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:14.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:14.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:14.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:14.257 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:14.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:14.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:14.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:14.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:14.508 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:14.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:14.541 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:59:14.544 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:59:14.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: 4.1d scrub starts 2024-03-26T03:59:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: 4.1d scrub ok 2024-03-26T03:59:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-03-26T03:59:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:59:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T03:59:14.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T03:59:14.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:14.634 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:14.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: 4.1d scrub starts 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: 4.1d scrub ok 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: 4.1d scrub starts 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: 4.1d scrub ok 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: pgmap v471: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 828 B/s rd, 0 op/s 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T03:59:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:14 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T03:59:14.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:14.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:14.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:14.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:14.886 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:14.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:14.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:14.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:15.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:15.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:15.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:15.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:15.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:15.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:15.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:15.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:15.388 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:15.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:15.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:15.599 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:59:15.599 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:59:15.599 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-03-26T03:59:15.599 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:59:15.608 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T03:59:15.608 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T03:59:15.608 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T03:59:15.608 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T03:59:15.608 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:15.609 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T03:59:15.609 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T03:59:15.609 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:15.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:15.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.641 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:15.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:15.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:15.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:15.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T03:59:15.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T03:59:15.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: from='client.15261 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T03:59:15.870 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:15.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:15.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:15.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:15.926 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T03:59:15.927 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T03:59:15.927 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-26T03:59:15.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-26T03:59:15.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:16.018 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:16.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.146 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:16.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:16.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T03:59:16.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:16.270 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:16.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:16.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:16.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:16.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:16.469 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:16.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:16.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:16 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T03:59:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:16 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T03:59:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:16 smithi086 ceph-mon[32965]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 677 B/s rd, 0 op/s 2024-03-26T03:59:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:16 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T03:59:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:16 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T03:59:16.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:16.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:16.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T03:59:16.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T03:59:16.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[37501]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 677 B/s rd, 0 op/s 2024-03-26T03:59:16.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T03:59:16.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T03:59:16.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T03:59:16.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T03:59:16.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[29446]: pgmap v472: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 677 B/s rd, 0 op/s 2024-03-26T03:59:16.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T03:59:16.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:16 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T03:59:16.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:16.776 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:16.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:16.903 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:16.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:16.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:16.959 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:59:16.982 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:59:17.004 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:59:17.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:17.024 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:59:17.027 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:17.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:17.044 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:59:17.064 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:59:17.084 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:59:17.104 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:59:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:59:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:59:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:59:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:59:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:59:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:59:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:59:17.106 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:59:17.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:17.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:17.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:17.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:17.276 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:17.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:17.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:17.383 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-03-26T03:59:13.392376+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2261,"num_read_kb":2075,"num_write":1093,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":2368758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1137,"ondisk_log_size":1137,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":448,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2389896,"kb_used_data":127080,"kb_used_omap":13,"kb_used_meta":2262706,"kb_avail":653826168,"statfs":{"total":671965249536,"available":669517996032,"internally_reserved":0,"allocated":130129920,"data_stored":126022384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14011,"internal_metadata":2317011269},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-259858,"num_write_kb":-8315393,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-42332,"num_bytes_recovered":-2507538432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.946324"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":394,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351703+0000","last_change":"2024-03-26T03:55:59.300239+0000","last_active":"2024-03-26T03:59:09.351703+0000","last_peered":"2024-03-26T03:59:09.351703+0000","last_clean":"2024-03-26T03:59:09.351703+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T03:59:09.351703+0000","last_undegraded":"2024-03-26T03:59:09.351703+0000","last_fullsized":"2024-03-26T03:59:09.351703+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.435103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.435103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373414+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":404,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974765+0000","last_change":"2024-03-26T03:58:37.457198+0000","last_active":"2024-03-26T03:59:09.974765+0000","last_peered":"2024-03-26T03:59:09.974765+0000","last_clean":"2024-03-26T03:59:09.974765+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T03:59:09.974765+0000","last_undegraded":"2024-03-26T03:59:09.974765+0000","last_fullsized":"2024-03-26T03:59:09.974765+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T03:58:37.457112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:59.336215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":415,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353298+0000","last_change":"2024-03-26T03:55:59.307852+0000","last_active":"2024-03-26T03:59:09.353298+0000","last_peered":"2024-03-26T03:59:09.353298+0000","last_clean":"2024-03-26T03:59:09.353298+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T03:59:09.353298+0000","last_undegraded":"2024-03-26T03:59:09.353298+0000","last_fullsized":"2024-03-26T03:59:09.353298+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848758+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":492,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351515+0000","last_change":"2024-03-26T03:57:49.470002+0000","last_active":"2024-03-26T03:59:09.351515+0000","last_peered":"2024-03-26T03:59:09.351515+0000","last_clean":"2024-03-26T03:59:09.351515+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T03:59:09.351515+0000","last_undegraded":"2024-03-26T03:59:09.351515+0000","last_fullsized":"2024-03-26T03:59:09.351515+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373949+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288064+0000","last_change":"2024-03-26T03:57:49.377568+0000","last_active":"2024-03-26T03:59:04.288064+0000","last_peered":"2024-03-26T03:59:04.288064+0000","last_clean":"2024-03-26T03:59:04.288064+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T03:59:04.288064+0000","last_undegraded":"2024-03-26T03:59:04.288064+0000","last_fullsized":"2024-03-26T03:59:04.288064+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134507+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":403,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351416+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T03:59:09.351416+0000","last_peered":"2024-03-26T03:59:09.351416+0000","last_clean":"2024-03-26T03:59:09.351416+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:59:09.351416+0000","last_undegraded":"2024-03-26T03:59:09.351416+0000","last_fullsized":"2024-03-26T03:59:09.351416+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373967+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288303+0000","last_change":"2024-03-26T03:57:49.382349+0000","last_active":"2024-03-26T03:59:04.288303+0000","last_peered":"2024-03-26T03:59:04.288303+0000","last_clean":"2024-03-26T03:59:04.288303+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T03:59:04.288303+0000","last_undegraded":"2024-03-26T03:59:04.288303+0000","last_fullsized":"2024-03-26T03:59:04.288303+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":117,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287718+0000","last_change":"2024-03-26T03:57:49.505699+0000","last_active":"2024-03-26T03:59:04.287718+0000","last_peered":"2024-03-26T03:59:04.287718+0000","last_clean":"2024-03-26T03:59:04.287718+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T03:59:04.287718+0000","last_undegraded":"2024-03-26T03:59:04.287718+0000","last_fullsized":"2024-03-26T03:59:04.287718+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'39","last_scrub_stamp":"2024-03-26T03:53:26.316929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:26.316929+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":393,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351678+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T03:59:09.351678+0000","last_peered":"2024-03-26T03:59:09.351678+0000","last_clean":"2024-03-26T03:59:09.351678+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:59:09.351678+0000","last_undegraded":"2024-03-26T03:59:09.351678+0000","last_fullsized":"2024-03-26T03:59:09.351678+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848980+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":390,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.507043+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T03:59:12.507043+0000","last_peered":"2024-03-26T03:59:12.507043+0000","last_clean":"2024-03-26T03:59:12.507043+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T03:59:12.507043+0000","last_undegraded":"2024-03-26T03:59:12.507043+0000","last_fullsized":"2024-03-26T03:59:12.507043+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.756144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":340,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.506158+0000","last_change":"2024-03-26T03:55:59.304086+0000","last_active":"2024-03-26T03:59:12.506158+0000","last_peered":"2024-03-26T03:59:12.506158+0000","last_clean":"2024-03-26T03:59:12.506158+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T03:59:12.506158+0000","last_undegraded":"2024-03-26T03:59:12.506158+0000","last_fullsized":"2024-03-26T03:59:12.506158+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:18.574929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:18.574929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755773+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":491,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352012+0000","last_change":"2024-03-26T03:57:49.407755+0000","last_active":"2024-03-26T03:59:09.352012+0000","last_peered":"2024-03-26T03:59:09.352012+0000","last_clean":"2024-03-26T03:59:09.352012+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T03:59:09.352012+0000","last_undegraded":"2024-03-26T03:59:09.352012+0000","last_fullsized":"2024-03-26T03:59:09.352012+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:53:34.457835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:34.457835+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373478+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":364,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.507259+0000","last_change":"2024-03-26T03:57:49.381007+0000","last_active":"2024-03-26T03:59:12.507259+0000","last_peered":"2024-03-26T03:59:12.507259+0000","last_clean":"2024-03-26T03:59:12.507259+0000","last_became_active":"2024-03-26T03:57:49.380178+0000","last_became_peered":"2024-03-26T03:57:49.380178+0000","last_unstale":"2024-03-26T03:59:12.507259+0000","last_undegraded":"2024-03-26T03:59:12.507259+0000","last_fullsized":"2024-03-26T03:59:12.507259+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755983+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":403,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447381+0000","last_change":"2024-03-26T03:58:28.050117+0000","last_active":"2024-03-26T03:59:08.447381+0000","last_peered":"2024-03-26T03:59:08.447381+0000","last_clean":"2024-03-26T03:59:08.447381+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:59:08.447381+0000","last_undegraded":"2024-03-26T03:59:08.447381+0000","last_fullsized":"2024-03-26T03:59:08.447381+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:28.050003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.050003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:32.555477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":365,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.502671+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T03:59:12.502671+0000","last_peered":"2024-03-26T03:59:12.502671+0000","last_clean":"2024-03-26T03:59:12.502671+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T03:59:12.502671+0000","last_undegraded":"2024-03-26T03:59:12.502671+0000","last_fullsized":"2024-03-26T03:59:12.502671+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.756002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":457,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351999+0000","last_change":"2024-03-26T03:57:49.485971+0000","last_active":"2024-03-26T03:59:09.351999+0000","last_peered":"2024-03-26T03:59:09.351999+0000","last_clean":"2024-03-26T03:59:09.351999+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T03:59:09.351999+0000","last_undegraded":"2024-03-26T03:59:09.351999+0000","last_fullsized":"2024-03-26T03:59:09.351999+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373883+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352494+0000","last_change":"2024-03-26T03:57:49.361032+0000","last_active":"2024-03-26T03:59:09.352494+0000","last_peered":"2024-03-26T03:59:09.352494+0000","last_clean":"2024-03-26T03:59:09.352494+0000","last_became_active":"2024-03-26T03:57:49.360508+0000","last_became_peered":"2024-03-26T03:57:49.360508+0000","last_unstale":"2024-03-26T03:59:09.352494+0000","last_undegraded":"2024-03-26T03:59:09.352494+0000","last_fullsized":"2024-03-26T03:59:09.352494+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:51.658144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:52:51.658144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373393+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974446+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T03:59:09.974446+0000","last_peered":"2024-03-26T03:59:09.974446+0000","last_clean":"2024-03-26T03:59:09.974446+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:59:09.974446+0000","last_undegraded":"2024-03-26T03:59:09.974446+0000","last_fullsized":"2024-03-26T03:59:09.974446+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310292+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352070+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T03:59:09.352070+0000","last_peered":"2024-03-26T03:59:09.352070+0000","last_clean":"2024-03-26T03:59:09.352070+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:59:09.352070+0000","last_undegraded":"2024-03-26T03:59:09.352070+0000","last_fullsized":"2024-03-26T03:59:09.352070+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.849005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":512,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352421+0000","last_change":"2024-03-26T03:58:28.238146+0000","last_active":"2024-03-26T03:59:09.352421+0000","last_peered":"2024-03-26T03:59:09.352421+0000","last_clean":"2024-03-26T03:59:09.352421+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T03:59:09.352421+0000","last_undegraded":"2024-03-26T03:59:09.352421+0000","last_fullsized":"2024-03-26T03:59:09.352421+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:28.237979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.237979+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.896289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288238+0000","last_change":"2024-03-26T03:57:49.377389+0000","last_active":"2024-03-26T03:59:04.288238+0000","last_peered":"2024-03-26T03:59:04.288238+0000","last_clean":"2024-03-26T03:59:04.288238+0000","last_became_active":"2024-03-26T03:57:49.376975+0000","last_became_peered":"2024-03-26T03:57:49.376975+0000","last_unstale":"2024-03-26T03:59:04.288238+0000","last_undegraded":"2024-03-26T03:59:04.288238+0000","last_fullsized":"2024-03-26T03:59:04.288238+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:29.809804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:54:29.809804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352511+0000","last_change":"2024-03-26T03:57:49.360798+0000","last_active":"2024-03-26T03:59:09.352511+0000","last_peered":"2024-03-26T03:59:09.352511+0000","last_clean":"2024-03-26T03:59:09.352511+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T03:59:09.352511+0000","last_undegraded":"2024-03-26T03:59:09.352511+0000","last_fullsized":"2024-03-26T03:59:09.352511+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373669+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":405,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352145+0000","last_change":"2024-03-26T03:53:30.443461+0000","last_active":"2024-03-26T03:59:09.352145+0000","last_peered":"2024-03-26T03:59:09.352145+0000","last_clean":"2024-03-26T03:59:09.352145+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:59:09.352145+0000","last_undegraded":"2024-03-26T03:59:09.352145+0000","last_fullsized":"2024-03-26T03:59:09.352145+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:30.443371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:30.443371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373496+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":98,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287884+0000","last_change":"2024-03-26T03:57:49.469098+0000","last_active":"2024-03-26T03:59:04.287884+0000","last_peered":"2024-03-26T03:59:04.287884+0000","last_clean":"2024-03-26T03:59:04.287884+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T03:59:04.287884+0000","last_undegraded":"2024-03-26T03:59:04.287884+0000","last_fullsized":"2024-03-26T03:59:04.287884+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-03-26T03:52:35.472878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:52:35.472878+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":84,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.502998+0000","last_change":"2024-03-26T03:58:40.497404+0000","last_active":"2024-03-26T03:59:12.502998+0000","last_peered":"2024-03-26T03:59:12.502998+0000","last_clean":"2024-03-26T03:59:12.502998+0000","last_became_active":"2024-03-26T03:58:40.496528+0000","last_became_peered":"2024-03-26T03:58:40.496528+0000","last_unstale":"2024-03-26T03:59:12.502998+0000","last_undegraded":"2024-03-26T03:59:12.502998+0000","last_fullsized":"2024-03-26T03:59:12.502998+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:58:37.045404+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:58:37.045404+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:04.461940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":472,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352745+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T03:59:09.352745+0000","last_peered":"2024-03-26T03:59:09.352745+0000","last_clean":"2024-03-26T03:59:09.352745+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:59:09.352745+0000","last_undegraded":"2024-03-26T03:59:09.352745+0000","last_fullsized":"2024-03-26T03:59:09.352745+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":424,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973882+0000","last_change":"2024-03-26T03:57:49.493948+0000","last_active":"2024-03-26T03:59:09.973882+0000","last_peered":"2024-03-26T03:59:09.973882+0000","last_clean":"2024-03-26T03:59:09.973882+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T03:59:09.973882+0000","last_undegraded":"2024-03-26T03:59:09.973882+0000","last_fullsized":"2024-03-26T03:59:09.973882+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:55:20.749583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.749583+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310197+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"241'26","reported_seq":484,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353694+0000","last_change":"2024-03-26T03:57:49.470100+0000","last_active":"2024-03-26T03:59:09.353694+0000","last_peered":"2024-03-26T03:59:09.353694+0000","last_clean":"2024-03-26T03:59:09.353694+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T03:59:09.353694+0000","last_undegraded":"2024-03-26T03:59:09.353694+0000","last_fullsized":"2024-03-26T03:59:09.353694+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":400,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973876+0000","last_change":"2024-03-26T03:58:39.479856+0000","last_active":"2024-03-26T03:59:09.973876+0000","last_peered":"2024-03-26T03:59:09.973876+0000","last_clean":"2024-03-26T03:59:09.973876+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T03:59:09.973876+0000","last_undegraded":"2024-03-26T03:59:09.973876+0000","last_fullsized":"2024-03-26T03:59:09.973876+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:39.479775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.479775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:46.748116+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":398,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353798+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:59:09.353798+0000","last_peered":"2024-03-26T03:59:09.353798+0000","last_clean":"2024-03-26T03:59:09.353798+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:59:09.353798+0000","last_undegraded":"2024-03-26T03:59:09.353798+0000","last_fullsized":"2024-03-26T03:59:09.353798+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848793+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287496+0000","last_change":"2024-03-26T03:57:49.399354+0000","last_active":"2024-03-26T03:59:04.287496+0000","last_peered":"2024-03-26T03:59:04.287496+0000","last_clean":"2024-03-26T03:59:04.287496+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T03:59:04.287496+0000","last_undegraded":"2024-03-26T03:59:04.287496+0000","last_fullsized":"2024-03-26T03:59:04.287496+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":406,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503437+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T03:59:12.503437+0000","last_peered":"2024-03-26T03:59:12.503437+0000","last_clean":"2024-03-26T03:59:12.503437+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T03:59:12.503437+0000","last_undegraded":"2024-03-26T03:59:12.503437+0000","last_fullsized":"2024-03-26T03:59:12.503437+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.756105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":399,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447544+0000","last_change":"2024-03-26T03:58:38.043643+0000","last_active":"2024-03-26T03:59:08.447544+0000","last_peered":"2024-03-26T03:59:08.447544+0000","last_clean":"2024-03-26T03:59:08.447544+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:59:08.447544+0000","last_undegraded":"2024-03-26T03:59:08.447544+0000","last_fullsized":"2024-03-26T03:59:08.447544+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:38.043527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:38.043527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:01.615166+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":335,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351429+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:59:09.351429+0000","last_peered":"2024-03-26T03:59:09.351429+0000","last_clean":"2024-03-26T03:59:09.351429+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:59:09.351429+0000","last_undegraded":"2024-03-26T03:59:09.351429+0000","last_fullsized":"2024-03-26T03:59:09.351429+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373649+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":426,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353409+0000","last_change":"2024-03-26T03:57:49.385528+0000","last_active":"2024-03-26T03:59:09.353409+0000","last_peered":"2024-03-26T03:59:09.353409+0000","last_clean":"2024-03-26T03:59:09.353409+0000","last_became_active":"2024-03-26T03:57:49.384847+0000","last_became_peered":"2024-03-26T03:57:49.384847+0000","last_unstale":"2024-03-26T03:59:09.353409+0000","last_undegraded":"2024-03-26T03:59:09.353409+0000","last_fullsized":"2024-03-26T03:59:09.353409+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848900+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":87,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287431+0000","last_change":"2024-03-26T03:57:49.492568+0000","last_active":"2024-03-26T03:59:04.287431+0000","last_peered":"2024-03-26T03:59:04.287431+0000","last_clean":"2024-03-26T03:59:04.287431+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T03:59:04.287431+0000","last_undegraded":"2024-03-26T03:59:04.287431+0000","last_fullsized":"2024-03-26T03:59:04.287431+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'6","last_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.271368+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134127+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287944+0000","last_change":"2024-03-26T03:57:49.383008+0000","last_active":"2024-03-26T03:59:04.287944+0000","last_peered":"2024-03-26T03:59:04.287944+0000","last_clean":"2024-03-26T03:59:04.287944+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T03:59:04.287944+0000","last_undegraded":"2024-03-26T03:59:04.287944+0000","last_fullsized":"2024-03-26T03:59:04.287944+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134526+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":409,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447619+0000","last_change":"2024-03-26T03:58:42.072686+0000","last_active":"2024-03-26T03:59:08.447619+0000","last_peered":"2024-03-26T03:59:08.447619+0000","last_clean":"2024-03-26T03:59:08.447619+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:59:08.447619+0000","last_undegraded":"2024-03-26T03:59:08.447619+0000","last_fullsized":"2024-03-26T03:59:08.447619+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:42.072569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.072569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:55.017162+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":373,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503012+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T03:59:12.503012+0000","last_peered":"2024-03-26T03:59:12.503012+0000","last_clean":"2024-03-26T03:59:12.503012+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T03:59:12.503012+0000","last_undegraded":"2024-03-26T03:59:12.503012+0000","last_fullsized":"2024-03-26T03:59:12.503012+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.756085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":101,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287895+0000","last_change":"2024-03-26T03:57:49.481780+0000","last_active":"2024-03-26T03:59:04.287895+0000","last_peered":"2024-03-26T03:59:04.287895+0000","last_clean":"2024-03-26T03:59:04.287895+0000","last_became_active":"2024-03-26T03:57:49.373634+0000","last_became_peered":"2024-03-26T03:57:49.373634+0000","last_unstale":"2024-03-26T03:59:04.287895+0000","last_undegraded":"2024-03-26T03:59:04.287895+0000","last_fullsized":"2024-03-26T03:59:04.287895+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'23","last_scrub_stamp":"2024-03-26T03:53:19.344750+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:53:19.344750+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351137+0000","last_change":"2024-03-26T03:55:59.324307+0000","last_active":"2024-03-26T03:59:09.351137+0000","last_peered":"2024-03-26T03:59:09.351137+0000","last_clean":"2024-03-26T03:59:09.351137+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T03:59:09.351137+0000","last_undegraded":"2024-03-26T03:59:09.351137+0000","last_fullsized":"2024-03-26T03:59:09.351137+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287837+0000","last_change":"2024-03-26T03:57:49.377475+0000","last_active":"2024-03-26T03:59:04.287837+0000","last_peered":"2024-03-26T03:59:04.287837+0000","last_clean":"2024-03-26T03:59:04.287837+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T03:59:04.287837+0000","last_undegraded":"2024-03-26T03:59:04.287837+0000","last_fullsized":"2024-03-26T03:59:04.287837+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134543+0000","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":423,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447732+0000","last_change":"2024-03-26T03:58:11.047294+0000","last_active":"2024-03-26T03:59:08.447732+0000","last_peered":"2024-03-26T03:59:08.447732+0000","last_clean":"2024-03-26T03:59:08.447732+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T03:59:08.447732+0000","last_undegraded":"2024-03-26T03:59:08.447732+0000","last_fullsized":"2024-03-26T03:59:08.447732+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.047184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.047184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:39.616278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":452,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973303+0000","last_change":"2024-03-26T03:55:30.839253+0000","last_active":"2024-03-26T03:59:09.973303+0000","last_peered":"2024-03-26T03:59:09.973303+0000","last_clean":"2024-03-26T03:59:09.973303+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:59:09.973303+0000","last_undegraded":"2024-03-26T03:59:09.973303+0000","last_fullsized":"2024-03-26T03:59:09.973303+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'13","last_scrub_stamp":"2024-03-26T03:55:30.839058+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:55:30.839058+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287603+0000","last_change":"2024-03-26T03:57:49.373069+0000","last_active":"2024-03-26T03:59:04.287603+0000","last_peered":"2024-03-26T03:59:04.287603+0000","last_clean":"2024-03-26T03:59:04.287603+0000","last_became_active":"2024-03-26T03:57:49.370042+0000","last_became_peered":"2024-03-26T03:57:49.370042+0000","last_unstale":"2024-03-26T03:59:04.287603+0000","last_undegraded":"2024-03-26T03:59:04.287603+0000","last_fullsized":"2024-03-26T03:59:04.287603+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.804652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134225+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":416,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974283+0000","last_change":"2024-03-26T03:58:20.461781+0000","last_active":"2024-03-26T03:59:09.974283+0000","last_peered":"2024-03-26T03:59:09.974283+0000","last_clean":"2024-03-26T03:59:09.974283+0000","last_became_active":"2024-03-26T03:57:49.379985+0000","last_became_peered":"2024-03-26T03:57:49.379985+0000","last_unstale":"2024-03-26T03:59:09.974283+0000","last_undegraded":"2024-03-26T03:59:09.974283+0000","last_fullsized":"2024-03-26T03:59:09.974283+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:20.461665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:20.461665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:46.093646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352130+0000","last_change":"2024-03-26T03:53:32.392793+0000","last_active":"2024-03-26T03:59:09.352130+0000","last_peered":"2024-03-26T03:59:09.352130+0000","last_clean":"2024-03-26T03:59:09.352130+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:59:09.352130+0000","last_undegraded":"2024-03-26T03:59:09.352130+0000","last_fullsized":"2024-03-26T03:59:09.352130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:32.392703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:32.392703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":451,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352211+0000","last_change":"2024-03-26T03:55:59.300136+0000","last_active":"2024-03-26T03:59:09.352211+0000","last_peered":"2024-03-26T03:59:09.352211+0000","last_clean":"2024-03-26T03:59:09.352211+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T03:59:09.352211+0000","last_undegraded":"2024-03-26T03:59:09.352211+0000","last_fullsized":"2024-03-26T03:59:09.352211+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-03-26T03:53:33.392055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:33.392055+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373458+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353403+0000","last_change":"2024-03-26T03:57:49.360446+0000","last_active":"2024-03-26T03:59:09.353403+0000","last_peered":"2024-03-26T03:59:09.353403+0000","last_clean":"2024-03-26T03:59:09.353403+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T03:59:09.353403+0000","last_undegraded":"2024-03-26T03:59:09.353403+0000","last_fullsized":"2024-03-26T03:59:09.353403+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:36.618401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:36.618401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848654+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":515,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.504602+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T03:59:12.504602+0000","last_peered":"2024-03-26T03:59:12.504602+0000","last_clean":"2024-03-26T03:59:12.504602+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T03:59:12.504602+0000","last_undegraded":"2024-03-26T03:59:12.504602+0000","last_fullsized":"2024-03-26T03:59:12.504602+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.756126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":411,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447390+0000","last_change":"2024-03-26T03:55:57.916905+0000","last_active":"2024-03-26T03:59:08.447390+0000","last_peered":"2024-03-26T03:59:08.447390+0000","last_clean":"2024-03-26T03:59:08.447390+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:59:08.447390+0000","last_undegraded":"2024-03-26T03:59:08.447390+0000","last_fullsized":"2024-03-26T03:59:08.447390+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.916786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.916786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":106,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288494+0000","last_change":"2024-03-26T03:57:49.435763+0000","last_active":"2024-03-26T03:59:04.288494+0000","last_peered":"2024-03-26T03:59:04.288494+0000","last_clean":"2024-03-26T03:59:04.288494+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T03:59:04.288494+0000","last_undegraded":"2024-03-26T03:59:04.288494+0000","last_fullsized":"2024-03-26T03:59:04.288494+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134376+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":402,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352766+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T03:59:09.352766+0000","last_peered":"2024-03-26T03:59:09.352766+0000","last_clean":"2024-03-26T03:59:09.352766+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T03:59:09.352766+0000","last_undegraded":"2024-03-26T03:59:09.352766+0000","last_fullsized":"2024-03-26T03:59:09.352766+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":96,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288458+0000","last_change":"2024-03-26T03:57:49.512060+0000","last_active":"2024-03-26T03:59:04.288458+0000","last_peered":"2024-03-26T03:59:04.288458+0000","last_clean":"2024-03-26T03:59:04.288458+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T03:59:04.288458+0000","last_undegraded":"2024-03-26T03:59:04.288458+0000","last_fullsized":"2024-03-26T03:59:04.288458+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":443,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351784+0000","last_change":"2024-03-26T03:57:49.480593+0000","last_active":"2024-03-26T03:59:09.351784+0000","last_peered":"2024-03-26T03:59:09.351784+0000","last_clean":"2024-03-26T03:59:09.351784+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T03:59:09.351784+0000","last_undegraded":"2024-03-26T03:59:09.351784+0000","last_fullsized":"2024-03-26T03:59:09.351784+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.849025+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":440,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.504758+0000","last_change":"2024-03-26T03:57:49.431007+0000","last_active":"2024-03-26T03:59:12.504758+0000","last_peered":"2024-03-26T03:59:12.504758+0000","last_clean":"2024-03-26T03:59:12.504758+0000","last_became_active":"2024-03-26T03:57:49.382991+0000","last_became_peered":"2024-03-26T03:57:49.382991+0000","last_unstale":"2024-03-26T03:59:12.504758+0000","last_undegraded":"2024-03-26T03:59:12.504758+0000","last_fullsized":"2024-03-26T03:59:12.504758+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'20","last_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.547507+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755757+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352693+0000","last_change":"2024-03-26T03:58:15.056185+0000","last_active":"2024-03-26T03:59:09.352693+0000","last_peered":"2024-03-26T03:59:09.352693+0000","last_clean":"2024-03-26T03:59:09.352693+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:59:09.352693+0000","last_undegraded":"2024-03-26T03:59:09.352693+0000","last_fullsized":"2024-03-26T03:59:09.352693+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:58:15.056047+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.056047+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:16.858445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288410+0000","last_change":"2024-03-26T03:57:49.368813+0000","last_active":"2024-03-26T03:59:04.288410+0000","last_peered":"2024-03-26T03:59:04.288410+0000","last_clean":"2024-03-26T03:59:04.288410+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T03:59:04.288410+0000","last_undegraded":"2024-03-26T03:59:04.288410+0000","last_fullsized":"2024-03-26T03:59:04.288410+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":114,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351750+0000","last_change":"2024-03-26T03:58:40.497439+0000","last_active":"2024-03-26T03:59:09.351750+0000","last_peered":"2024-03-26T03:59:09.351750+0000","last_clean":"2024-03-26T03:59:09.351750+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T03:59:09.351750+0000","last_undegraded":"2024-03-26T03:59:09.351750+0000","last_fullsized":"2024-03-26T03:59:09.351750+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:15.134741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.134741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:21.059643+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":490,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353735+0000","last_change":"2024-03-26T03:57:49.445142+0000","last_active":"2024-03-26T03:59:09.353735+0000","last_peered":"2024-03-26T03:59:09.353735+0000","last_clean":"2024-03-26T03:59:09.353735+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T03:59:09.353735+0000","last_undegraded":"2024-03-26T03:59:09.353735+0000","last_fullsized":"2024-03-26T03:59:09.353735+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848964+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":406,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447557+0000","last_change":"2024-03-26T03:58:16.141306+0000","last_active":"2024-03-26T03:59:08.447557+0000","last_peered":"2024-03-26T03:59:08.447557+0000","last_clean":"2024-03-26T03:59:08.447557+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T03:59:08.447557+0000","last_undegraded":"2024-03-26T03:59:08.447557+0000","last_fullsized":"2024-03-26T03:59:08.447557+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.141216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.141216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:27.650832+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":370,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503168+0000","last_change":"2024-03-26T03:57:49.381012+0000","last_active":"2024-03-26T03:59:12.503168+0000","last_peered":"2024-03-26T03:59:12.503168+0000","last_clean":"2024-03-26T03:59:12.503168+0000","last_became_active":"2024-03-26T03:57:49.380167+0000","last_became_peered":"2024-03-26T03:57:49.380167+0000","last_unstale":"2024-03-26T03:59:12.503168+0000","last_undegraded":"2024-03-26T03:59:12.503168+0000","last_fullsized":"2024-03-26T03:59:12.503168+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.537750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755734+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":356,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351530+0000","last_change":"2024-03-26T03:57:49.378443+0000","last_active":"2024-03-26T03:59:09.351530+0000","last_peered":"2024-03-26T03:59:09.351530+0000","last_clean":"2024-03-26T03:59:09.351530+0000","last_became_active":"2024-03-26T03:57:49.377869+0000","last_became_peered":"2024-03-26T03:57:49.377869+0000","last_unstale":"2024-03-26T03:59:09.351530+0000","last_undegraded":"2024-03-26T03:59:09.351530+0000","last_fullsized":"2024-03-26T03:59:09.351530+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.708447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.708447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848671+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":94,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288345+0000","last_change":"2024-03-26T03:57:49.511256+0000","last_active":"2024-03-26T03:59:04.288345+0000","last_peered":"2024-03-26T03:59:04.288345+0000","last_clean":"2024-03-26T03:59:04.288345+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T03:59:04.288345+0000","last_undegraded":"2024-03-26T03:59:04.288345+0000","last_fullsized":"2024-03-26T03:59:04.288345+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":357,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.508502+0000","last_change":"2024-03-26T03:55:59.325043+0000","last_active":"2024-03-26T03:59:12.508502+0000","last_peered":"2024-03-26T03:59:12.508502+0000","last_clean":"2024-03-26T03:59:12.508502+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T03:59:12.508502+0000","last_undegraded":"2024-03-26T03:59:12.508502+0000","last_fullsized":"2024-03-26T03:59:12.508502+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:10.731144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:10.731144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755871+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":423,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.448013+0000","last_change":"2024-03-26T03:58:29.964096+0000","last_active":"2024-03-26T03:59:08.448013+0000","last_peered":"2024-03-26T03:59:08.448013+0000","last_clean":"2024-03-26T03:59:08.448013+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:59:08.448013+0000","last_undegraded":"2024-03-26T03:59:08.448013+0000","last_fullsized":"2024-03-26T03:59:08.448013+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:58:29.964005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.964005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:50.334946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":448,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.505510+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T03:59:12.505510+0000","last_peered":"2024-03-26T03:59:12.505510+0000","last_clean":"2024-03-26T03:59:12.505510+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T03:59:12.505510+0000","last_undegraded":"2024-03-26T03:59:12.505510+0000","last_fullsized":"2024-03-26T03:59:12.505510+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755947+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":547,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447986+0000","last_change":"2024-03-26T03:55:52.900155+0000","last_active":"2024-03-26T03:59:08.447986+0000","last_peered":"2024-03-26T03:59:08.447986+0000","last_clean":"2024-03-26T03:59:08.447986+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:59:08.447986+0000","last_undegraded":"2024-03-26T03:59:08.447986+0000","last_fullsized":"2024-03-26T03:59:08.447986+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:52.900040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:52.900040+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287800+0000","last_change":"2024-03-26T03:57:49.372738+0000","last_active":"2024-03-26T03:59:04.287800+0000","last_peered":"2024-03-26T03:59:04.287800+0000","last_clean":"2024-03-26T03:59:04.287800+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T03:59:04.287800+0000","last_undegraded":"2024-03-26T03:59:04.287800+0000","last_fullsized":"2024-03-26T03:59:04.287800+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:30.756903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:30.756903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":411,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.448036+0000","last_change":"2024-03-26T03:58:25.086917+0000","last_active":"2024-03-26T03:59:08.448036+0000","last_peered":"2024-03-26T03:59:08.448036+0000","last_clean":"2024-03-26T03:59:08.448036+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:59:08.448036+0000","last_undegraded":"2024-03-26T03:59:08.448036+0000","last_fullsized":"2024-03-26T03:59:08.448036+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:25.086805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:25.086805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:33.816834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":501,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351898+0000","last_change":"2024-03-26T03:57:49.408864+0000","last_active":"2024-03-26T03:59:09.351898+0000","last_peered":"2024-03-26T03:59:09.351898+0000","last_clean":"2024-03-26T03:59:09.351898+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T03:59:09.351898+0000","last_undegraded":"2024-03-26T03:59:09.351898+0000","last_fullsized":"2024-03-26T03:59:09.351898+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373806+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":413,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351153+0000","last_change":"2024-03-26T03:58:18.180129+0000","last_active":"2024-03-26T03:59:09.351153+0000","last_peered":"2024-03-26T03:59:09.351153+0000","last_clean":"2024-03-26T03:59:09.351153+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:59:09.351153+0000","last_undegraded":"2024-03-26T03:59:09.351153+0000","last_fullsized":"2024-03-26T03:59:09.351153+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:18.179988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:18.179988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:22.302751+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":364,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351399+0000","last_change":"2024-03-26T03:58:40.502649+0000","last_active":"2024-03-26T03:59:09.351399+0000","last_peered":"2024-03-26T03:59:09.351399+0000","last_clean":"2024-03-26T03:59:09.351399+0000","last_became_active":"2024-03-26T03:58:40.502368+0000","last_became_peered":"2024-03-26T03:58:40.502368+0000","last_unstale":"2024-03-26T03:59:09.351399+0000","last_undegraded":"2024-03-26T03:59:09.351399+0000","last_fullsized":"2024-03-26T03:59:09.351399+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:21.112131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:21.112131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:24.542944+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":422,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351975+0000","last_change":"2024-03-26T03:58:16.032385+0000","last_active":"2024-03-26T03:59:09.351975+0000","last_peered":"2024-03-26T03:59:09.351975+0000","last_clean":"2024-03-26T03:59:09.351975+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T03:59:09.351975+0000","last_undegraded":"2024-03-26T03:59:09.351975+0000","last_fullsized":"2024-03-26T03:59:09.351975+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.032069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.032069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:41.260369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":394,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.506531+0000","last_change":"2024-03-26T03:58:40.500096+0000","last_active":"2024-03-26T03:59:12.506531+0000","last_peered":"2024-03-26T03:59:12.506531+0000","last_clean":"2024-03-26T03:59:12.506531+0000","last_became_active":"2024-03-26T03:58:40.499811+0000","last_became_peered":"2024-03-26T03:58:40.499811+0000","last_unstale":"2024-03-26T03:59:12.506531+0000","last_undegraded":"2024-03-26T03:59:12.506531+0000","last_fullsized":"2024-03-26T03:59:12.506531+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:29.010725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.010725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.625132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"241'41","reported_seq":451,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974259+0000","last_change":"2024-03-26T03:55:59.302041+0000","last_active":"2024-03-26T03:59:09.974259+0000","last_peered":"2024-03-26T03:59:09.974259+0000","last_clean":"2024-03-26T03:59:09.974259+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T03:59:09.974259+0000","last_undegraded":"2024-03-26T03:59:09.974259+0000","last_fullsized":"2024-03-26T03:59:09.974259+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:55:14.773103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.773103+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310127+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":393,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352330+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T03:59:09.352330+0000","last_peered":"2024-03-26T03:59:09.352330+0000","last_clean":"2024-03-26T03:59:09.352330+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:59:09.352330+0000","last_undegraded":"2024-03-26T03:59:09.352330+0000","last_fullsized":"2024-03-26T03:59:09.352330+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848933+0000","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":425,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352279+0000","last_change":"2024-03-26T03:58:10.028917+0000","last_active":"2024-03-26T03:59:09.352279+0000","last_peered":"2024-03-26T03:59:09.352279+0000","last_clean":"2024-03-26T03:59:09.352279+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T03:59:09.352279+0000","last_undegraded":"2024-03-26T03:59:09.352279+0000","last_fullsized":"2024-03-26T03:59:09.352279+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:10.028829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:58:10.028829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:15.966391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":392,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.502848+0000","last_change":"2024-03-26T03:57:49.368242+0000","last_active":"2024-03-26T03:59:12.502848+0000","last_peered":"2024-03-26T03:59:12.502848+0000","last_clean":"2024-03-26T03:59:12.502848+0000","last_became_active":"2024-03-26T03:57:49.367885+0000","last_became_peered":"2024-03-26T03:57:49.367885+0000","last_unstale":"2024-03-26T03:59:12.502848+0000","last_undegraded":"2024-03-26T03:59:12.502848+0000","last_fullsized":"2024-03-26T03:59:12.502848+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755909+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":465,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352235+0000","last_change":"2024-03-26T03:55:59.309207+0000","last_active":"2024-03-26T03:59:09.352235+0000","last_peered":"2024-03-26T03:59:09.352235+0000","last_clean":"2024-03-26T03:59:09.352235+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T03:59:09.352235+0000","last_undegraded":"2024-03-26T03:59:09.352235+0000","last_fullsized":"2024-03-26T03:59:09.352235+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:54:17.818140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.818140+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848690+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288297+0000","last_change":"2024-03-26T03:57:49.361707+0000","last_active":"2024-03-26T03:59:04.288297+0000","last_peered":"2024-03-26T03:59:04.288297+0000","last_clean":"2024-03-26T03:59:04.288297+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T03:59:04.288297+0000","last_undegraded":"2024-03-26T03:59:04.288297+0000","last_fullsized":"2024-03-26T03:59:04.288297+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134411+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"69'2","reported_seq":360,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.507601+0000","last_change":"2024-03-26T03:55:59.316898+0000","last_active":"2024-03-26T03:59:12.507601+0000","last_peered":"2024-03-26T03:59:12.507601+0000","last_clean":"2024-03-26T03:59:12.507601+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T03:59:12.507601+0000","last_undegraded":"2024-03-26T03:59:12.507601+0000","last_fullsized":"2024-03-26T03:59:12.507601+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"69'2","last_scrub_stamp":"2024-03-26T03:53:37.581310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.581310+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":411,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351871+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T03:59:09.351871+0000","last_peered":"2024-03-26T03:59:09.351871+0000","last_clean":"2024-03-26T03:59:09.351871+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:59:09.351871+0000","last_undegraded":"2024-03-26T03:59:09.351871+0000","last_fullsized":"2024-03-26T03:59:09.351871+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373904+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"241'48","reported_seq":663,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973467+0000","last_change":"2024-03-26T03:57:49.488773+0000","last_active":"2024-03-26T03:59:09.973467+0000","last_peered":"2024-03-26T03:59:09.973467+0000","last_clean":"2024-03-26T03:59:09.973467+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T03:59:09.973467+0000","last_undegraded":"2024-03-26T03:59:09.973467+0000","last_fullsized":"2024-03-26T03:59:09.973467+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'34","last_scrub_stamp":"2024-03-26T03:54:17.896827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.896827+0000","objects_scrubbed":11,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.309970+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352558+0000","last_change":"2024-03-26T03:54:25.806373+0000","last_active":"2024-03-26T03:59:09.352558+0000","last_peered":"2024-03-26T03:59:09.352558+0000","last_clean":"2024-03-26T03:59:09.352558+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:59:09.352558+0000","last_undegraded":"2024-03-26T03:59:09.352558+0000","last_fullsized":"2024-03-26T03:59:09.352558+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.806277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.806277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":426,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.448249+0000","last_change":"2024-03-26T03:58:41.051187+0000","last_active":"2024-03-26T03:59:08.448249+0000","last_peered":"2024-03-26T03:59:08.448249+0000","last_clean":"2024-03-26T03:59:08.448249+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T03:59:08.448249+0000","last_undegraded":"2024-03-26T03:59:08.448249+0000","last_fullsized":"2024-03-26T03:59:08.448249+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.050966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.050966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:48.348801+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":418,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973297+0000","last_change":"2024-03-26T03:55:29.863335+0000","last_active":"2024-03-26T03:59:09.973297+0000","last_peered":"2024-03-26T03:59:09.973297+0000","last_clean":"2024-03-26T03:59:09.973297+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:59:09.973297+0000","last_undegraded":"2024-03-26T03:59:09.973297+0000","last_fullsized":"2024-03-26T03:59:09.973297+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.863201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.863201+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310262+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":443,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351703+0000","last_change":"2024-03-26T03:58:11.071645+0000","last_active":"2024-03-26T03:59:09.351703+0000","last_peered":"2024-03-26T03:59:09.351703+0000","last_clean":"2024-03-26T03:59:09.351703+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T03:59:09.351703+0000","last_undegraded":"2024-03-26T03:59:09.351703+0000","last_fullsized":"2024-03-26T03:59:09.351703+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.071522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.071522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:31.854810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'23","reported_seq":453,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.975100+0000","last_change":"2024-03-26T03:58:29.467581+0000","last_active":"2024-03-26T03:59:09.975100+0000","last_peered":"2024-03-26T03:59:09.975100+0000","last_clean":"2024-03-26T03:59:09.975100+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T03:59:09.975100+0000","last_undegraded":"2024-03-26T03:59:09.975100+0000","last_fullsized":"2024-03-26T03:59:09.975100+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:58:29.467279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.467279+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:36.091054+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":414,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353120+0000","last_change":"2024-03-26T03:58:08.985333+0000","last_active":"2024-03-26T03:59:09.353120+0000","last_peered":"2024-03-26T03:59:09.353120+0000","last_clean":"2024-03-26T03:59:09.353120+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T03:59:09.353120+0000","last_undegraded":"2024-03-26T03:59:09.353120+0000","last_fullsized":"2024-03-26T03:59:09.353120+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:08.985207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:08.985207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:34.567248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":366,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503601+0000","last_change":"2024-03-26T03:57:49.368565+0000","last_active":"2024-03-26T03:59:12.503601+0000","last_peered":"2024-03-26T03:59:12.503601+0000","last_clean":"2024-03-26T03:59:12.503601+0000","last_became_active":"2024-03-26T03:57:49.367895+0000","last_became_peered":"2024-03-26T03:57:49.367895+0000","last_unstale":"2024-03-26T03:59:12.503601+0000","last_undegraded":"2024-03-26T03:59:12.503601+0000","last_fullsized":"2024-03-26T03:59:12.503601+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:25.552105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:25.552105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755789+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":676,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447236+0000","last_change":"2024-03-26T03:58:39.055551+0000","last_active":"2024-03-26T03:59:08.447236+0000","last_peered":"2024-03-26T03:59:08.447236+0000","last_clean":"2024-03-26T03:59:08.447236+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:59:08.447236+0000","last_undegraded":"2024-03-26T03:59:08.447236+0000","last_fullsized":"2024-03-26T03:59:08.447236+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:58:39.055403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.055403+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:01.714922+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287625+0000","last_change":"2024-03-26T03:57:49.369789+0000","last_active":"2024-03-26T03:59:04.287625+0000","last_peered":"2024-03-26T03:59:04.287625+0000","last_clean":"2024-03-26T03:59:04.287625+0000","last_became_active":"2024-03-26T03:57:49.369463+0000","last_became_peered":"2024-03-26T03:57:49.369463+0000","last_unstale":"2024-03-26T03:59:04.287625+0000","last_undegraded":"2024-03-26T03:59:04.287625+0000","last_fullsized":"2024-03-26T03:59:04.287625+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.756123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.756123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":516,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352691+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T03:59:09.352691+0000","last_peered":"2024-03-26T03:59:09.352691+0000","last_clean":"2024-03-26T03:59:09.352691+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:59:09.352691+0000","last_undegraded":"2024-03-26T03:59:09.352691+0000","last_fullsized":"2024-03-26T03:59:09.352691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373927+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":296,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352803+0000","last_change":"2024-03-26T03:58:40.497788+0000","last_active":"2024-03-26T03:59:09.352803+0000","last_peered":"2024-03-26T03:59:09.352803+0000","last_clean":"2024-03-26T03:59:09.352803+0000","last_became_active":"2024-03-26T03:58:40.497499+0000","last_became_peered":"2024-03-26T03:58:40.497499+0000","last_unstale":"2024-03-26T03:59:09.352803+0000","last_undegraded":"2024-03-26T03:59:09.352803+0000","last_fullsized":"2024-03-26T03:59:09.352803+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:13.093867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:58:13.093867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:22.696380+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":412,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974456+0000","last_change":"2024-03-26T03:58:40.462393+0000","last_active":"2024-03-26T03:59:09.974456+0000","last_peered":"2024-03-26T03:59:09.974456+0000","last_clean":"2024-03-26T03:59:09.974456+0000","last_became_active":"2024-03-26T03:57:49.379569+0000","last_became_peered":"2024-03-26T03:57:49.379569+0000","last_unstale":"2024-03-26T03:59:09.974456+0000","last_undegraded":"2024-03-26T03:59:09.974456+0000","last_fullsized":"2024-03-26T03:59:09.974456+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:40.462316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:40.462316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:55.226953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":435,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352115+0000","last_change":"2024-03-26T03:57:49.364364+0000","last_active":"2024-03-26T03:59:09.352115+0000","last_peered":"2024-03-26T03:59:09.352115+0000","last_clean":"2024-03-26T03:59:09.352115+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T03:59:09.352115+0000","last_undegraded":"2024-03-26T03:59:09.352115+0000","last_fullsized":"2024-03-26T03:59:09.352115+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848947+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1041,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447258+0000","last_change":"2024-03-26T03:55:58.942740+0000","last_active":"2024-03-26T03:59:08.447258+0000","last_peered":"2024-03-26T03:59:08.447258+0000","last_clean":"2024-03-26T03:59:08.447258+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:59:08.447258+0000","last_undegraded":"2024-03-26T03:59:08.447258+0000","last_fullsized":"2024-03-26T03:59:08.447258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:55:58.942635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.942635+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":617,"num_read_kb":612,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":437,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353523+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T03:59:09.353523+0000","last_peered":"2024-03-26T03:59:09.353523+0000","last_clean":"2024-03-26T03:59:09.353523+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:59:09.353523+0000","last_undegraded":"2024-03-26T03:59:09.353523+0000","last_fullsized":"2024-03-26T03:59:09.353523+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":347,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503882+0000","last_change":"2024-03-26T03:55:59.311162+0000","last_active":"2024-03-26T03:59:12.503882+0000","last_peered":"2024-03-26T03:59:12.503882+0000","last_clean":"2024-03-26T03:59:12.503882+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T03:59:12.503882+0000","last_undegraded":"2024-03-26T03:59:12.503882+0000","last_fullsized":"2024-03-26T03:59:12.503882+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.750684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.750684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973304+0000","last_change":"2024-03-26T03:55:15.751265+0000","last_active":"2024-03-26T03:59:09.973304+0000","last_peered":"2024-03-26T03:59:09.973304+0000","last_clean":"2024-03-26T03:59:09.973304+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:59:09.973304+0000","last_undegraded":"2024-03-26T03:59:09.973304+0000","last_fullsized":"2024-03-26T03:59:09.973304+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:15.751136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:15.751136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310159+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":421,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974400+0000","last_change":"2024-03-26T03:58:41.426711+0000","last_active":"2024-03-26T03:59:09.974400+0000","last_peered":"2024-03-26T03:59:09.974400+0000","last_clean":"2024-03-26T03:59:09.974400+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T03:59:09.974400+0000","last_undegraded":"2024-03-26T03:59:09.974400+0000","last_fullsized":"2024-03-26T03:59:09.974400+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.426593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.426593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:06.577500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"241'35","reported_seq":473,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352027+0000","last_change":"2024-03-26T03:55:59.309535+0000","last_active":"2024-03-26T03:59:09.352027+0000","last_peered":"2024-03-26T03:59:09.352027+0000","last_clean":"2024-03-26T03:59:09.352027+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T03:59:09.352027+0000","last_undegraded":"2024-03-26T03:59:09.352027+0000","last_fullsized":"2024-03-26T03:59:09.352027+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'25","last_scrub_stamp":"2024-03-26T03:54:22.831125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:22.831125+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848708+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":345,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351723+0000","last_change":"2024-03-26T03:55:59.308367+0000","last_active":"2024-03-26T03:59:09.351723+0000","last_peered":"2024-03-26T03:59:09.351723+0000","last_clean":"2024-03-26T03:59:09.351723+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T03:59:09.351723+0000","last_undegraded":"2024-03-26T03:59:09.351723+0000","last_fullsized":"2024-03-26T03:59:09.351723+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.813687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.813687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848725+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":407,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351267+0000","last_change":"2024-03-26T03:57:49.378513+0000","last_active":"2024-03-26T03:59:09.351267+0000","last_peered":"2024-03-26T03:59:09.351267+0000","last_clean":"2024-03-26T03:59:09.351267+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T03:59:09.351267+0000","last_undegraded":"2024-03-26T03:59:09.351267+0000","last_fullsized":"2024-03-26T03:59:09.351267+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:16.572888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:53:16.572888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848598+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":378,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.506400+0000","last_change":"2024-03-26T03:57:49.368358+0000","last_active":"2024-03-26T03:59:12.506400+0000","last_peered":"2024-03-26T03:59:12.506400+0000","last_clean":"2024-03-26T03:59:12.506400+0000","last_became_active":"2024-03-26T03:57:49.367990+0000","last_became_peered":"2024-03-26T03:57:49.367990+0000","last_unstale":"2024-03-26T03:59:12.506400+0000","last_undegraded":"2024-03-26T03:59:12.506400+0000","last_fullsized":"2024-03-26T03:59:12.506400+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755965+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":450,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974426+0000","last_change":"2024-03-26T03:55:59.299304+0000","last_active":"2024-03-26T03:59:09.974426+0000","last_peered":"2024-03-26T03:59:09.974426+0000","last_clean":"2024-03-26T03:59:09.974426+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T03:59:09.974426+0000","last_undegraded":"2024-03-26T03:59:09.974426+0000","last_fullsized":"2024-03-26T03:59:09.974426+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:15.883507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.883507+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.309855+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"215'13","reported_seq":564,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351793+0000","last_change":"2024-03-26T03:57:49.447118+0000","last_active":"2024-03-26T03:59:09.351793+0000","last_peered":"2024-03-26T03:59:09.351793+0000","last_clean":"2024-03-26T03:59:09.351793+0000","last_became_active":"2024-03-26T03:57:49.361026+0000","last_became_peered":"2024-03-26T03:57:49.361026+0000","last_unstale":"2024-03-26T03:59:09.351793+0000","last_undegraded":"2024-03-26T03:59:09.351793+0000","last_fullsized":"2024-03-26T03:59:09.351793+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:53:29.559353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:53:29.559353+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848638+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":405,"reported_epoch":248,"state":"active+clean","last_fresh":"2024-03-26T03:59:08.447733+0000","last_change":"2024-03-26T03:58:23.160475+0000","last_active":"2024-03-26T03:59:08.447733+0000","last_peered":"2024-03-26T03:59:08.447733+0000","last_clean":"2024-03-26T03:59:08.447733+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:59:08.447733+0000","last_undegraded":"2024-03-26T03:59:08.447733+0000","last_fullsized":"2024-03-26T03:59:08.447733+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:23.160383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:58:23.160383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:31.341937+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":406,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353522+0000","last_change":"2024-03-26T03:53:27.558266+0000","last_active":"2024-03-26T03:59:09.353522+0000","last_peered":"2024-03-26T03:59:09.353522+0000","last_clean":"2024-03-26T03:59:09.353522+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:59:09.353522+0000","last_undegraded":"2024-03-26T03:59:09.353522+0000","last_fullsized":"2024-03-26T03:59:09.353522+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:53:27.558020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:53:27.558020+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:12.848619+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":489,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351936+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T03:59:09.351936+0000","last_peered":"2024-03-26T03:59:09.351936+0000","last_clean":"2024-03-26T03:59:09.351936+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T03:59:09.351936+0000","last_undegraded":"2024-03-26T03:59:09.351936+0000","last_fullsized":"2024-03-26T03:59:09.351936+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":96,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287436+0000","last_change":"2024-03-26T03:57:49.461921+0000","last_active":"2024-03-26T03:59:04.287436+0000","last_peered":"2024-03-26T03:59:04.287436+0000","last_clean":"2024-03-26T03:59:04.287436+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T03:59:04.287436+0000","last_undegraded":"2024-03-26T03:59:04.287436+0000","last_fullsized":"2024-03-26T03:59:04.287436+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":397,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351792+0000","last_change":"2024-03-26T03:57:49.360270+0000","last_active":"2024-03-26T03:59:09.351792+0000","last_peered":"2024-03-26T03:59:09.351792+0000","last_clean":"2024-03-26T03:59:09.351792+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T03:59:09.351792+0000","last_undegraded":"2024-03-26T03:59:09.351792+0000","last_fullsized":"2024-03-26T03:59:09.351792+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288471+0000","last_change":"2024-03-26T03:57:49.362068+0000","last_active":"2024-03-26T03:59:04.288471+0000","last_peered":"2024-03-26T03:59:04.288471+0000","last_clean":"2024-03-26T03:59:04.288471+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T03:59:04.288471+0000","last_undegraded":"2024-03-26T03:59:04.288471+0000","last_fullsized":"2024-03-26T03:59:04.288471+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:52:45.556274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:52:45.556274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":115,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287607+0000","last_change":"2024-03-26T03:57:49.531398+0000","last_active":"2024-03-26T03:59:04.287607+0000","last_peered":"2024-03-26T03:59:04.287607+0000","last_clean":"2024-03-26T03:59:04.287607+0000","last_became_active":"2024-03-26T03:57:49.398824+0000","last_became_peered":"2024-03-26T03:57:49.398824+0000","last_unstale":"2024-03-26T03:59:04.287607+0000","last_undegraded":"2024-03-26T03:59:04.287607+0000","last_fullsized":"2024-03-26T03:59:04.287607+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'40","last_scrub_stamp":"2024-03-26T03:54:27.003417+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.003417+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":409,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351492+0000","last_change":"2024-03-26T03:57:49.360193+0000","last_active":"2024-03-26T03:59:09.351492+0000","last_peered":"2024-03-26T03:59:09.351492+0000","last_clean":"2024-03-26T03:59:09.351492+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T03:59:09.351492+0000","last_undegraded":"2024-03-26T03:59:09.351492+0000","last_fullsized":"2024-03-26T03:59:09.351492+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373765+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":325,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.506257+0000","last_change":"2024-03-26T03:55:59.341468+0000","last_active":"2024-03-26T03:59:12.506257+0000","last_peered":"2024-03-26T03:59:12.506257+0000","last_clean":"2024-03-26T03:59:12.506257+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T03:59:12.506257+0000","last_undegraded":"2024-03-26T03:59:12.506257+0000","last_fullsized":"2024-03-26T03:59:12.506257+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:53:28.635838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":542,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352559+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T03:59:09.352559+0000","last_peered":"2024-03-26T03:59:09.352559+0000","last_clean":"2024-03-26T03:59:09.352559+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T03:59:09.352559+0000","last_undegraded":"2024-03-26T03:59:09.352559+0000","last_fullsized":"2024-03-26T03:59:09.352559+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373630+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287684+0000","last_change":"2024-03-26T03:57:49.373904+0000","last_active":"2024-03-26T03:59:04.287684+0000","last_peered":"2024-03-26T03:59:04.287684+0000","last_clean":"2024-03-26T03:59:04.287684+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T03:59:04.287684+0000","last_undegraded":"2024-03-26T03:59:04.287684+0000","last_fullsized":"2024-03-26T03:59:04.287684+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.922297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":494,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.973677+0000","last_change":"2024-03-26T03:58:42.470991+0000","last_active":"2024-03-26T03:59:09.973677+0000","last_peered":"2024-03-26T03:59:09.973677+0000","last_clean":"2024-03-26T03:59:09.973677+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T03:59:09.973677+0000","last_undegraded":"2024-03-26T03:59:09.973677+0000","last_fullsized":"2024-03-26T03:59:09.973677+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:42.470857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.470857+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:45.803581+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.288451+0000","last_change":"2024-03-26T03:57:49.373101+0000","last_active":"2024-03-26T03:59:04.288451+0000","last_peered":"2024-03-26T03:59:04.288451+0000","last_clean":"2024-03-26T03:59:04.288451+0000","last_became_active":"2024-03-26T03:57:49.372744+0000","last_became_peered":"2024-03-26T03:57:49.372744+0000","last_unstale":"2024-03-26T03:59:04.288451+0000","last_undegraded":"2024-03-26T03:59:04.288451+0000","last_fullsized":"2024-03-26T03:59:04.288451+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":367,"reported_epoch":251,"state":"active+clean","last_fresh":"2024-03-26T03:59:12.503520+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T03:59:12.503520+0000","last_peered":"2024-03-26T03:59:12.503520+0000","last_clean":"2024-03-26T03:59:12.503520+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T03:59:12.503520+0000","last_undegraded":"2024-03-26T03:59:12.503520+0000","last_fullsized":"2024-03-26T03:59:12.503520+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:13.755927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":405,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351634+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:59:09.351634+0000","last_peered":"2024-03-26T03:59:09.351634+0000","last_clean":"2024-03-26T03:59:09.351634+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:59:09.351634+0000","last_undegraded":"2024-03-26T03:59:09.351634+0000","last_fullsized":"2024-03-26T03:59:09.351634+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":513,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352436+0000","last_change":"2024-03-26T03:57:49.495410+0000","last_active":"2024-03-26T03:59:09.352436+0000","last_peered":"2024-03-26T03:59:09.352436+0000","last_clean":"2024-03-26T03:59:09.352436+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T03:59:09.352436+0000","last_undegraded":"2024-03-26T03:59:09.352436+0000","last_fullsized":"2024-03-26T03:59:09.352436+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:53:37.345803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:53:37.345803+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373437+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":429,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.353148+0000","last_change":"2024-03-26T03:58:31.314191+0000","last_active":"2024-03-26T03:59:09.353148+0000","last_peered":"2024-03-26T03:59:09.353148+0000","last_clean":"2024-03-26T03:59:09.353148+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T03:59:09.353148+0000","last_undegraded":"2024-03-26T03:59:09.353148+0000","last_fullsized":"2024-03-26T03:59:09.353148+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:31.314073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:31.314073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:57.668269+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":395,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.974663+0000","last_change":"2024-03-26T03:54:27.851354+0000","last_active":"2024-03-26T03:59:09.974663+0000","last_peered":"2024-03-26T03:59:09.974663+0000","last_clean":"2024-03-26T03:59:09.974663+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:59:09.974663+0000","last_undegraded":"2024-03-26T03:59:09.974663+0000","last_fullsized":"2024-03-26T03:59:09.974663+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:27.851270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:27.851270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:14.310013+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":356,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351831+0000","last_change":"2024-03-26T03:57:49.360677+0000","last_active":"2024-03-26T03:59:09.351831+0000","last_peered":"2024-03-26T03:59:09.351831+0000","last_clean":"2024-03-26T03:59:09.351831+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T03:59:09.351831+0000","last_undegraded":"2024-03-26T03:59:09.351831+0000","last_fullsized":"2024-03-26T03:59:09.351831+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.312193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.312193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373554+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":450,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352839+0000","last_change":"2024-03-26T03:54:13.380126+0000","last_active":"2024-03-26T03:59:09.352839+0000","last_peered":"2024-03-26T03:59:09.352839+0000","last_clean":"2024-03-26T03:59:09.352839+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:59:09.352839+0000","last_undegraded":"2024-03-26T03:59:09.352839+0000","last_fullsized":"2024-03-26T03:59:09.352839+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-03-26T03:54:13.380006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:13.380006+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373534+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":393,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352628+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:59:09.352628+0000","last_peered":"2024-03-26T03:59:09.352628+0000","last_clean":"2024-03-26T03:59:09.352628+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:59:09.352628+0000","last_undegraded":"2024-03-26T03:59:09.352628+0000","last_fullsized":"2024-03-26T03:59:09.352628+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373592+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":558,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.351678+0000","last_change":"2024-03-26T03:58:33.177842+0000","last_active":"2024-03-26T03:59:09.351678+0000","last_peered":"2024-03-26T03:59:09.351678+0000","last_clean":"2024-03-26T03:59:09.351678+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T03:59:09.351678+0000","last_undegraded":"2024-03-26T03:59:09.351678+0000","last_fullsized":"2024-03-26T03:59:09.351678+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:58:33.177720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:33.177720+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:39.489885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":75,"reported_epoch":247,"state":"active+clean","last_fresh":"2024-03-26T03:59:04.287735+0000","last_change":"2024-03-26T03:57:49.360595+0000","last_active":"2024-03-26T03:59:04.287735+0000","last_peered":"2024-03-26T03:59:04.287735+0000","last_clean":"2024-03-26T03:59:04.287735+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T03:59:04.287735+0000","last_undegraded":"2024-03-26T03:59:04.287735+0000","last_fullsized":"2024-03-26T03:59:04.287735+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:09.134394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":485,"reported_epoch":249,"state":"active+clean","last_fresh":"2024-03-26T03:59:09.352288+0000","last_change":"2024-03-26T03:57:49.507032+0000","last_active":"2024-03-26T03:59:09.352288+0000","last_peered":"2024-03-26T03:59:09.352288+0000","last_clean":"2024-03-26T03:59:09.352288+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T03:59:09.352288+0000","last_undegraded":"2024-03-26T03:59:09.352288+0000","last_fullsized":"2024-03-26T03:59:09.352288+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:10.373826+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":617,"num_read_kb":612,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1408,"num_read_kb":1133,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070593,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":239832,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93505320,"statfs":{"total":95995035648,"available":95749447680,"internally_reserved":0,"allocated":1847296,"data_stored":1268051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":243726794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364935,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575008,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":343996,"kb_used_data":1840,"kb_used_omap":1,"kb_used_meta":342142,"kb_avail":93401156,"statfs":{"total":95995035648,"available":95642783744,"internally_reserved":0,"allocated":1884160,"data_stored":1274122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":350353861},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738533,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362036,"kb_used_data":1832,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":93383116,"statfs":{"total":95995035648,"available":95624310784,"internally_reserved":0,"allocated":1875968,"data_stored":1275091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411483,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":401672,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93343480,"statfs":{"total":95995035648,"available":95583723520,"internally_reserved":0,"allocated":1236992,"data_stored":682427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182549,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303132,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93442020,"statfs":{"total":95995035648,"available":95684628480,"internally_reserved":0,"allocated":1261568,"data_stored":670811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953616,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":451752,"kb_used_data":117912,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":93293400,"statfs":{"total":95995035648,"available":95532441600,"internally_reserved":0,"allocated":120741888,"data_stored":120155045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113320,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":287476,"kb_used_data":1252,"kb_used_omap":4,"kb_used_meta":286203,"kb_avail":93457676,"statfs":{"total":95995035648,"available":95700660224,"internally_reserved":0,"allocated":1282048,"data_stored":696837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":293072544},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:59:17.389 INFO:tasks.thrashosds.thrasher:Setting 3.12 to [6, 7, 4, 1, 2, 0] 2024-03-26T03:59:17.389 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.12', '6', '7', '4', '1', '2', '0'] 2024-03-26T03:59:17.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 3.12 6 7 4 1 2 0 2024-03-26T03:59:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.403 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:17.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:17.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:17.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:17.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T03:59:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 4.10 scrub ok 2024-03-26T03:59:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T03:59:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T03:59:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T03:59:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T03:59:17.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:17 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3251362079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:17.622 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:17.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:17.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 4.10 scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3251362079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 4.10 scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T03:59:17.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:17 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3251362079' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:17.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:17.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:17.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:17.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:17.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:17.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:17.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:17.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:17.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:18.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:18.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:18.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:18.193 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:18.193 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:18.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:18.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:18.332 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:18.332 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:18.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:18.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:18.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:18.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:18.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:18.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:18.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:18.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:18.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: osdmap e252: 8 total, 8 up, 7 in 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T03:59:18.797 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:18 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T03:59:18.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T03:59:18.886 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: osdmap e252: 8 total, 8 up, 7 in 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:18.887 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: pgmap v473: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 0 op/s 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: osdmap e252: 8 total, 8 up, 7 in 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T03:59:18.888 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:18 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T03:59:18.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:18.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:18.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:18.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:19.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:19.054 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:19.104 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:19.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:19.180 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:19.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:19.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:19.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:19.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:19.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:19.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:19.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T03:59:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T03:59:19.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T03:59:19.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T03:59:19.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T03:59:19.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T03:59:19.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:19 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T03:59:19.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:19.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:19.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:19.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:19.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T03:59:19.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T03:59:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T03:59:19.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:19 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T03:59:19.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:19.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:19.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:20.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:59:20.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:20.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:20.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:20.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:20.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:20.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:20.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:20.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:20.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:20.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:20.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:20.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.562 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:20.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:20.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:20.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:20.750 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:20.751 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:20.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:20 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T03:59:20.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:20 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T03:59:20.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:20 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T03:59:20.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:20 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T03:59:20.751 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:20 smithi086 ceph-mon[32965]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:20.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:20.813 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[37501]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T03:59:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T03:59:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T03:59:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T03:59:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:20 smithi006 ceph-mon[29446]: pgmap v475: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:20.915 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:20.938 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:20.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:20.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:21.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:21.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:21.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:21.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:21.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:21.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:21.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:21.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T03:59:21.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:21.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:21.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:21.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:21.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:21.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:21 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T03:59:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:21 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T03:59:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:21 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T03:59:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:21 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T03:59:21.638 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:21.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:21.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:21.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:21.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:21.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T03:59:21.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T03:59:21.875 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:21 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T03:59:21.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:21.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:21.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:21.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:22.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:22.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:22.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:22.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:22.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:22.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:22.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:22.443 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:22.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:22.525 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T03:59:22.525 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T03:59:22.525 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T03:59:22.525 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T03:59:22.526 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:59:22.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:22.591 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:22.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:22.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:22.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:22.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T03:59:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: pgmap v476: 132 pgs: 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T03:59:22.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-26T03:59:22.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:22.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:22.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:22.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:22.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:22.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:22.987 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_2' created 2024-03-26T03:59:23.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:23.092 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:23.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:23.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:23.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:23.246 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:23.246 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:23.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:23.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:23.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:23.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:23.522 INFO:teuthology.orchestra.run.smithi006.stderr:set 3.12 pg_upmap_items mapping to [6->7,4->1,2->0] 2024-03-26T03:59:23.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:23.595 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:23.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:23.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: osdmap e253: 8 total, 8 up, 7 in 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2218011103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:23.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: osdmap e253: 8 total, 8 up, 7 in 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2218011103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T03:59:23.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:23 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:23.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:23.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:23.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:23.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T03:59:23.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-26T03:59:23.809 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T03:59:23.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:23.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T03:59:23.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T03:59:23.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T03:59:23.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T03:59:23.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2283619688' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: osdmap e253: 8 total, 8 up, 7 in 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2218011103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]: dispatch 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T03:59:23.824 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:23 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:23.847 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:23.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:23.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:23.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:23.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:24.041 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:24.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:24.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:24.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:24.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:24.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:24.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:24.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:24.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:24.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:24.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:24.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:24.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:24.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: pgmap v478: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:59:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]': finished 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: osdmap e254: 8 total, 8 up, 7 in 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: pgmap v478: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:59:24.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]': finished 2024-03-26T03:59:24.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:24.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: osdmap e254: 8 total, 8 up, 7 in 2024-03-26T03:59:24.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:24.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:24.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:24.707 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:24.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:24.790 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T03:59:24.790 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T03:59:24.790 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T03:59:24.790 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: pgmap v478: 148 pgs: 4 creating+peering, 12 unknown, 132 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.12", "id": [6, 7, 4, 1, 2, 0]}]': finished 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2837505449' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: osdmap e254: 8 total, 8 up, 7 in 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:24.791 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:24.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:24.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:24.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:24.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:24.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:24.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:24.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:25.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:25.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:25.105 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:25.171 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:25.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:25.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:25.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:25.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:25.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:25.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:25.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:25.611 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:25.611 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: osdmap e255: 8 total, 8 up, 7 in 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T03:59:25.616 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T03:59:25.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T03:59:25.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T03:59:25.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: osdmap e255: 8 total, 8 up, 7 in 2024-03-26T03:59:25.617 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:25.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T03:59:25.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T03:59:25.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T03:59:25.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T03:59:25.621 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: osdmap e255: 8 total, 8 up, 7 in 2024-03-26T03:59:25.622 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:25.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:25.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:25.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:25.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:25.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:25.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:25.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:26.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:26.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:26.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:26.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T03:59:26.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:26.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:26.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:26.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:26.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:26.441 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:26.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:26.485 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:26.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:26.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:26.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:26.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:26.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: pgmap v481: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T03:59:26.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:26 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T03:59:26.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:26.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:26.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:26.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: pgmap v481: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:59:26.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: pgmap v481: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T03:59:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:26 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T03:59:26.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:26.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:26.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:27.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:27.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T03:59:27.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.114 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:27.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:27.238 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:27.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:27.364 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:27.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:27.494 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: 3.19 deep-scrub starts 2024-03-26T03:59:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: 3.19 deep-scrub ok 2024-03-26T03:59:27.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:27.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:27.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T03:59:27.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T03:59:27.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:27 smithi086 ceph-mon[32965]: osdmap e256: 8 total, 8 up, 7 in 2024-03-26T03:59:27.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:27.619 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: 3.19 deep-scrub starts 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: 3.19 deep-scrub ok 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T03:59:27.713 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[37501]: osdmap e256: 8 total, 8 up, 7 in 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: 3.19 deep-scrub starts 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: 3.19 deep-scrub ok 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T03:59:27.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:27 smithi006 ceph-mon[29446]: osdmap e256: 8 total, 8 up, 7 in 2024-03-26T03:59:27.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:27.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:27.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:27.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:27.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:27.996 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:27.996 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:27.998 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:27.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:28.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:28.122 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:28.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:28.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:28.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:28.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:28.368 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:59:28.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:28.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:28.387 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:59:28.406 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:59:28.426 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:59:28.446 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:59:28.450 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:28.475 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:59:28.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:28.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:28.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:28.500 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:59:28.520 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:59:28.522 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:59:28.544 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-26T03:59:28.578 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 3.8 scrub starts 2024-03-26T03:59:28.578 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 3.8 scrub ok 2024-03-26T03:59:28.578 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: pgmap v483: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 2.2 deep-scrub starts 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 2.2 deep-scrub ok 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4143049337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2883073347' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 3.8 scrub starts 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 3.8 scrub ok 2024-03-26T03:59:28.579 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: pgmap v483: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 2.2 deep-scrub starts 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 2.2 deep-scrub ok 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4143049337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2883073347' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.580 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:28 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 3.8 scrub starts 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 3.8 scrub ok 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: pgmap v483: 148 pgs: 4 creating+peering, 144 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 2.2 deep-scrub starts 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 2.2 deep-scrub ok 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T03:59:28.594 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T03:59:28.595 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4143049337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:28.595 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2883073347' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.595 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:28 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T03:59:28.600 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:28.625 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:28.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:28.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T03:59:28.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:28.750 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:28.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:28.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:28.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:28.936 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:28.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:29.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:29.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:29.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:29.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:29.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:29.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:29.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:29.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:29.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:29.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:29.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:29.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:29.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:29.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T03:59:29.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T03:59:29.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[29446]: osdmap e257: 8 total, 8 up, 7 in 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:59:29.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:29 smithi006 ceph-mon[37501]: osdmap e257: 8 total, 8 up, 7 in 2024-03-26T03:59:29.690 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T03:59:29.691 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T03:59:29.691 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-26T03:59:29.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T03:59:29.729 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:29 smithi086 ceph-mon[32965]: osdmap e257: 8 total, 8 up, 7 in 2024-03-26T03:59:29.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:29.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:29.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:29.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:29.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:29.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:29.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:29.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:30.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:30.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:30.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:30.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:30.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:30.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:30.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:30.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:30.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:30.512 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:30.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:30.563 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:59:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:59:30.564 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-03-26T03:59:30.564 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:59:30.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:30.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:30.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:30.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:30.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:30.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:30.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:59:30.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:30 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T03:59:30.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T03:59:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T03:59:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: pgmap v485: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T03:59:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T03:59:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T03:59:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T03:59:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:30 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T03:59:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:30.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:30.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:30.993 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.017 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:31.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:31.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:31.141 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:31.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T03:59:31.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:31.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:31.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:31.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:31.419 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:31.431 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:31.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:31.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:31.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T03:59:31.553 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T03:59:31.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T03:59:31.554 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:31 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T03:59:31.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:31.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:31.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:31.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:31.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T03:59:31.814 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:31 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T03:59:31.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:31.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:31.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:31.959 DEBUG:teuthology.orchestra.run.smithi006:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-26T03:59:31.961 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T03:59:31.962 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T03:59:31.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:32.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:59:32.068 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:hints = 1 2024-03-26T03:59:32.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T03:59:32.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object prefix: benchmark_data_smithi006_132277 2024-03-26T03:59:32.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:59:32.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 0 0 0 0 0 0 - 0 2024-03-26T03:59:32.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:32.169 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:32.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:32.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:32.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:32.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:32.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:32.573 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T03:59:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.586 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:32.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:32.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:32.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:32.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:32 smithi086 ceph-mon[32965]: pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 743 B/s rd, 0 op/s 2024-03-26T03:59:32.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:32 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T03:59:32.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:32 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T03:59:32.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:32 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3453765852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.820 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:32 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:32.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:32.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:32.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:32.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[29446]: pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 743 B/s rd, 0 op/s 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3453765852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[37501]: pgmap v486: 148 pgs: 148 active+clean; 582 KiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 743 B/s rd, 0 op/s 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3453765852' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:32 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T03:59:32.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.005 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:33.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:33.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 1 15 3193 3178 198.613 198.625 0.00417409 0.00501254 2024-03-26T03:59:33.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.147 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:33.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:33.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:33.277 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:59:33.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T03:59:33.278 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T03:59:33.278 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T03:59:33.278 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:59:33.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:33.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:33.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:33.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:33.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:33.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:33.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:33.704 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:33.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:33.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:33.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:33 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T03:59:33.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:33 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T03:59:33.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:33 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:33.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:33 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:59:33.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:33 smithi086 ceph-mon[32965]: osdmap e258: 8 total, 8 up, 7 in 2024-03-26T03:59:33.848 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:33.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T03:59:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[37501]: osdmap e258: 8 total, 8 up, 7 in 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T03:59:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:33 smithi006 ceph-mon[29446]: osdmap e258: 8 total, 8 up, 7 in 2024-03-26T03:59:33.891 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:59:33.892 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 258, 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-03-26T03:43:14.534401+0000", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-03-26T03:59:32.546407+0000", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-03-26T03:55:58.386605+0000", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-03-26T03:57:48.463474+0000", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5801984, 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "noscrub", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-03-26T03:59:33.893 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 19, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 9, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 8, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "luminous", 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-03-26T03:59:33.894 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:46:30.183658+0000", 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.895 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 1, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 1, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 1, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 1, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 1, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.896 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "165", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.897 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.898 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.899 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 7.8899998664855957, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 7.8899998664855957, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.37999999523162842, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 3, 2024-03-26T03:59:33.900 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 3, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "datapool", 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:41.622497+0000", 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.901 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 3, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 3, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 3, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 3, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 3, 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.902 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 69, 2024-03-26T03:59:33.903 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.904 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.905 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.6700000762939453, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.6700000762939453, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.75, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.906 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".rgw.root", 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:42.191818+0000", 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.907 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.908 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "67", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.909 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.910 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.911 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.912 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 4, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.log", 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:44.079106+0000", 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.913 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.914 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.915 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.916 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.917 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4900000095367432, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4900000095367432, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 2.190000057220459, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 2.190000057220459, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 5, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.control", 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:46.035261+0000", 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.919 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "71", 2024-03-26T03:59:33.920 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.921 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.922 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.923 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.4900000095367432, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.4900000095367432, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.309999942779541, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.309999942779541, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.924 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 6, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.meta", 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:48.086810+0000", 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.925 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.926 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "73", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.927 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_bias": 4 2024-03-26T03:59:33.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.7400000095367432, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.7400000095367432, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.5299999713897705, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.5299999713897705, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.930 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 9, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_2", 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:59:22.209060+0000", 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T03:59:33.931 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 16, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 16, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 16, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 16, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 16, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T03:59:33.932 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "257", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T03:59:33.933 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T03:59:33.934 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: "rados": {} 2024-03-26T03:59:33.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.9900000095367432, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.9900000095367432, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.87999999523162842, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.75, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.75, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T03:59:33.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e", 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 8, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 253, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.938 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-03-26T03:59:33.939 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/1246039834", 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6802/1246039834", 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6804/1246039834", 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6803/1246039834", 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.940 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "753f7459-62ec-447b-a0da-64d06e813f16", 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 13, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 253, 2024-03-26T03:59:33.941 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.942 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6805/260311411", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6806/260311411", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6808/260311411", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6807/260311411", 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.944 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b7d7409f-341a-4711-8570-92fe9082a528", 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 18, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 236, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.945 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.946 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.947 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3103276794 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/3103276794", 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6810/3103276794", 2024-03-26T03:59:33.949 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6812/3103276794", 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6811/3103276794", 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6", 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.950 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 25, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 253, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.952 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6813/2795883342", 2024-03-26T03:59:33.953 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6814/2795883342", 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6816/2795883342", 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6815/2795883342", 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8", 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.954 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 32, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 236, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6800", 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.955 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6801", 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6803", 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:59:33.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6802", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6800/931538411", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6801/931538411", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6803/931538411", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6802/931538411", 2024-03-26T03:59:33.957 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc", 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 41, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 167, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 180, 2024-03-26T03:59:33.958 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 253, 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 168, 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6804", 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:33.960 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.960 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6805", 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6807", 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6806", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6804/909197223", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6805/909197223", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6807/909197223", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6806/909197223", 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.962 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "58b58555-59a3-4620-a15f-024f7041713f", 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 50, 2024-03-26T03:59:33.963 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 95, 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6808", 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6809", 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6811", 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.965 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6810", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6808/1580133777", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6809/1580133777", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6811/1580133777", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6810/1580133777", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.966 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7", 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 0.7586517333984375, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 59, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 253, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T03:59:33.967 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6812", 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6813", 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:59:33.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6815", 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T03:59:33.969 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6814", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6812/3905426306", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6813/3905426306", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6815/3905426306", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6814/3905426306", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.970 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:59:11.145539+0000", 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.971 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:59:00.306470+0000", 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:59:17.320033+0000", 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:59:02.566063+0000", 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.973 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:47.161748+0000", 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.974 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:32.173892+0000", 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 167 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:59:26.333434+0000", 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T03:59:33.975 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T03:58:53.055285+0000", 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [], 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [ 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "3.1f", 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.976 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 0, 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.b", 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 5 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.17", 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:59:33.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.1f", 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T03:59:33.978 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [], 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": { 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1978459317": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1771670410": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/3964053112": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/14180211": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/2547319348": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/601470125": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/3870917014": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1892511221": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/435737617": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/93567168": "2024-03-27T03:49:23.135700+0000", 2024-03-26T03:59:33.979 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/2150730999": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/1166209788": "2024-03-27T03:43:49.288943+0000", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1248481299": "2024-03-27T03:44:16.095969+0000", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1832156537": "2024-03-27T03:44:16.095969+0000" 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [], 2024-03-26T03:59:33.980 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T03:59:33.981 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T03:59:33.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T03:59:34.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:34.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:34.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 2 16 6036 6020 188.111 177.625 0.00369898 0.00530461 2024-03-26T03:59:34.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:34.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:34.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:34.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:34.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:34.368 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:34.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:34.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:34.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:34.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:34.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:34.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:34.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:34 smithi006 ceph-mon[29446]: pgmap v488: 148 pgs: 148 active+clean; 60 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 744 B/s rd, 8.6 MiB/s wr, 138 op/s 2024-03-26T03:59:34.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/395729464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:59:34.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:34 smithi006 ceph-mon[37501]: pgmap v488: 148 pgs: 148 active+clean; 60 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 744 B/s rd, 8.6 MiB/s wr, 138 op/s 2024-03-26T03:59:34.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/395729464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:59:34.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:34.722 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:34.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:34.782 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:34 smithi086 ceph-mon[32965]: pgmap v488: 148 pgs: 148 active+clean; 60 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 744 B/s rd, 8.6 MiB/s wr, 138 op/s 2024-03-26T03:59:34.782 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/395729464' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T03:59:34.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:34.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:34.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:34.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:34.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:34.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:34.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:35.021 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:35.025 INFO:tasks.thrashosds.thrasher:j is {'epoch': 258, 'fsid': 'de980954-eb22-11ee-95d0-87774f69a715', 'created': '2024-03-26T03:43:14.534401+0000', 'modified': '2024-03-26T03:59:32.546407+0000', 'last_up_change': '2024-03-26T03:55:58.386605+0000', 'last_in_change': '2024-03-26T03:57:48.463474+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T03:46:30.183658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '165', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T03:48:41.622497+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T03:48:42.191818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T03:48:44.079106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.490000009536743, 'score_stable': 2.490000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T03:48:46.035261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T03:48:48.086810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.7400000095367432, 'score_stable': 1.7400000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.5299999713897705, 'raw_score_stable': 1.5299999713897705, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-03-26T03:59:22.209060+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '257', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '1d1c84cf-5fa7-437e-8004-3b7aca831d6e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6801', 'nonce': 1246039834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 1246039834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 1246039834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6803', 'nonce': 1246039834}]}, 'public_addr': '172.21.15.6:6801/1246039834', 'cluster_addr': '172.21.15.6:6802/1246039834', 'heartbeat_back_addr': '172.21.15.6:6804/1246039834', 'heartbeat_front_addr': '172.21.15.6:6803/1246039834', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '753f7459-62ec-447b-a0da-64d06e813f16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6805', 'nonce': 260311411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 260311411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 260311411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6807', 'nonce': 260311411}]}, 'public_addr': '172.21.15.6:6805/260311411', 'cluster_addr': '172.21.15.6:6806/260311411', 'heartbeat_back_addr': '172.21.15.6:6808/260311411', 'heartbeat_front_addr': '172.21.15.6:6807/260311411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b7d7409f-341a-4711-8570-92fe9082a528', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 236, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6809', 'nonce': 3103276794}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 3103276794}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 3103276794}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6811', 'nonce': 3103276794}]}, 'public_addr': '172.21.15.6:6809/3103276794', 'cluster_addr': '172.21.15.6:6810/3103276794', 'heartbeat_back_addr': '172.21.15.6:6812/3103276794', 'heartbeat_front_addr': '172.21.15.6:6811/3103276794', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a91459a0-dee4-420c-ae2f-940756c985a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6813', 'nonce': 2795883342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 2795883342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2795883342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6815', 'nonce': 2795883342}]}, 'public_addr': '172.21.15.6:6813/2795883342', 'cluster_addr': '172.21.15.6:6814/2795883342', 'heartbeat_back_addr': '172.21.15.6:6816/2795883342', 'heartbeat_front_addr': '172.21.15.6:6815/2795883342', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '087da212-7bf7-4982-b7cc-11f3f66c7ff8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 236, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 931538411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 931538411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6803', 'nonce': 931538411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6802', 'nonce': 931538411}]}, 'public_addr': '172.21.15.86:6800/931538411', 'cluster_addr': '172.21.15.86:6801/931538411', 'heartbeat_back_addr': '172.21.15.86:6803/931538411', 'heartbeat_front_addr': '172.21.15.86:6802/931538411', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7a688745-b213-4bbc-a735-42ffd2a5bfdc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 167, 'up_from': 180, 'up_thru': 253, 'down_at': 168, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 909197223}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 909197223}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6807', 'nonce': 909197223}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6806', 'nonce': 909197223}]}, 'public_addr': '172.21.15.86:6804/909197223', 'cluster_addr': '172.21.15.86:6805/909197223', 'heartbeat_back_addr': '172.21.15.86:6807/909197223', 'heartbeat_front_addr': '172.21.15.86:6806/909197223', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '58b58555-59a3-4620-a15f-024f7041713f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 1580133777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 1580133777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6811', 'nonce': 1580133777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6810', 'nonce': 1580133777}]}, 'public_addr': '172.21.15.86:6808/1580133777', 'cluster_addr': '172.21.15.86:6809/1580133777', 'heartbeat_back_addr': '172.21.15.86:6811/1580133777', 'heartbeat_front_addr': '172.21.15.86:6810/1580133777', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '49b1833d-0a24-4168-a195-47b8ac1d9bc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7586517333984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 253, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 3905426306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 3905426306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6815', 'nonce': 3905426306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6814', 'nonce': 3905426306}]}, 'public_addr': '172.21.15.86:6812/3905426306', 'cluster_addr': '172.21.15.86:6813/3905426306', 'heartbeat_back_addr': '172.21.15.86:6815/3905426306', 'heartbeat_front_addr': '172.21.15.86:6814/3905426306', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:59:11.145539+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:59:00.306470+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:59:17.320033+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:59:02.566063+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:47.161748+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:32.173892+0000', 'dead_epoch': 167}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T03:59:26.333434+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T03:58:53.055285+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.1f', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.6:0/1978459317': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/1771670410': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/3964053112': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/14180211': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:6800/2547319348': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/601470125': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:6800/3870917014': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1892511221': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/435737617': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/93567168': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/2150730999': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:6800/1166209788': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/1248481299': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1832156537': '2024-03-27T03:44:16.095969+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T03:59:35.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T03:59:35.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 3 16 9032 9016 187.819 187.25 0.0116431 0.00531316 2024-03-26T03:59:35.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:35.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:35.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.143 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:35.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:35.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:35.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:35.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:35.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:35.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:35.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:35.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:35.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:35.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:35.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:35.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:35.829 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:35.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:35.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:35.970 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:35.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:36.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T03:59:36.069 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 4 16 11793 11777 183.994 172.562 0.0103892 0.00542732 2024-03-26T03:59:36.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:36.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:36.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:36.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.262 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:36.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:6261 7.5 7:ad7093d4:::benchmark_data_smithi006_77260_object6260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 517.79146485499996, 2024-03-26T03:59:36.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.083550506999999996, 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 6261 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482040+0000", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482041+0000", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.051e-06 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482096+0000", 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4685000000000002e-05 2024-03-26T03:59:36.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482108+0000", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1993e-05 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482118+0000", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6209999999999999e-06 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482166+0000", 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7883000000000002e-05 2024-03-26T03:59:36.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482287+0000", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012121300000000001 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482399+0000", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112178 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.482822+0000", 2024-03-26T03:59:36.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00042292099999999998 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.483327+0000", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00050472400000000001 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565474+0000", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082147297999999994 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565538+0000", 2024-03-26T03:59:36.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4065000000000004e-05 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:50:58.565591+0000", 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2874999999999998e-05 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46138 7.5 7:a2b32a51:::benchmark_data_smithi006_77260_object46137:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 493.83696399899998, 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.132391285, 2024-03-26T03:59:36.284 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46138 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436541+0000", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436543+0000", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5600000000000001e-06 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436591+0000", 2024-03-26T03:59:36.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8069999999999999e-05 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436603+0000", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1931e-05 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436610+0000", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1570000000000003e-06 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436675+0000", 2024-03-26T03:59:36.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4828999999999999e-05 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436816+0000", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000140953 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.436983+0000", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016772 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439654+0000", 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026707150000000002 2024-03-26T03:59:36.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.443582+0000", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039278680000000002 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568851+0000", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.12526948500000001 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568887+0000", 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5108999999999999e-05 2024-03-26T03:59:36.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.568932+0000", 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5887999999999998e-05 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:46141 7.5 7:a45f0658:::benchmark_data_smithi006_77260_object46140:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 493.83521155699998, 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.13568999400000001, 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.290 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 46141 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438294+0000", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438295+0000", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0130000000000001e-06 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438342+0000", 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7655999999999999e-05 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438361+0000", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8967999999999999e-05 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438370+0000", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5660000000000007e-06 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438410+0000", 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0234999999999997e-05 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.293 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438505+0000", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5279999999999996e-05 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.438604+0000", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8387999999999994e-05 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.439762+0000", 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001158563 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.294 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.443943+0000", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004181176 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573919+0000", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.129975919 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573956+0000", 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6600000000000002e-05 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.295 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:51:22.573984+0000", 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7630000000000001e-05 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106682 7.5 7:ab108606:::benchmark_data_smithi006_77260_object106681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 435.980935429, 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099219867000000003, 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.296 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106682 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292570+0000", 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.297 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292571+0000", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2979999999999999e-06 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292604+0000", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2778999999999999e-05 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292619+0000", 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4980999999999999e-05 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.298 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292627+0000", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6209999999999993e-06 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292687+0000", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0056999999999998e-05 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292828+0000", 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000140908 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.299 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.293028+0000", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019991400000000001 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.296547+0000", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035184069999999999 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.300475+0000", 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039280000000000001 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391667+0000", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091192208999999996 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391736+0000", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9216999999999998e-05 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391790+0000", 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3476000000000001e-05 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106684 7.5 7:a3280a30:::benchmark_data_smithi006_77260_object106683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 435.98067280700002, 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.099095332999999994, 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106684 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292832+0000", 2024-03-26T03:59:36.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292833+0000", 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3900000000000005e-07 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292890+0000", 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7488000000000003e-05 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292898+0000", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9000000000000006e-06 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.292903+0000", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.583e-06 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.294905+0000", 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020025569999999999 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.295041+0000", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013517100000000001 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.295184+0000", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014320399999999999 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.296697+0000", 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001513183 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.300509+0000", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003812092 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391836+0000", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.091327152999999994 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391888+0000", 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1773e-05 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.391928+0000", 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9690000000000001e-05 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:106698 7.5 7:aa9cc05b:::benchmark_data_smithi006_77260_object106697:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 435.96198196099999, 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.094714796000000004, 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.308 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 106698 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311523+0000", 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311528+0000", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4880000000000001e-06 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311576+0000", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8798999999999998e-05 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311588+0000", 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1221e-05 2024-03-26T03:59:36.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311594+0000", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5980000000000002e-06 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311641+0000", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6535000000000001e-05 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311732+0000", 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1625000000000005e-05 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.311833+0000", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010040899999999999 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.313748+0000", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019150180000000001 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.394986+0000", 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.081237986999999998 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406121+0000", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.011135107999999999 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406188+0000", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7009999999999997e-05 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:20.406238+0000", 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9997999999999998e-05 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.313 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:111024 7.5 7:af5debec:::benchmark_data_smithi006_77260_object111023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 431.44496325400002, 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.082518326000000003, 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 111024 2024-03-26T03:59:36.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828542+0000", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828543+0000", 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.268e-06 2024-03-26T03:59:36.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828574+0000", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0793000000000001e-05 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828585+0000", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0773e-05 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828591+0000", 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5699999999999998e-06 2024-03-26T03:59:36.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828636+0000", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4672000000000002e-05 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828747+0000", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011108900000000001 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.828863+0000", 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115643 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.830833+0000", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001970795 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.831829+0000", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00099511799999999991 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.910945+0000", 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.079116540999999999 2024-03-26T03:59:36.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.911007+0000", 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1715000000000001e-05 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:24.911060+0000", 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3349e-05 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:131664 7.5 7:a6c5c53b:::benchmark_data_smithi006_77260_object19310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 409.28810335499998, 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10817117900000001, 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131664 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985402+0000", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985403+0000", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1119999999999999e-06 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985407+0000", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4739999999999999e-06 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985417+0000", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9559999999999992e-06 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985425+0000", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6090000000000003e-06 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985467+0000", 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2314000000000002e-05 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985554+0000", 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7089000000000006e-05 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.985630+0000", 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5465999999999999e-05 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.987810+0000", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002180305 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.998012+0000", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010202093000000001 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093484+0000", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.095471577000000002 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093523+0000", 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8995999999999997e-05 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093573+0000", 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0188000000000001e-05 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:131670 7.5 7:a85b9c68:::benchmark_data_smithi006_77260_object19316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 409.28225107399999, 2024-03-26T03:59:36.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.102408474, 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 131670 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991254+0000", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991256+0000", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7010000000000001e-06 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991262+0000", 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5810000000000002e-06 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991275+0000", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3040999999999999e-05 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991283+0000", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0590000000000003e-06 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991338+0000", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4768000000000001e-05 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991476+0000", 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013815400000000001 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.991604+0000", 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012799 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.993610+0000", 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002005682 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:46.998045+0000", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00443452 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093608+0000", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.095563674000000001 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093638+0000", 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9791e-05 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:47.093663+0000", 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4513000000000001e-05 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:133496 7.5 7:a5640036:::benchmark_data_smithi006_77260_object21142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 407.06527948299998, 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076748271000000007, 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.331 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 133496 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208226+0000", 2024-03-26T03:59:36.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208227+0000", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.051e-06 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208232+0000", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0769999999999998e-06 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208242+0000", 2024-03-26T03:59:36.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8199999999999992e-06 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208248+0000", 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1140000000000002e-06 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208286+0000", 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8500000000000001e-05 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208381+0000", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4513999999999997e-05 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.208455+0000", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4407999999999996e-05 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.210647+0000", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021914780000000002 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.215727+0000", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0050805080000000001 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284873+0000", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.069146142999999993 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284929+0000", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5832999999999999e-05 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:49.284974+0000", 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4824999999999999e-05 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:134171 7.5 7:ac24066f:::benchmark_data_smithi006_77260_object21817:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e134)", 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 406.23580292399998, 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.084603088000000007, 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.337 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 134171 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037702+0000", 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037704+0000", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4720000000000001e-06 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037709+0000", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7150000000000003e-06 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037720+0000", 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0473e-05 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037726+0000", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4180000000000002e-06 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037792+0000", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5982000000000006e-05 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.037958+0000", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016622100000000001 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.038050+0000", 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1804000000000004e-05 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.040390+0000", 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023399929999999998 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.043713+0000", 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003322532 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122230+0000", 2024-03-26T03:59:36.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.078516846000000001 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122268+0000", 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8077000000000002e-05 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:52:50.122305+0000", 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7555000000000002e-05 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:177853 7.5 7:aa96cdec:::benchmark_data_smithi006_77260_object65499:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:53:34.385529+0000", 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 361.88797642700001, 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073957160999999993, 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 177853 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385529+0000", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385529+0000", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385537+0000", 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9039999999999992e-06 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385547+0000", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0019e-05 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385607+0000", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0754999999999997e-05 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385630+0000", 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2734e-05 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385732+0000", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101969 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385872+0000", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014007000000000001 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.385996+0000", 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012356800000000001 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.388022+0000", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020263049999999999 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.445006+0000", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.056983587000000002 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.459417+0000", 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014411462999999999 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.459452+0000", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4479e-05 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:53:34.459486+0000", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4307999999999997e-05 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24704.0:212929 7.5 7:af84d7e6:::benchmark_data_smithi006_77260_object100575:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:54:09.868571+0000", 2024-03-26T03:59:36.349 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 326.404933911, 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.074075683000000003, 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24704", 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3338233922", 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 212929 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868571+0000", 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868571+0000", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868573+0000", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0329999999999998e-06 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868584+0000", 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0803e-05 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868610+0000", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5720000000000001e-05 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868631+0000", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1549e-05 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868771+0000", 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013968100000000001 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.868980+0000", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020884000000000001 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,2", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.869113+0000", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000133471 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.871544+0000", 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002430649 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.871934+0000", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00039030299999999999 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.942581+0000", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.070646796999999997 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.942617+0000", 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6006000000000001e-05 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.354 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:09.942647+0000", 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9830999999999999e-05 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.24349.0:1982 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94232629224576 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:54:29.136873+0000", 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 307.13663193500003, 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.31794046300000001, 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.24349", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 1982 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136873+0000", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136873+0000", 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136874+0000", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.13e-06 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136880+0000", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9220000000000002e-06 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136891+0000", 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0727000000000001e-05 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136900+0000", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6219999999999998e-06 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.136996+0000", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6325999999999999e-05 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.137010+0000", 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3573e-05 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.389948+0000", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.252938787 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for degraded object", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.390146+0000", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000198116 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.442488+0000", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.052341150000000003 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.442550+0000", 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2293999999999997e-05 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:54:29.454814+0000", 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012263816 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:79654 8.8 8:182bd937:::benchmark_data_smithi006_103169_object79653:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.852650+0000", 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 241.4208553, 2024-03-26T03:59:36.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076992222999999999, 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 79654 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852650+0000", 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852650+0000", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852651+0000", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1459999999999999e-06 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852706+0000", 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4638000000000001e-05 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852716+0000", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0414000000000001e-05 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852723+0000", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3769999999999998e-06 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852770+0000", 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6705000000000001e-05 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.852917+0000", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014700800000000001 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853043+0000", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012592699999999999 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.857164+0000", 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041208879999999996 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929534+0000", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072369959999999997 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929575+0000", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0658999999999998e-05 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929642+0000", 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7501000000000003e-05 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:79656 8.8 8:13c441a5:::benchmark_data_smithi006_103169_object79655:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.853018+0000", 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 241.420487399, 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.076758246000000002, 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 79656 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853018+0000", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853018+0000", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853018+0000", 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5099999999999999e-07 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853068+0000", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9608999999999999e-05 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853079+0000", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0766e-05 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.853087+0000", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1540000000000002e-06 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.855161+0000", 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020738219999999999 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.855288+0000", 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000127611 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.855425+0000", 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136582 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.857274+0000", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001848939 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929681+0000", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.072407318999999998 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929715+0000", 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3806000000000001e-05 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.929776+0000", 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0887000000000002e-05 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:79669 8.d 8:ba7f4e8f:::benchmark_data_smithi006_103169_object79668:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:34.861171+0000", 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 241.41233452099999, 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.074782527000000001, 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.371 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 79669 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.861171+0000", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.861171+0000", 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.861172+0000", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.051e-06 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.861211+0000", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8961000000000002e-05 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.925812+0000", 2024-03-26T03:59:36.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.064601764000000006 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.925832+0000", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9837999999999999e-05 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.928005+0000", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021722999999999998 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.928127+0000", 2024-03-26T03:59:36.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012234799999999999 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.928280+0000", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015316 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.931873+0000", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0035929360000000001 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.935808+0000", 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0039351890000000004 2024-03-26T03:59:36.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.935868+0000", 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9379000000000002e-05 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:34.935953+0000", 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5600999999999996e-05 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:100651 8.8 8:13519e81:::benchmark_data_smithi006_103169_object100650:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T03:59:36.376 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:50.070709+0000", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 226.202795971, 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.092367383999999997, 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 100651 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070709+0000", 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070709+0000", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070710+0000", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3039999999999999e-06 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070765+0000", 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4784000000000003e-05 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070779+0000", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3814e-05 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070788+0000", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6789999999999994e-06 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.070855+0000", 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6854999999999996e-05 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.071017+0000", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016275 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.071206+0000", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018846800000000001 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.076230+0000", 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00502382 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.162970+0000", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.086740167000000007 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.163009+0000", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9273999999999997e-05 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.163077+0000", 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7469000000000001e-05 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.381 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:100657 8.8 8:109fabe0:::benchmark_data_smithi006_103169_object100656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:50.072621+0000", 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 226.20088439099999, 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.090593079000000007, 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 100657 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.382 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072621+0000", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072621+0000", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072622+0000", 2024-03-26T03:59:36.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7439999999999999e-06 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072688+0000", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5155999999999998e-05 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072701+0000", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2961e-05 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.072709+0000", 2024-03-26T03:59:36.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.85e-06 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.073516+0000", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00080701099999999995 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.073665+0000", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000148259 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.074005+0000", 2024-03-26T03:59:36.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00034000000000000002 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.077091+0000", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003086461 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.163111+0000", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.08601934 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.163147+0000", 2024-03-26T03:59:36.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6681000000000003e-05 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:50.163214+0000", 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6616000000000001e-05 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:102374 8.8 8:118f54bb:::benchmark_data_smithi006_103169_object102373:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e176)", 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:55:51.528743+0000", 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 224.74476254499999, 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.073696883000000005, 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T03:59:36.387 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 102374 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528743+0000", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528743+0000", 2024-03-26T03:59:36.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528744+0000", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.192e-06 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528806+0000", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2324999999999997e-05 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528817+0000", 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0720000000000001e-05 2024-03-26T03:59:36.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528824+0000", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5800000000000003e-06 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.528886+0000", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1705999999999995e-05 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.529019+0000", 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000133137 2024-03-26T03:59:36.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.529186+0000", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016633300000000001 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.530933+0000", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017476939999999999 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.602205+0000", 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.071272104000000003 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.602305+0000", 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9951999999999994e-05 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:55:51.602439+0000", 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013414000000000001 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T03:59:36.392 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T03:59:36.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:36.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:36.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:36.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:36.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:36.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:36 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T03:59:36.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:36 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T03:59:36.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:36 smithi086 ceph-mon[32965]: pgmap v489: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:36.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:36 smithi086 ceph-mon[32965]: osdmap e259: 8 total, 8 up, 7 in 2024-03-26T03:59:36.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:36.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:36.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:36.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:36.841 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:36.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:36.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T03:59:36.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T03:59:36.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[29446]: pgmap v489: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:36.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[29446]: osdmap e259: 8 total, 8 up, 7 in 2024-03-26T03:59:36.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T03:59:36.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T03:59:36.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[37501]: pgmap v489: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 1.2 KiB/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:36.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:36 smithi006 ceph-mon[37501]: osdmap e259: 8 total, 8 up, 7 in 2024-03-26T03:59:36.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:36.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:36.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:36.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:37.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 5 16 14432 14416 180.18 164.938 0.00582626 0.00554336 2024-03-26T03:59:37.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:37.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:37.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:37.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:37.273 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:37.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:37.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T03:59:37.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:37.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:37.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:37.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:37.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:37.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:37.703 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:37.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:37.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:37.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:37.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:37.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:37.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:37.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:37.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:38.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 6 16 16619 16603 172.929 136.688 0.00202104 0.00575646 2024-03-26T03:59:38.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:38.122 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:38.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:38.262 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:38.365 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:38.389 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:38.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:38.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:38.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:38.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:38.662 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:38.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:38.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:38.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:38.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:38 smithi086 ceph-mon[32965]: pgmap v491: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:38 smithi006 ceph-mon[37501]: pgmap v491: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:38 smithi006 ceph-mon[29446]: pgmap v491: 148 pgs: 148 active+clean; 458 MiB data, 3.2 GiB used, 623 GiB / 626 GiB avail; 639 B/s rd, 57 MiB/s wr, 914 op/s 2024-03-26T03:59:38.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:38.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:38.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:38.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:39.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:39.068 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:39.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.069 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:39.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 7 16 18858 18842 168.214 139.938 0.00597055 0.00593738 2024-03-26T03:59:39.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:39.205 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:39.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:39.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:39.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:39.449 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.493 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:39.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:39.553 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:59:39.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:39.595 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:59:39.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:39.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:39.637 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:59:39.682 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:59:39.726 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:59:39.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:39.766 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:59:39.769 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:39.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:39.808 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:59:39.848 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:59:39.853 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:59:39.853 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:59:39.853 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:59:39.853 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:59:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:59:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:59:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:59:39.854 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:59:39.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:39.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:39.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:40.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:40.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:40.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:40.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 8 16 21123 21107 164.881 141.562 0.00713146 0.00605767 2024-03-26T03:59:40.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:40.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:40.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:40.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:40.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:40 smithi086 ceph-mon[32965]: pgmap v492: 148 pgs: 148 active+clean; 815 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-26T03:59:40.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2315843578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:40.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:40.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:40.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[37501]: pgmap v492: 148 pgs: 148 active+clean; 815 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2315843578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:40.429 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[29446]: pgmap v492: 148 pgs: 148 active+clean; 815 MiB data, 4.2 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-03-26T03:59:40.430 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2315843578' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:40.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:40.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:40.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:40.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:40.722 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:40.723 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:40.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:40.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:40.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:40.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:40.997 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:40.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:40.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:41.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 9 16 23407 23391 162.42 142.75 0.00633632 0.0061511 2024-03-26T03:59:41.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:41.138 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:41.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:41.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:41.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:41.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:41.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:41.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T03:59:41.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:41.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:41.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:41.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:41.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:41.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:41.842 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:41.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:41.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:41.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:42.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 10 16 25493 25477 159.215 130.375 0.00160666 0.00625541 2024-03-26T03:59:42.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:42.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:42.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:42.246 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:42.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:42.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:42.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:42.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:42.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.526 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:42.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:42.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:42 smithi086 ceph-mon[32965]: pgmap v493: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T03:59:42.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:42.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:42.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:42.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:42.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:42.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:42.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:42 smithi006 ceph-mon[37501]: pgmap v493: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T03:59:42.989 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:42 smithi006 ceph-mon[29446]: pgmap v493: 148 pgs: 148 active+clean; 1.2 GiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-26T03:59:42.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:43.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:43.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:43.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 11 16 27698 27682 157.268 137.812 0.0032473 0.00635261 2024-03-26T03:59:43.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:43.167 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:43.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:43.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:43.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:43.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:43.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:43.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:43.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:43.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:43.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:43.708 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:43.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:43.850 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:43.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:43.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:43.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:43.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:43.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:44.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 12 16 29464 29448 153.359 110.375 0.00263609 0.00651146 2024-03-26T03:59:44.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:44.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:44.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:44.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:44.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:44.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:44.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:44.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:44.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:44.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:44.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:44 smithi006 ceph-mon[37501]: pgmap v494: 148 pgs: 148 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:59:44.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:44 smithi006 ceph-mon[29446]: pgmap v494: 148 pgs: 148 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:59:44.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:44.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:44.686 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:44.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:44.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:44.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:44.813 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:44 smithi086 ceph-mon[32965]: pgmap v494: 148 pgs: 148 active+clean; 1.4 GiB data, 6.1 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-26T03:59:44.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:44.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:44.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:45.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:45.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 13 16 30828 30812 148.12 85.25 0.00630822 0.0067372 2024-03-26T03:59:45.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:45.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:45.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:45.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:45.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:45.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:45.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:45.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:45.473 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:45.474 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:45.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:45.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:45.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:45.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:45 smithi006 ceph-mon[37501]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:45.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:45 smithi006 ceph-mon[29446]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:45.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:45.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:45.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:45 smithi086 ceph-mon[32965]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T03:59:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:45.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:45.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:45.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:45.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:46.027 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:46.071 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T03:59:46.072 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 14 16 32632 32616 145.593 112.75 0.014325 0.00686304 2024-03-26T03:59:46.081 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T03:59:46.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T03:59:46.129 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:46.183 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:46.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T03:59:46.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:46.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:46.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:46.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:46.504 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:46.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:46.629 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T03:59:46.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:46.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:46.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:46.763 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:46 smithi086 ceph-mon[32965]: pgmap v495: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:59:46.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:46.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:46.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:46 smithi006 ceph-mon[29446]: pgmap v495: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:59:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:46 smithi006 ceph-mon[37501]: pgmap v495: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T03:59:46.919 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:46.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:46.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:47.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:47.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:47.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:47.070 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 15 16 34453 34437 143.473 113.812 0.0109348 0.00696328 2024-03-26T03:59:47.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:47.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:47.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:47.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:47.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:47.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:47.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:47.465 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:47.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:47.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:47.605 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:47.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:47.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:47.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:47.852 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:47.868 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-03-26T03:59:43.401903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1485380008,"num_objects":22886,"num_object_clones":0,"num_object_copies":68658,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22886,"num_whiteouts":0,"num_read":2291,"num_read_kb":2105,"num_write":23749,"num_write_kb":1452445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":2368758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23795,"ondisk_log_size":23795,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6392224,"kb_used_data":3990632,"kb_used_omap":13,"kb_used_meta":2401522,"kb_avail":649823840,"statfs":{"total":671965249536,"available":665419612160,"internally_reserved":0,"allocated":4086407168,"data_stored":4083600578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14011,"internal_metadata":2459158853},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[10,6,4,2,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1422917632,"num_objects":21712,"num_object_clones":0,"num_object_copies":65136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21712,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":21712,"num_write_kb":1389568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004063"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":422,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594241+0000","last_change":"2024-03-26T03:59:12.322291+0000","last_active":"2024-03-26T03:59:35.594241+0000","last_peered":"2024-03-26T03:59:35.594241+0000","last_clean":"2024-03-26T03:59:35.594241+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T03:59:35.594241+0000","last_undegraded":"2024-03-26T03:59:35.594241+0000","last_fullsized":"2024-03-26T03:59:35.594241+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.322148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.322148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:12.821832+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":424,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590104+0000","last_change":"2024-03-26T03:58:37.457198+0000","last_active":"2024-03-26T03:59:35.590104+0000","last_peered":"2024-03-26T03:59:35.590104+0000","last_clean":"2024-03-26T03:59:35.590104+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T03:59:35.590104+0000","last_undegraded":"2024-03-26T03:59:35.590104+0000","last_fullsized":"2024-03-26T03:59:35.590104+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T03:58:37.457112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:59.336215+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":435,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588796+0000","last_change":"2024-03-26T03:55:59.307852+0000","last_active":"2024-03-26T03:59:35.588796+0000","last_peered":"2024-03-26T03:59:35.588796+0000","last_clean":"2024-03-26T03:59:35.588796+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T03:59:35.588796+0000","last_undegraded":"2024-03-26T03:59:35.588796+0000","last_fullsized":"2024-03-26T03:59:35.588796+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.751225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.751225+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694322+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":512,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.593658+0000","last_change":"2024-03-26T03:57:49.470002+0000","last_active":"2024-03-26T03:59:35.593658+0000","last_peered":"2024-03-26T03:59:35.593658+0000","last_clean":"2024-03-26T03:59:35.593658+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T03:59:35.593658+0000","last_undegraded":"2024-03-26T03:59:35.593658+0000","last_fullsized":"2024-03-26T03:59:35.593658+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341729+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584334+0000","last_change":"2024-03-26T03:57:49.377568+0000","last_active":"2024-03-26T03:59:35.584334+0000","last_peered":"2024-03-26T03:59:35.584334+0000","last_clean":"2024-03-26T03:59:35.584334+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T03:59:35.584334+0000","last_undegraded":"2024-03-26T03:59:35.584334+0000","last_fullsized":"2024-03-26T03:59:35.584334+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":423,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.593541+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T03:59:35.593541+0000","last_peered":"2024-03-26T03:59:35.593541+0000","last_clean":"2024-03-26T03:59:35.593541+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T03:59:35.593541+0000","last_undegraded":"2024-03-26T03:59:35.593541+0000","last_fullsized":"2024-03-26T03:59:35.593541+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341760+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585494+0000","last_change":"2024-03-26T03:57:49.382349+0000","last_active":"2024-03-26T03:59:35.585494+0000","last_peered":"2024-03-26T03:59:35.585494+0000","last_clean":"2024-03-26T03:59:35.585494+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T03:59:35.585494+0000","last_undegraded":"2024-03-26T03:59:35.585494+0000","last_fullsized":"2024-03-26T03:59:35.585494+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":148,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583774+0000","last_change":"2024-03-26T03:59:19.277977+0000","last_active":"2024-03-26T03:59:35.583774+0000","last_peered":"2024-03-26T03:59:35.583774+0000","last_clean":"2024-03-26T03:59:35.583774+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T03:59:35.583774+0000","last_undegraded":"2024-03-26T03:59:35.583774+0000","last_fullsized":"2024-03-26T03:59:35.583774+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T03:59:19.277872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.277872+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.852334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":413,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589294+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T03:59:35.589294+0000","last_peered":"2024-03-26T03:59:35.589294+0000","last_clean":"2024-03-26T03:59:35.589294+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T03:59:35.589294+0000","last_undegraded":"2024-03-26T03:59:35.589294+0000","last_fullsized":"2024-03-26T03:59:35.589294+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694626+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":406,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580700+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T03:59:35.580700+0000","last_peered":"2024-03-26T03:59:35.580700+0000","last_clean":"2024-03-26T03:59:35.580700+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T03:59:35.580700+0000","last_undegraded":"2024-03-26T03:59:35.580700+0000","last_fullsized":"2024-03-26T03:59:35.580700+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":364,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580791+0000","last_change":"2024-03-26T03:59:20.650716+0000","last_active":"2024-03-26T03:59:35.580791+0000","last_peered":"2024-03-26T03:59:35.580791+0000","last_clean":"2024-03-26T03:59:35.580791+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T03:59:35.580791+0000","last_undegraded":"2024-03-26T03:59:35.580791+0000","last_fullsized":"2024-03-26T03:59:35.580791+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.717054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":519,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594528+0000","last_change":"2024-03-26T03:59:21.358672+0000","last_active":"2024-03-26T03:59:35.594528+0000","last_peered":"2024-03-26T03:59:35.594528+0000","last_clean":"2024-03-26T03:59:35.594528+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T03:59:35.594528+0000","last_undegraded":"2024-03-26T03:59:35.594528+0000","last_fullsized":"2024-03-26T03:59:35.594528+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T03:59:21.358504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.358504+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:39.467201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":380,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580354+0000","last_change":"2024-03-26T03:57:49.381007+0000","last_active":"2024-03-26T03:59:35.580354+0000","last_peered":"2024-03-26T03:59:35.580354+0000","last_clean":"2024-03-26T03:59:35.580354+0000","last_became_active":"2024-03-26T03:57:49.380178+0000","last_became_peered":"2024-03-26T03:57:49.380178+0000","last_unstale":"2024-03-26T03:59:35.580354+0000","last_undegraded":"2024-03-26T03:59:35.580354+0000","last_fullsized":"2024-03-26T03:59:35.580354+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421488+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":425,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583460+0000","last_change":"2024-03-26T03:58:28.050117+0000","last_active":"2024-03-26T03:59:35.583460+0000","last_peered":"2024-03-26T03:59:35.583460+0000","last_clean":"2024-03-26T03:59:35.583460+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T03:59:35.583460+0000","last_undegraded":"2024-03-26T03:59:35.583460+0000","last_fullsized":"2024-03-26T03:59:35.583460+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:28.050003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.050003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":381,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580455+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T03:59:35.580455+0000","last_peered":"2024-03-26T03:59:35.580455+0000","last_clean":"2024-03-26T03:59:35.580455+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T03:59:35.580455+0000","last_undegraded":"2024-03-26T03:59:35.580455+0000","last_fullsized":"2024-03-26T03:59:35.580455+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":477,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594983+0000","last_change":"2024-03-26T03:57:49.485971+0000","last_active":"2024-03-26T03:59:35.594983+0000","last_peered":"2024-03-26T03:59:35.594983+0000","last_clean":"2024-03-26T03:59:35.594983+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T03:59:35.594983+0000","last_undegraded":"2024-03-26T03:59:35.594983+0000","last_fullsized":"2024-03-26T03:59:35.594983+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341624+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595534+0000","last_change":"2024-03-26T03:59:11.325209+0000","last_active":"2024-03-26T03:59:35.595534+0000","last_peered":"2024-03-26T03:59:35.595534+0000","last_clean":"2024-03-26T03:59:35.595534+0000","last_became_active":"2024-03-26T03:57:49.360508+0000","last_became_peered":"2024-03-26T03:57:49.360508+0000","last_unstale":"2024-03-26T03:59:35.595534+0000","last_undegraded":"2024-03-26T03:59:35.595534+0000","last_fullsized":"2024-03-26T03:59:35.595534+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:11.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:11.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:21.774440+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":415,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.591247+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T03:59:35.591247+0000","last_peered":"2024-03-26T03:59:35.591247+0000","last_clean":"2024-03-26T03:59:35.591247+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T03:59:35.591247+0000","last_undegraded":"2024-03-26T03:59:35.591247+0000","last_fullsized":"2024-03-26T03:59:35.591247+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.401418+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":433,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588685+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T03:59:35.588685+0000","last_peered":"2024-03-26T03:59:35.588685+0000","last_clean":"2024-03-26T03:59:35.588685+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T03:59:35.588685+0000","last_undegraded":"2024-03-26T03:59:35.588685+0000","last_fullsized":"2024-03-26T03:59:35.588685+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":532,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.586676+0000","last_change":"2024-03-26T03:58:28.238146+0000","last_active":"2024-03-26T03:59:35.586676+0000","last_peered":"2024-03-26T03:59:35.586676+0000","last_clean":"2024-03-26T03:59:35.586676+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T03:59:35.586676+0000","last_undegraded":"2024-03-26T03:59:35.586676+0000","last_fullsized":"2024-03-26T03:59:35.586676+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:28.237979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.237979+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.896289+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584987+0000","last_change":"2024-03-26T03:59:22.373594+0000","last_active":"2024-03-26T03:59:35.584987+0000","last_peered":"2024-03-26T03:59:35.584987+0000","last_clean":"2024-03-26T03:59:35.584987+0000","last_became_active":"2024-03-26T03:57:49.376975+0000","last_became_peered":"2024-03-26T03:57:49.376975+0000","last_unstale":"2024-03-26T03:59:35.584987+0000","last_undegraded":"2024-03-26T03:59:35.584987+0000","last_fullsized":"2024-03-26T03:59:35.584987+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.373494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.373494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:35.705499+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":429,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595257+0000","last_change":"2024-03-26T03:57:49.360798+0000","last_active":"2024-03-26T03:59:35.595257+0000","last_peered":"2024-03-26T03:59:35.595257+0000","last_clean":"2024-03-26T03:59:35.595257+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T03:59:35.595257+0000","last_undegraded":"2024-03-26T03:59:35.595257+0000","last_fullsized":"2024-03-26T03:59:35.595257+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341433+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":433,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594633+0000","last_change":"2024-03-26T03:59:23.332947+0000","last_active":"2024-03-26T03:59:35.594633+0000","last_peered":"2024-03-26T03:59:35.594633+0000","last_clean":"2024-03-26T03:59:35.594633+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T03:59:35.594633+0000","last_undegraded":"2024-03-26T03:59:35.594633+0000","last_fullsized":"2024-03-26T03:59:35.594633+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:23.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.172598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":129,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585163+0000","last_change":"2024-03-26T03:59:13.198844+0000","last_active":"2024-03-26T03:59:35.585163+0000","last_peered":"2024-03-26T03:59:35.585163+0000","last_clean":"2024-03-26T03:59:35.585163+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T03:59:35.585163+0000","last_undegraded":"2024-03-26T03:59:35.585163+0000","last_fullsized":"2024-03-26T03:59:35.585163+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T03:59:13.198735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.198735+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.242387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":100,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580549+0000","last_change":"2024-03-26T03:58:40.497404+0000","last_active":"2024-03-26T03:59:35.580549+0000","last_peered":"2024-03-26T03:59:35.580549+0000","last_clean":"2024-03-26T03:59:35.580549+0000","last_became_active":"2024-03-26T03:58:40.496528+0000","last_became_peered":"2024-03-26T03:58:40.496528+0000","last_unstale":"2024-03-26T03:59:35.580549+0000","last_undegraded":"2024-03-26T03:59:35.580549+0000","last_fullsized":"2024-03-26T03:59:35.580549+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T03:58:37.045404+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T03:58:37.045404+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:04.461940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":498,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595479+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T03:59:35.595479+0000","last_peered":"2024-03-26T03:59:35.595479+0000","last_clean":"2024-03-26T03:59:35.595479+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T03:59:35.595479+0000","last_undegraded":"2024-03-26T03:59:35.595479+0000","last_fullsized":"2024-03-26T03:59:35.595479+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341555+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":453,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589973+0000","last_change":"2024-03-26T03:59:26.325088+0000","last_active":"2024-03-26T03:59:35.589973+0000","last_peered":"2024-03-26T03:59:35.589973+0000","last_clean":"2024-03-26T03:59:35.589973+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T03:59:35.589973+0000","last_undegraded":"2024-03-26T03:59:35.589973+0000","last_fullsized":"2024-03-26T03:59:35.589973+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.324998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:43.900699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"241'26","reported_seq":504,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588684+0000","last_change":"2024-03-26T03:57:49.470100+0000","last_active":"2024-03-26T03:59:35.588684+0000","last_peered":"2024-03-26T03:59:35.588684+0000","last_clean":"2024-03-26T03:59:35.588684+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T03:59:35.588684+0000","last_undegraded":"2024-03-26T03:59:35.588684+0000","last_fullsized":"2024-03-26T03:59:35.588684+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":420,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590792+0000","last_change":"2024-03-26T03:58:39.479856+0000","last_active":"2024-03-26T03:59:35.590792+0000","last_peered":"2024-03-26T03:59:35.590792+0000","last_clean":"2024-03-26T03:59:35.590792+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T03:59:35.590792+0000","last_undegraded":"2024-03-26T03:59:35.590792+0000","last_fullsized":"2024-03-26T03:59:35.590792+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:39.479775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.479775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:46.748116+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":418,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589048+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T03:59:35.589048+0000","last_peered":"2024-03-26T03:59:35.589048+0000","last_clean":"2024-03-26T03:59:35.589048+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T03:59:35.589048+0000","last_undegraded":"2024-03-26T03:59:35.589048+0000","last_fullsized":"2024-03-26T03:59:35.589048+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694387+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583730+0000","last_change":"2024-03-26T03:57:49.399354+0000","last_active":"2024-03-26T03:59:35.583730+0000","last_peered":"2024-03-26T03:59:35.583730+0000","last_clean":"2024-03-26T03:59:35.583730+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T03:59:35.583730+0000","last_undegraded":"2024-03-26T03:59:35.583730+0000","last_fullsized":"2024-03-26T03:59:35.583730+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":422,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581557+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T03:59:35.581557+0000","last_peered":"2024-03-26T03:59:35.581557+0000","last_clean":"2024-03-26T03:59:35.581557+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T03:59:35.581557+0000","last_undegraded":"2024-03-26T03:59:35.581557+0000","last_fullsized":"2024-03-26T03:59:35.581557+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":421,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583692+0000","last_change":"2024-03-26T03:58:38.043643+0000","last_active":"2024-03-26T03:59:35.583692+0000","last_peered":"2024-03-26T03:59:35.583692+0000","last_clean":"2024-03-26T03:59:35.583692+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T03:59:35.583692+0000","last_undegraded":"2024-03-26T03:59:35.583692+0000","last_fullsized":"2024-03-26T03:59:35.583692+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:38.043527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:38.043527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:01.615166+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":355,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594119+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T03:59:35.594119+0000","last_peered":"2024-03-26T03:59:35.594119+0000","last_clean":"2024-03-26T03:59:35.594119+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T03:59:35.594119+0000","last_undegraded":"2024-03-26T03:59:35.594119+0000","last_fullsized":"2024-03-26T03:59:35.594119+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341372+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":446,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589039+0000","last_change":"2024-03-26T03:57:49.385528+0000","last_active":"2024-03-26T03:59:35.589039+0000","last_peered":"2024-03-26T03:59:35.589039+0000","last_clean":"2024-03-26T03:59:35.589039+0000","last_became_active":"2024-03-26T03:57:49.384847+0000","last_became_peered":"2024-03-26T03:57:49.384847+0000","last_unstale":"2024-03-26T03:59:35.589039+0000","last_undegraded":"2024-03-26T03:59:35.589039+0000","last_fullsized":"2024-03-26T03:59:35.589039+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694470+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":118,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583632+0000","last_change":"2024-03-26T03:59:16.277806+0000","last_active":"2024-03-26T03:59:35.583632+0000","last_peered":"2024-03-26T03:59:35.583632+0000","last_clean":"2024-03-26T03:59:35.583632+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T03:59:35.583632+0000","last_undegraded":"2024-03-26T03:59:35.583632+0000","last_fullsized":"2024-03-26T03:59:35.583632+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T03:59:16.277717+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.277717+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.141535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584234+0000","last_change":"2024-03-26T03:57:49.383008+0000","last_active":"2024-03-26T03:59:35.584234+0000","last_peered":"2024-03-26T03:59:35.584234+0000","last_clean":"2024-03-26T03:59:35.584234+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T03:59:35.584234+0000","last_undegraded":"2024-03-26T03:59:35.584234+0000","last_fullsized":"2024-03-26T03:59:35.584234+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":431,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583735+0000","last_change":"2024-03-26T03:58:42.072686+0000","last_active":"2024-03-26T03:59:35.583735+0000","last_peered":"2024-03-26T03:59:35.583735+0000","last_clean":"2024-03-26T03:59:35.583735+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T03:59:35.583735+0000","last_undegraded":"2024-03-26T03:59:35.583735+0000","last_fullsized":"2024-03-26T03:59:35.583735+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:42.072569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.072569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:55.017162+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":389,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580932+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T03:59:35.580932+0000","last_peered":"2024-03-26T03:59:35.580932+0000","last_clean":"2024-03-26T03:59:35.580932+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T03:59:35.580932+0000","last_undegraded":"2024-03-26T03:59:35.580932+0000","last_fullsized":"2024-03-26T03:59:35.580932+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":132,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583938+0000","last_change":"2024-03-26T03:59:14.247318+0000","last_active":"2024-03-26T03:59:35.583938+0000","last_peered":"2024-03-26T03:59:35.583938+0000","last_clean":"2024-03-26T03:59:35.583938+0000","last_became_active":"2024-03-26T03:57:49.373634+0000","last_became_peered":"2024-03-26T03:57:49.373634+0000","last_unstale":"2024-03-26T03:59:35.583938+0000","last_undegraded":"2024-03-26T03:59:35.583938+0000","last_fullsized":"2024-03-26T03:59:35.583938+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:14.247185+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.247185+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.620476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":422,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588725+0000","last_change":"2024-03-26T03:55:59.324307+0000","last_active":"2024-03-26T03:59:35.588725+0000","last_peered":"2024-03-26T03:59:35.588725+0000","last_clean":"2024-03-26T03:59:35.588725+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T03:59:35.588725+0000","last_undegraded":"2024-03-26T03:59:35.588725+0000","last_fullsized":"2024-03-26T03:59:35.588725+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.830730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.830730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694357+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584130+0000","last_change":"2024-03-26T03:57:49.377475+0000","last_active":"2024-03-26T03:59:35.584130+0000","last_peered":"2024-03-26T03:59:35.584130+0000","last_clean":"2024-03-26T03:59:35.584130+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T03:59:35.584130+0000","last_undegraded":"2024-03-26T03:59:35.584130+0000","last_fullsized":"2024-03-26T03:59:35.584130+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":445,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583646+0000","last_change":"2024-03-26T03:58:11.047294+0000","last_active":"2024-03-26T03:59:35.583646+0000","last_peered":"2024-03-26T03:59:35.583646+0000","last_clean":"2024-03-26T03:59:35.583646+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T03:59:35.583646+0000","last_undegraded":"2024-03-26T03:59:35.583646+0000","last_fullsized":"2024-03-26T03:59:35.583646+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.047184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.047184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:39.616278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":480,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.593390+0000","last_change":"2024-03-26T03:59:27.366321+0000","last_active":"2024-03-26T03:59:35.593390+0000","last_peered":"2024-03-26T03:59:35.593390+0000","last_clean":"2024-03-26T03:59:35.593390+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T03:59:35.593390+0000","last_undegraded":"2024-03-26T03:59:35.593390+0000","last_fullsized":"2024-03-26T03:59:35.593390+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:27.366230+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.366230+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:52.869773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583728+0000","last_change":"2024-03-26T03:59:28.149466+0000","last_active":"2024-03-26T03:59:35.583728+0000","last_peered":"2024-03-26T03:59:35.583728+0000","last_clean":"2024-03-26T03:59:35.583728+0000","last_became_active":"2024-03-26T03:57:49.370042+0000","last_became_peered":"2024-03-26T03:57:49.370042+0000","last_unstale":"2024-03-26T03:59:35.583728+0000","last_undegraded":"2024-03-26T03:59:35.583728+0000","last_fullsized":"2024-03-26T03:59:35.583728+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.149370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.149370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.085072+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":436,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.592876+0000","last_change":"2024-03-26T03:58:20.461781+0000","last_active":"2024-03-26T03:59:35.592876+0000","last_peered":"2024-03-26T03:59:35.592876+0000","last_clean":"2024-03-26T03:59:35.592876+0000","last_became_active":"2024-03-26T03:57:49.379985+0000","last_became_peered":"2024-03-26T03:57:49.379985+0000","last_unstale":"2024-03-26T03:59:35.592876+0000","last_undegraded":"2024-03-26T03:59:35.592876+0000","last_fullsized":"2024-03-26T03:59:35.592876+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:20.461665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:20.461665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:46.093646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595130+0000","last_change":"2024-03-26T03:59:24.336092+0000","last_active":"2024-03-26T03:59:35.595130+0000","last_peered":"2024-03-26T03:59:35.595130+0000","last_clean":"2024-03-26T03:59:35.595130+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T03:59:35.595130+0000","last_undegraded":"2024-03-26T03:59:35.595130+0000","last_fullsized":"2024-03-26T03:59:35.595130+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:24.336010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:24.336010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.514695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":479,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595089+0000","last_change":"2024-03-26T03:59:18.367293+0000","last_active":"2024-03-26T03:59:35.595089+0000","last_peered":"2024-03-26T03:59:35.595089+0000","last_clean":"2024-03-26T03:59:35.595089+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T03:59:35.595089+0000","last_undegraded":"2024-03-26T03:59:35.595089+0000","last_fullsized":"2024-03-26T03:59:35.595089+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:18.367180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.367180+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:25.855644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":425,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588537+0000","last_change":"2024-03-26T03:59:17.820332+0000","last_active":"2024-03-26T03:59:35.588537+0000","last_peered":"2024-03-26T03:59:35.588537+0000","last_clean":"2024-03-26T03:59:35.588537+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T03:59:35.588537+0000","last_undegraded":"2024-03-26T03:59:35.588537+0000","last_fullsized":"2024-03-26T03:59:35.588537+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:17.820216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.820216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.723506+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":537,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580711+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T03:59:35.580711+0000","last_peered":"2024-03-26T03:59:35.580711+0000","last_clean":"2024-03-26T03:59:35.580711+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T03:59:35.580711+0000","last_undegraded":"2024-03-26T03:59:35.580711+0000","last_fullsized":"2024-03-26T03:59:35.580711+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583967+0000","last_change":"2024-03-26T03:59:17.888655+0000","last_active":"2024-03-26T03:59:35.583967+0000","last_peered":"2024-03-26T03:59:35.583967+0000","last_clean":"2024-03-26T03:59:35.583967+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T03:59:35.583967+0000","last_undegraded":"2024-03-26T03:59:35.583967+0000","last_fullsized":"2024-03-26T03:59:35.583967+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:17.888578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.888578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:31.797567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":129,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584926+0000","last_change":"2024-03-26T03:57:49.435763+0000","last_active":"2024-03-26T03:59:35.584926+0000","last_peered":"2024-03-26T03:59:35.584926+0000","last_clean":"2024-03-26T03:59:35.584926+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T03:59:35.584926+0000","last_undegraded":"2024-03-26T03:59:35.584926+0000","last_fullsized":"2024-03-26T03:59:35.584926+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":422,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595482+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T03:59:35.595482+0000","last_peered":"2024-03-26T03:59:35.595482+0000","last_clean":"2024-03-26T03:59:35.595482+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T03:59:35.595482+0000","last_undegraded":"2024-03-26T03:59:35.595482+0000","last_fullsized":"2024-03-26T03:59:35.595482+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":125,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584755+0000","last_change":"2024-03-26T03:57:49.512060+0000","last_active":"2024-03-26T03:59:35.584755+0000","last_peered":"2024-03-26T03:59:35.584755+0000","last_clean":"2024-03-26T03:59:35.584755+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T03:59:35.584755+0000","last_undegraded":"2024-03-26T03:59:35.584755+0000","last_fullsized":"2024-03-26T03:59:35.584755+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":463,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588100+0000","last_change":"2024-03-26T03:57:49.480593+0000","last_active":"2024-03-26T03:59:35.588100+0000","last_peered":"2024-03-26T03:59:35.588100+0000","last_clean":"2024-03-26T03:59:35.588100+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T03:59:35.588100+0000","last_undegraded":"2024-03-26T03:59:35.588100+0000","last_fullsized":"2024-03-26T03:59:35.588100+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694688+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":464,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580770+0000","last_change":"2024-03-26T03:59:18.719422+0000","last_active":"2024-03-26T03:59:35.580770+0000","last_peered":"2024-03-26T03:59:35.580770+0000","last_clean":"2024-03-26T03:59:35.580770+0000","last_became_active":"2024-03-26T03:57:49.382991+0000","last_became_peered":"2024-03-26T03:57:49.382991+0000","last_unstale":"2024-03-26T03:59:35.580770+0000","last_undegraded":"2024-03-26T03:59:35.580770+0000","last_fullsized":"2024-03-26T03:59:35.580770+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T03:59:18.719326+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.719326+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:18.748523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":439,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587026+0000","last_change":"2024-03-26T03:59:17.399536+0000","last_active":"2024-03-26T03:59:35.587026+0000","last_peered":"2024-03-26T03:59:35.587026+0000","last_clean":"2024-03-26T03:59:35.587026+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T03:59:35.587026+0000","last_undegraded":"2024-03-26T03:59:35.587026+0000","last_fullsized":"2024-03-26T03:59:35.587026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:59:17.399444+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.399444+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.422968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585349+0000","last_change":"2024-03-26T03:57:49.368813+0000","last_active":"2024-03-26T03:59:35.585349+0000","last_peered":"2024-03-26T03:59:35.585349+0000","last_clean":"2024-03-26T03:59:35.585349+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T03:59:35.585349+0000","last_undegraded":"2024-03-26T03:59:35.585349+0000","last_fullsized":"2024-03-26T03:59:35.585349+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":134,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594455+0000","last_change":"2024-03-26T03:58:40.497439+0000","last_active":"2024-03-26T03:59:35.594455+0000","last_peered":"2024-03-26T03:59:35.594455+0000","last_clean":"2024-03-26T03:59:35.594455+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T03:59:35.594455+0000","last_undegraded":"2024-03-26T03:59:35.594455+0000","last_fullsized":"2024-03-26T03:59:35.594455+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:15.134741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.134741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341791+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":510,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588811+0000","last_change":"2024-03-26T03:57:49.445142+0000","last_active":"2024-03-26T03:59:35.588811+0000","last_peered":"2024-03-26T03:59:35.588811+0000","last_clean":"2024-03-26T03:59:35.588811+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T03:59:35.588811+0000","last_undegraded":"2024-03-26T03:59:35.588811+0000","last_fullsized":"2024-03-26T03:59:35.588811+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694596+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":436,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583566+0000","last_change":"2024-03-26T03:59:28.005851+0000","last_active":"2024-03-26T03:59:35.583566+0000","last_peered":"2024-03-26T03:59:35.583566+0000","last_clean":"2024-03-26T03:59:35.583566+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T03:59:35.583566+0000","last_undegraded":"2024-03-26T03:59:35.583566+0000","last_fullsized":"2024-03-26T03:59:35.583566+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.005705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.005705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.608573+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":394,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581042+0000","last_change":"2024-03-26T03:59:15.662289+0000","last_active":"2024-03-26T03:59:35.581042+0000","last_peered":"2024-03-26T03:59:35.581042+0000","last_clean":"2024-03-26T03:59:35.581042+0000","last_became_active":"2024-03-26T03:57:49.380167+0000","last_became_peered":"2024-03-26T03:57:49.380167+0000","last_unstale":"2024-03-26T03:59:35.581042+0000","last_undegraded":"2024-03-26T03:59:35.581042+0000","last_fullsized":"2024-03-26T03:59:35.581042+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:15.662211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.662211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.275532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":384,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589146+0000","last_change":"2024-03-26T03:59:20.739789+0000","last_active":"2024-03-26T03:59:35.589146+0000","last_peered":"2024-03-26T03:59:35.589146+0000","last_clean":"2024-03-26T03:59:35.589146+0000","last_became_active":"2024-03-26T03:57:49.377869+0000","last_became_peered":"2024-03-26T03:57:49.377869+0000","last_unstale":"2024-03-26T03:59:35.589146+0000","last_undegraded":"2024-03-26T03:59:35.589146+0000","last_fullsized":"2024-03-26T03:59:35.589146+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:43.234343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":117,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585182+0000","last_change":"2024-03-26T03:57:49.511256+0000","last_active":"2024-03-26T03:59:35.585182+0000","last_peered":"2024-03-26T03:59:35.585182+0000","last_clean":"2024-03-26T03:59:35.585182+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T03:59:35.585182+0000","last_undegraded":"2024-03-26T03:59:35.585182+0000","last_fullsized":"2024-03-26T03:59:35.585182+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"259'1351","reported_seq":1373,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:40.192903+0000","last_change":"2024-03-26T03:59:23.518905+0000","last_active":"2024-03-26T03:59:40.192903+0000","last_peered":"2024-03-26T03:59:40.192903+0000","last_clean":"2024-03-26T03:59:40.192903+0000","last_became_active":"2024-03-26T03:59:23.518616+0000","last_became_peered":"2024-03-26T03:59:23.518616+0000","last_unstale":"2024-03-26T03:59:40.192903+0000","last_undegraded":"2024-03-26T03:59:40.192903+0000","last_fullsized":"2024-03-26T03:59:40.192903+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1351,"log_dups_size":0,"ondisk_log_size":1351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.143028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88539136,"num_objects":1351,"num_object_clones":0,"num_object_copies":4053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1351,"num_write_kb":86464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":381,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580980+0000","last_change":"2024-03-26T03:59:31.507063+0000","last_active":"2024-03-26T03:59:35.580980+0000","last_peered":"2024-03-26T03:59:35.580980+0000","last_clean":"2024-03-26T03:59:35.580980+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T03:59:35.580980+0000","last_undegraded":"2024-03-26T03:59:35.580980+0000","last_fullsized":"2024-03-26T03:59:35.580980+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:31.506978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:31.506978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.951142+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":445,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583789+0000","last_change":"2024-03-26T03:58:29.964096+0000","last_active":"2024-03-26T03:59:35.583789+0000","last_peered":"2024-03-26T03:59:35.583789+0000","last_clean":"2024-03-26T03:59:35.583789+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T03:59:35.583789+0000","last_undegraded":"2024-03-26T03:59:35.583789+0000","last_fullsized":"2024-03-26T03:59:35.583789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:58:29.964005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.964005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:50.334946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":464,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580485+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T03:59:35.580485+0000","last_peered":"2024-03-26T03:59:35.580485+0000","last_clean":"2024-03-26T03:59:35.580485+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T03:59:35.580485+0000","last_undegraded":"2024-03-26T03:59:35.580485+0000","last_fullsized":"2024-03-26T03:59:35.580485+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":583,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584204+0000","last_change":"2024-03-26T03:59:11.877979+0000","last_active":"2024-03-26T03:59:35.584204+0000","last_peered":"2024-03-26T03:59:35.584204+0000","last_clean":"2024-03-26T03:59:35.584204+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T03:59:35.584204+0000","last_undegraded":"2024-03-26T03:59:35.584204+0000","last_fullsized":"2024-03-26T03:59:35.584204+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:59:11.877821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:11.877821+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.352750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"259'1323","reported_seq":1345,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:40.309921+0000","last_change":"2024-03-26T03:59:23.931890+0000","last_active":"2024-03-26T03:59:40.309921+0000","last_peered":"2024-03-26T03:59:40.309921+0000","last_clean":"2024-03-26T03:59:40.309921+0000","last_became_active":"2024-03-26T03:59:23.931611+0000","last_became_peered":"2024-03-26T03:59:23.931611+0000","last_unstale":"2024-03-26T03:59:40.309921+0000","last_undegraded":"2024-03-26T03:59:40.309921+0000","last_fullsized":"2024-03-26T03:59:40.309921+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.332800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1323,"num_write_kb":84672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584032+0000","last_change":"2024-03-26T03:59:30.159583+0000","last_active":"2024-03-26T03:59:35.584032+0000","last_peered":"2024-03-26T03:59:35.584032+0000","last_clean":"2024-03-26T03:59:35.584032+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T03:59:35.584032+0000","last_undegraded":"2024-03-26T03:59:35.584032+0000","last_fullsized":"2024-03-26T03:59:35.584032+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.159481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.159481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.184050+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":433,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584004+0000","last_change":"2024-03-26T03:58:25.086917+0000","last_active":"2024-03-26T03:59:35.584004+0000","last_peered":"2024-03-26T03:59:35.584004+0000","last_clean":"2024-03-26T03:59:35.584004+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T03:59:35.584004+0000","last_undegraded":"2024-03-26T03:59:35.584004+0000","last_fullsized":"2024-03-26T03:59:35.584004+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:25.086805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:25.086805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":522,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594781+0000","last_change":"2024-03-26T03:57:49.408864+0000","last_active":"2024-03-26T03:59:35.594781+0000","last_peered":"2024-03-26T03:59:35.594781+0000","last_clean":"2024-03-26T03:59:35.594781+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T03:59:35.594781+0000","last_undegraded":"2024-03-26T03:59:35.594781+0000","last_fullsized":"2024-03-26T03:59:35.594781+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341575+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":442,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.586301+0000","last_change":"2024-03-26T03:59:22.387740+0000","last_active":"2024-03-26T03:59:35.586301+0000","last_peered":"2024-03-26T03:59:35.586301+0000","last_clean":"2024-03-26T03:59:35.586301+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T03:59:35.586301+0000","last_undegraded":"2024-03-26T03:59:35.586301+0000","last_fullsized":"2024-03-26T03:59:35.586301+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.387664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.387664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.979624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"259'1809","reported_seq":1831,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:43.282178+0000","last_change":"2024-03-26T03:59:23.521792+0000","last_active":"2024-03-26T03:59:43.282178+0000","last_peered":"2024-03-26T03:59:43.282178+0000","last_clean":"2024-03-26T03:59:43.282178+0000","last_became_active":"2024-03-26T03:59:23.521399+0000","last_became_peered":"2024-03-26T03:59:23.521399+0000","last_unstale":"2024-03-26T03:59:43.282178+0000","last_undegraded":"2024-03-26T03:59:43.282178+0000","last_fullsized":"2024-03-26T03:59:43.282178+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1809,"log_dups_size":0,"ondisk_log_size":1809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.327375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":118554624,"num_objects":1809,"num_object_clones":0,"num_object_copies":5427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1809,"num_write_kb":115776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":392,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.586475+0000","last_change":"2024-03-26T03:59:30.465692+0000","last_active":"2024-03-26T03:59:35.586475+0000","last_peered":"2024-03-26T03:59:35.586475+0000","last_clean":"2024-03-26T03:59:35.586475+0000","last_became_active":"2024-03-26T03:58:40.502368+0000","last_became_peered":"2024-03-26T03:58:40.502368+0000","last_unstale":"2024-03-26T03:59:35.586475+0000","last_undegraded":"2024-03-26T03:59:35.586475+0000","last_fullsized":"2024-03-26T03:59:35.586475+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.465607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.465607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.813985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":442,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.586549+0000","last_change":"2024-03-26T03:58:16.032385+0000","last_active":"2024-03-26T03:59:35.586549+0000","last_peered":"2024-03-26T03:59:35.586549+0000","last_clean":"2024-03-26T03:59:35.586549+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T03:59:35.586549+0000","last_undegraded":"2024-03-26T03:59:35.586549+0000","last_fullsized":"2024-03-26T03:59:35.586549+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.032069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.032069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:41.260369+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":410,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581189+0000","last_change":"2024-03-26T03:58:40.500096+0000","last_active":"2024-03-26T03:59:35.581189+0000","last_peered":"2024-03-26T03:59:35.581189+0000","last_clean":"2024-03-26T03:59:35.581189+0000","last_became_active":"2024-03-26T03:58:40.499811+0000","last_became_peered":"2024-03-26T03:58:40.499811+0000","last_unstale":"2024-03-26T03:59:35.581189+0000","last_undegraded":"2024-03-26T03:59:35.581189+0000","last_fullsized":"2024-03-26T03:59:35.581189+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:29.010725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.010725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.625132+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"241'41","reported_seq":480,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.592693+0000","last_change":"2024-03-26T03:59:23.343922+0000","last_active":"2024-03-26T03:59:35.592693+0000","last_peered":"2024-03-26T03:59:35.592693+0000","last_clean":"2024-03-26T03:59:35.592693+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T03:59:35.592693+0000","last_undegraded":"2024-03-26T03:59:35.592693+0000","last_fullsized":"2024-03-26T03:59:35.592693+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-26T03:59:23.343802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.343802+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.007422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"259'1780","reported_seq":1802,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:43.287932+0000","last_change":"2024-03-26T03:59:23.932187+0000","last_active":"2024-03-26T03:59:43.287932+0000","last_peered":"2024-03-26T03:59:43.287932+0000","last_clean":"2024-03-26T03:59:43.287932+0000","last_became_active":"2024-03-26T03:59:23.931234+0000","last_became_peered":"2024-03-26T03:59:23.931234+0000","last_unstale":"2024-03-26T03:59:43.287932+0000","last_undegraded":"2024-03-26T03:59:43.287932+0000","last_fullsized":"2024-03-26T03:59:43.287932+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1780,"log_dups_size":0,"ondisk_log_size":1780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.549151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1780,"num_write_kb":113920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":414,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589469+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T03:59:35.589469+0000","last_peered":"2024-03-26T03:59:35.589469+0000","last_clean":"2024-03-26T03:59:35.589469+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T03:59:35.589469+0000","last_undegraded":"2024-03-26T03:59:35.589469+0000","last_fullsized":"2024-03-26T03:59:35.589469+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":453,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587590+0000","last_change":"2024-03-26T03:59:16.377443+0000","last_active":"2024-03-26T03:59:35.587590+0000","last_peered":"2024-03-26T03:59:35.587590+0000","last_clean":"2024-03-26T03:59:35.587590+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T03:59:35.587590+0000","last_undegraded":"2024-03-26T03:59:35.587590+0000","last_fullsized":"2024-03-26T03:59:35.587590+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:16.377329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.377329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:36.622396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":408,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580365+0000","last_change":"2024-03-26T03:57:49.368242+0000","last_active":"2024-03-26T03:59:35.580365+0000","last_peered":"2024-03-26T03:59:35.580365+0000","last_clean":"2024-03-26T03:59:35.580365+0000","last_became_active":"2024-03-26T03:57:49.367885+0000","last_became_peered":"2024-03-26T03:57:49.367885+0000","last_unstale":"2024-03-26T03:59:35.580365+0000","last_undegraded":"2024-03-26T03:59:35.580365+0000","last_fullsized":"2024-03-26T03:59:35.580365+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421367+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":493,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.589005+0000","last_change":"2024-03-26T03:59:22.808667+0000","last_active":"2024-03-26T03:59:35.589005+0000","last_peered":"2024-03-26T03:59:35.589005+0000","last_clean":"2024-03-26T03:59:35.589005+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T03:59:35.589005+0000","last_undegraded":"2024-03-26T03:59:35.589005+0000","last_fullsized":"2024-03-26T03:59:35.589005+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T03:59:22.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.808562+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.465010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"259'1358","reported_seq":1380,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:40.202234+0000","last_change":"2024-03-26T03:59:23.520900+0000","last_active":"2024-03-26T03:59:40.202234+0000","last_peered":"2024-03-26T03:59:40.202234+0000","last_clean":"2024-03-26T03:59:40.202234+0000","last_became_active":"2024-03-26T03:59:23.520581+0000","last_became_peered":"2024-03-26T03:59:23.520581+0000","last_unstale":"2024-03-26T03:59:40.202234+0000","last_undegraded":"2024-03-26T03:59:40.202234+0000","last_fullsized":"2024-03-26T03:59:40.202234+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1358,"log_dups_size":0,"ondisk_log_size":1358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.337899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88997888,"num_objects":1358,"num_object_clones":0,"num_object_copies":4074,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1358,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1358,"num_write_kb":86912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585391+0000","last_change":"2024-03-26T03:57:49.361707+0000","last_active":"2024-03-26T03:59:35.585391+0000","last_peered":"2024-03-26T03:59:35.585391+0000","last_clean":"2024-03-26T03:59:35.585391+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T03:59:35.585391+0000","last_undegraded":"2024-03-26T03:59:35.585391+0000","last_fullsized":"2024-03-26T03:59:35.585391+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":385,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580866+0000","last_change":"2024-03-26T03:59:27.539611+0000","last_active":"2024-03-26T03:59:35.580866+0000","last_peered":"2024-03-26T03:59:35.580866+0000","last_clean":"2024-03-26T03:59:35.580866+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T03:59:35.580866+0000","last_undegraded":"2024-03-26T03:59:35.580866+0000","last_fullsized":"2024-03-26T03:59:35.580866+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.539528+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.024809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":432,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594877+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T03:59:35.594877+0000","last_peered":"2024-03-26T03:59:35.594877+0000","last_clean":"2024-03-26T03:59:35.594877+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T03:59:35.594877+0000","last_undegraded":"2024-03-26T03:59:35.594877+0000","last_fullsized":"2024-03-26T03:59:35.594877+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341655+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"241'48","reported_seq":698,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.592545+0000","last_change":"2024-03-26T03:59:18.299385+0000","last_active":"2024-03-26T03:59:35.592545+0000","last_peered":"2024-03-26T03:59:35.592545+0000","last_clean":"2024-03-26T03:59:35.592545+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T03:59:35.592545+0000","last_undegraded":"2024-03-26T03:59:35.592545+0000","last_fullsized":"2024-03-26T03:59:35.592545+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T03:59:18.299233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.299233+0000","objects_scrubbed":11,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.944708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":426,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588039+0000","last_change":"2024-03-26T03:59:29.728348+0000","last_active":"2024-03-26T03:59:35.588039+0000","last_peered":"2024-03-26T03:59:35.588039+0000","last_clean":"2024-03-26T03:59:35.588039+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T03:59:35.588039+0000","last_undegraded":"2024-03-26T03:59:35.588039+0000","last_fullsized":"2024-03-26T03:59:35.588039+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.728266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.728266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:32.334402+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"259'1769","reported_seq":1790,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:43.293021+0000","last_change":"2024-03-26T03:59:23.522695+0000","last_active":"2024-03-26T03:59:43.293021+0000","last_peered":"2024-03-26T03:59:43.293021+0000","last_clean":"2024-03-26T03:59:43.293021+0000","last_became_active":"2024-03-26T03:59:23.522433+0000","last_became_peered":"2024-03-26T03:59:23.522433+0000","last_unstale":"2024-03-26T03:59:43.293021+0000","last_undegraded":"2024-03-26T03:59:43.293021+0000","last_fullsized":"2024-03-26T03:59:43.293021+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1769,"log_dups_size":0,"ondisk_log_size":1769,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.498587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115933184,"num_objects":1769,"num_object_clones":0,"num_object_copies":5307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1769,"num_write_kb":113216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":449,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583985+0000","last_change":"2024-03-26T03:58:41.051187+0000","last_active":"2024-03-26T03:59:35.583985+0000","last_peered":"2024-03-26T03:59:35.583985+0000","last_clean":"2024-03-26T03:59:35.583985+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T03:59:35.583985+0000","last_undegraded":"2024-03-26T03:59:35.583985+0000","last_fullsized":"2024-03-26T03:59:35.583985+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.050966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.050966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:48.348801+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":446,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588086+0000","last_change":"2024-03-26T03:59:30.302063+0000","last_active":"2024-03-26T03:59:35.588086+0000","last_peered":"2024-03-26T03:59:35.588086+0000","last_clean":"2024-03-26T03:59:35.588086+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T03:59:35.588086+0000","last_undegraded":"2024-03-26T03:59:35.588086+0000","last_fullsized":"2024-03-26T03:59:35.588086+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.301949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.301949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.722337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":463,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585873+0000","last_change":"2024-03-26T03:58:11.071645+0000","last_active":"2024-03-26T03:59:35.585873+0000","last_peered":"2024-03-26T03:59:35.585873+0000","last_clean":"2024-03-26T03:59:35.585873+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T03:59:35.585873+0000","last_undegraded":"2024-03-26T03:59:35.585873+0000","last_fullsized":"2024-03-26T03:59:35.585873+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.071522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.071522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.381109+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'23","reported_seq":474,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590857+0000","last_change":"2024-03-26T03:58:29.467581+0000","last_active":"2024-03-26T03:59:35.590857+0000","last_peered":"2024-03-26T03:59:35.590857+0000","last_clean":"2024-03-26T03:59:35.590857+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T03:59:35.590857+0000","last_undegraded":"2024-03-26T03:59:35.590857+0000","last_fullsized":"2024-03-26T03:59:35.590857+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:58:29.467279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.467279+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":435,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587163+0000","last_change":"2024-03-26T03:58:08.985333+0000","last_active":"2024-03-26T03:59:35.587163+0000","last_peered":"2024-03-26T03:59:35.587163+0000","last_clean":"2024-03-26T03:59:35.587163+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T03:59:35.587163+0000","last_undegraded":"2024-03-26T03:59:35.587163+0000","last_fullsized":"2024-03-26T03:59:35.587163+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:08.985207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:08.985207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"259'1248","reported_seq":1270,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.895931+0000","last_change":"2024-03-26T03:59:23.518791+0000","last_active":"2024-03-26T03:59:39.895931+0000","last_peered":"2024-03-26T03:59:39.895931+0000","last_clean":"2024-03-26T03:59:39.895931+0000","last_became_active":"2024-03-26T03:59:23.518459+0000","last_became_peered":"2024-03-26T03:59:23.518459+0000","last_unstale":"2024-03-26T03:59:39.895931+0000","last_undegraded":"2024-03-26T03:59:39.895931+0000","last_fullsized":"2024-03-26T03:59:39.895931+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:48.705785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":79872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":390,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581610+0000","last_change":"2024-03-26T03:59:21.654528+0000","last_active":"2024-03-26T03:59:35.581610+0000","last_peered":"2024-03-26T03:59:35.581610+0000","last_clean":"2024-03-26T03:59:35.581610+0000","last_became_active":"2024-03-26T03:57:49.367895+0000","last_became_peered":"2024-03-26T03:57:49.367895+0000","last_unstale":"2024-03-26T03:59:35.581610+0000","last_undegraded":"2024-03-26T03:59:35.581610+0000","last_fullsized":"2024-03-26T03:59:35.581610+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:21.654428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.654428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:23.596896+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":698,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583210+0000","last_change":"2024-03-26T03:58:39.055551+0000","last_active":"2024-03-26T03:59:35.583210+0000","last_peered":"2024-03-26T03:59:35.583210+0000","last_clean":"2024-03-26T03:59:35.583210+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T03:59:35.583210+0000","last_undegraded":"2024-03-26T03:59:35.583210+0000","last_fullsized":"2024-03-26T03:59:35.583210+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:58:39.055403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.055403+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:01.714922+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584303+0000","last_change":"2024-03-26T03:59:29.150185+0000","last_active":"2024-03-26T03:59:35.584303+0000","last_peered":"2024-03-26T03:59:35.584303+0000","last_clean":"2024-03-26T03:59:35.584303+0000","last_became_active":"2024-03-26T03:57:49.369463+0000","last_became_peered":"2024-03-26T03:57:49.369463+0000","last_unstale":"2024-03-26T03:59:35.584303+0000","last_undegraded":"2024-03-26T03:59:35.584303+0000","last_fullsized":"2024-03-26T03:59:35.584303+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.150101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.150101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:38.172188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":542,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595551+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T03:59:35.595551+0000","last_peered":"2024-03-26T03:59:35.595551+0000","last_clean":"2024-03-26T03:59:35.595551+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T03:59:35.595551+0000","last_undegraded":"2024-03-26T03:59:35.595551+0000","last_fullsized":"2024-03-26T03:59:35.595551+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341691+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":324,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587126+0000","last_change":"2024-03-26T03:59:25.418140+0000","last_active":"2024-03-26T03:59:35.587126+0000","last_peered":"2024-03-26T03:59:35.587126+0000","last_clean":"2024-03-26T03:59:35.587126+0000","last_became_active":"2024-03-26T03:58:40.497499+0000","last_became_peered":"2024-03-26T03:58:40.497499+0000","last_unstale":"2024-03-26T03:59:35.587126+0000","last_undegraded":"2024-03-26T03:59:35.587126+0000","last_fullsized":"2024-03-26T03:59:35.587126+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.418009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.418009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:52.675582+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"259'1364","reported_seq":1386,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:40.201194+0000","last_change":"2024-03-26T03:59:23.931889+0000","last_active":"2024-03-26T03:59:40.201194+0000","last_peered":"2024-03-26T03:59:40.201194+0000","last_clean":"2024-03-26T03:59:40.201194+0000","last_became_active":"2024-03-26T03:59:23.931557+0000","last_became_peered":"2024-03-26T03:59:23.931557+0000","last_unstale":"2024-03-26T03:59:40.201194+0000","last_undegraded":"2024-03-26T03:59:40.201194+0000","last_fullsized":"2024-03-26T03:59:40.201194+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1364,"log_dups_size":0,"ondisk_log_size":1364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.030948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89391104,"num_objects":1364,"num_object_clones":0,"num_object_copies":4092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1364,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1364,"num_write_kb":87296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":433,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590890+0000","last_change":"2024-03-26T03:58:40.462393+0000","last_active":"2024-03-26T03:59:35.590890+0000","last_peered":"2024-03-26T03:59:35.590890+0000","last_clean":"2024-03-26T03:59:35.590890+0000","last_became_active":"2024-03-26T03:57:49.379569+0000","last_became_peered":"2024-03-26T03:57:49.379569+0000","last_unstale":"2024-03-26T03:59:35.590890+0000","last_undegraded":"2024-03-26T03:59:35.590890+0000","last_fullsized":"2024-03-26T03:59:35.590890+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:40.462316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:40.462316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:55.226953+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":455,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588637+0000","last_change":"2024-03-26T03:57:49.364364+0000","last_active":"2024-03-26T03:59:35.588637+0000","last_peered":"2024-03-26T03:59:35.588637+0000","last_clean":"2024-03-26T03:59:35.588637+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T03:59:35.588637+0000","last_undegraded":"2024-03-26T03:59:35.588637+0000","last_fullsized":"2024-03-26T03:59:35.588637+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:38.694562+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1101,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:37.594296+0000","last_change":"2024-03-26T03:59:09.978226+0000","last_active":"2024-03-26T03:59:37.594296+0000","last_peered":"2024-03-26T03:59:37.594296+0000","last_clean":"2024-03-26T03:59:37.594296+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T03:59:37.594296+0000","last_undegraded":"2024-03-26T03:59:37.594296+0000","last_fullsized":"2024-03-26T03:59:37.594296+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:59:09.978111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:59:09.978111+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:15.087807+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":647,"num_read_kb":642,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":458,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588621+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T03:59:35.588621+0000","last_peered":"2024-03-26T03:59:35.588621+0000","last_clean":"2024-03-26T03:59:35.588621+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T03:59:35.588621+0000","last_undegraded":"2024-03-26T03:59:35.588621+0000","last_fullsized":"2024-03-26T03:59:35.588621+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":371,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581623+0000","last_change":"2024-03-26T03:59:32.538750+0000","last_active":"2024-03-26T03:59:35.581623+0000","last_peered":"2024-03-26T03:59:35.581623+0000","last_clean":"2024-03-26T03:59:35.581623+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T03:59:35.581623+0000","last_undegraded":"2024-03-26T03:59:35.581623+0000","last_fullsized":"2024-03-26T03:59:35.581623+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:32.538628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:32.538628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.022689+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"259'1144","reported_seq":1166,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.269433+0000","last_change":"2024-03-26T03:59:23.930380+0000","last_active":"2024-03-26T03:59:39.269433+0000","last_peered":"2024-03-26T03:59:39.269433+0000","last_clean":"2024-03-26T03:59:39.269433+0000","last_became_active":"2024-03-26T03:59:23.930042+0000","last_became_peered":"2024-03-26T03:59:23.930042+0000","last_unstale":"2024-03-26T03:59:39.269433+0000","last_undegraded":"2024-03-26T03:59:39.269433+0000","last_fullsized":"2024-03-26T03:59:39.269433+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1144,"log_dups_size":0,"ondisk_log_size":1144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:38.160641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74973184,"num_objects":1144,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1144,"num_write_kb":73216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":426,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590201+0000","last_change":"2024-03-26T03:59:24.317142+0000","last_active":"2024-03-26T03:59:35.590201+0000","last_peered":"2024-03-26T03:59:35.590201+0000","last_clean":"2024-03-26T03:59:35.590201+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T03:59:35.590201+0000","last_undegraded":"2024-03-26T03:59:35.590201+0000","last_fullsized":"2024-03-26T03:59:35.590201+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:24.317061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:24.317061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.745057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":441,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.591194+0000","last_change":"2024-03-26T03:58:41.426711+0000","last_active":"2024-03-26T03:59:35.591194+0000","last_peered":"2024-03-26T03:59:35.591194+0000","last_clean":"2024-03-26T03:59:35.591194+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T03:59:35.591194+0000","last_undegraded":"2024-03-26T03:59:35.591194+0000","last_fullsized":"2024-03-26T03:59:35.591194+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.426593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.426593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:06.577500+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"241'35","reported_seq":502,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588832+0000","last_change":"2024-03-26T03:59:25.757129+0000","last_active":"2024-03-26T03:59:35.588832+0000","last_peered":"2024-03-26T03:59:35.588832+0000","last_clean":"2024-03-26T03:59:35.588832+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T03:59:35.588832+0000","last_undegraded":"2024-03-26T03:59:35.588832+0000","last_fullsized":"2024-03-26T03:59:35.588832+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'35","last_scrub_stamp":"2024-03-26T03:59:25.756981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.756981+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.517629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":373,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587987+0000","last_change":"2024-03-26T03:59:27.734291+0000","last_active":"2024-03-26T03:59:35.587987+0000","last_peered":"2024-03-26T03:59:35.587987+0000","last_clean":"2024-03-26T03:59:35.587987+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T03:59:35.587987+0000","last_undegraded":"2024-03-26T03:59:35.587987+0000","last_fullsized":"2024-03-26T03:59:35.587987+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.534460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"259'1748","reported_seq":1770,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:43.285250+0000","last_change":"2024-03-26T03:59:23.523599+0000","last_active":"2024-03-26T03:59:43.285250+0000","last_peered":"2024-03-26T03:59:43.285250+0000","last_clean":"2024-03-26T03:59:43.285250+0000","last_became_active":"2024-03-26T03:59:23.523293+0000","last_became_peered":"2024-03-26T03:59:23.523293+0000","last_unstale":"2024-03-26T03:59:43.285250+0000","last_undegraded":"2024-03-26T03:59:43.285250+0000","last_fullsized":"2024-03-26T03:59:43.285250+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1748,"log_dups_size":0,"ondisk_log_size":1748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.604819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114556928,"num_objects":1748,"num_object_clones":0,"num_object_copies":5244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1748,"num_write_kb":111872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":435,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588826+0000","last_change":"2024-03-26T03:59:13.839565+0000","last_active":"2024-03-26T03:59:35.588826+0000","last_peered":"2024-03-26T03:59:35.588826+0000","last_clean":"2024-03-26T03:59:35.588826+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T03:59:35.588826+0000","last_undegraded":"2024-03-26T03:59:35.588826+0000","last_fullsized":"2024-03-26T03:59:35.588826+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:13.839415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.839415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.685298+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":394,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581140+0000","last_change":"2024-03-26T03:57:49.368358+0000","last_active":"2024-03-26T03:59:35.581140+0000","last_peered":"2024-03-26T03:59:35.581140+0000","last_clean":"2024-03-26T03:59:35.581140+0000","last_became_active":"2024-03-26T03:57:49.367990+0000","last_became_peered":"2024-03-26T03:57:49.367990+0000","last_unstale":"2024-03-26T03:59:35.581140+0000","last_undegraded":"2024-03-26T03:59:35.581140+0000","last_fullsized":"2024-03-26T03:59:35.581140+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421460+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":479,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.593285+0000","last_change":"2024-03-26T03:59:15.299800+0000","last_active":"2024-03-26T03:59:35.593285+0000","last_peered":"2024-03-26T03:59:35.593285+0000","last_clean":"2024-03-26T03:59:35.593285+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T03:59:35.593285+0000","last_undegraded":"2024-03-26T03:59:35.593285+0000","last_fullsized":"2024-03-26T03:59:35.593285+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T03:59:15.299680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.299680+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.178450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"215'13","reported_seq":599,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588563+0000","last_change":"2024-03-26T03:59:16.761537+0000","last_active":"2024-03-26T03:59:35.588563+0000","last_peered":"2024-03-26T03:59:35.588563+0000","last_clean":"2024-03-26T03:59:35.588563+0000","last_became_active":"2024-03-26T03:57:49.361026+0000","last_became_peered":"2024-03-26T03:57:49.361026+0000","last_unstale":"2024-03-26T03:59:35.588563+0000","last_undegraded":"2024-03-26T03:59:35.588563+0000","last_fullsized":"2024-03-26T03:59:35.588563+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'13","last_scrub_stamp":"2024-03-26T03:59:16.761422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.761422+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.563005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"259'1307","reported_seq":1329,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.906501+0000","last_change":"2024-03-26T03:59:23.930568+0000","last_active":"2024-03-26T03:59:39.906501+0000","last_peered":"2024-03-26T03:59:39.906501+0000","last_clean":"2024-03-26T03:59:39.906501+0000","last_became_active":"2024-03-26T03:59:23.930275+0000","last_became_peered":"2024-03-26T03:59:23.930275+0000","last_unstale":"2024-03-26T03:59:39.906501+0000","last_undegraded":"2024-03-26T03:59:39.906501+0000","last_fullsized":"2024-03-26T03:59:39.906501+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1307,"log_dups_size":0,"ondisk_log_size":1307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:42.507204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85655552,"num_objects":1307,"num_object_clones":0,"num_object_copies":3921,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1307,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1307,"num_write_kb":83648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":428,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584088+0000","last_change":"2024-03-26T03:58:23.160475+0000","last_active":"2024-03-26T03:59:35.584088+0000","last_peered":"2024-03-26T03:59:35.584088+0000","last_clean":"2024-03-26T03:59:35.584088+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T03:59:35.584088+0000","last_undegraded":"2024-03-26T03:59:35.584088+0000","last_fullsized":"2024-03-26T03:59:35.584088+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:23.160383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T03:58:23.160383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":435,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.588638+0000","last_change":"2024-03-26T03:59:14.764278+0000","last_active":"2024-03-26T03:59:35.588638+0000","last_peered":"2024-03-26T03:59:35.588638+0000","last_clean":"2024-03-26T03:59:35.588638+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T03:59:35.588638+0000","last_undegraded":"2024-03-26T03:59:35.588638+0000","last_fullsized":"2024-03-26T03:59:35.588638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:59:14.764155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.764155+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.349897+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":509,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594611+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T03:59:35.594611+0000","last_peered":"2024-03-26T03:59:35.594611+0000","last_clean":"2024-03-26T03:59:35.594611+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T03:59:35.594611+0000","last_undegraded":"2024-03-26T03:59:35.594611+0000","last_fullsized":"2024-03-26T03:59:35.594611+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341512+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":125,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583578+0000","last_change":"2024-03-26T03:57:49.461921+0000","last_active":"2024-03-26T03:59:35.583578+0000","last_peered":"2024-03-26T03:59:35.583578+0000","last_clean":"2024-03-26T03:59:35.583578+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T03:59:35.583578+0000","last_undegraded":"2024-03-26T03:59:35.583578+0000","last_fullsized":"2024-03-26T03:59:35.583578+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"259'1274","reported_seq":1296,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.270614+0000","last_change":"2024-03-26T03:59:23.525085+0000","last_active":"2024-03-26T03:59:39.270614+0000","last_peered":"2024-03-26T03:59:39.270614+0000","last_clean":"2024-03-26T03:59:39.270614+0000","last_became_active":"2024-03-26T03:59:23.524729+0000","last_became_peered":"2024-03-26T03:59:23.524729+0000","last_unstale":"2024-03-26T03:59:39.270614+0000","last_undegraded":"2024-03-26T03:59:39.270614+0000","last_fullsized":"2024-03-26T03:59:39.270614+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1274,"log_dups_size":0,"ondisk_log_size":1274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:35.776916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1274,"num_write_kb":81536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":418,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594325+0000","last_change":"2024-03-26T03:57:49.360270+0000","last_active":"2024-03-26T03:59:35.594325+0000","last_peered":"2024-03-26T03:59:35.594325+0000","last_clean":"2024-03-26T03:59:35.594325+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T03:59:35.594325+0000","last_undegraded":"2024-03-26T03:59:35.594325+0000","last_fullsized":"2024-03-26T03:59:35.594325+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341164+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.585436+0000","last_change":"2024-03-26T03:59:12.184503+0000","last_active":"2024-03-26T03:59:35.585436+0000","last_peered":"2024-03-26T03:59:35.585436+0000","last_clean":"2024-03-26T03:59:35.585436+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T03:59:35.585436+0000","last_undegraded":"2024-03-26T03:59:35.585436+0000","last_fullsized":"2024-03-26T03:59:35.585436+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.184357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.184357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.537755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":146,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583822+0000","last_change":"2024-03-26T03:59:21.245414+0000","last_active":"2024-03-26T03:59:35.583822+0000","last_peered":"2024-03-26T03:59:35.583822+0000","last_clean":"2024-03-26T03:59:35.583822+0000","last_became_active":"2024-03-26T03:57:49.398824+0000","last_became_peered":"2024-03-26T03:57:49.398824+0000","last_unstale":"2024-03-26T03:59:35.583822+0000","last_undegraded":"2024-03-26T03:59:35.583822+0000","last_fullsized":"2024-03-26T03:59:35.583822+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T03:59:21.245251+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.245251+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:31.393654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":430,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594218+0000","last_change":"2024-03-26T03:57:49.360193+0000","last_active":"2024-03-26T03:59:35.594218+0000","last_peered":"2024-03-26T03:59:35.594218+0000","last_clean":"2024-03-26T03:59:35.594218+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T03:59:35.594218+0000","last_undegraded":"2024-03-26T03:59:35.594218+0000","last_fullsized":"2024-03-26T03:59:35.594218+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"259'1348","reported_seq":1370,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.895705+0000","last_change":"2024-03-26T03:59:23.932941+0000","last_active":"2024-03-26T03:59:39.895705+0000","last_peered":"2024-03-26T03:59:39.895705+0000","last_clean":"2024-03-26T03:59:39.895705+0000","last_became_active":"2024-03-26T03:59:23.932638+0000","last_became_peered":"2024-03-26T03:59:23.932638+0000","last_unstale":"2024-03-26T03:59:39.895705+0000","last_undegraded":"2024-03-26T03:59:39.895705+0000","last_fullsized":"2024-03-26T03:59:39.895705+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1348,"log_dups_size":0,"ondisk_log_size":1348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.461400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88342528,"num_objects":1348,"num_object_clones":0,"num_object_copies":4044,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1348,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1348,"num_write_kb":86272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":349,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.580983+0000","last_change":"2024-03-26T03:59:25.561119+0000","last_active":"2024-03-26T03:59:35.580983+0000","last_peered":"2024-03-26T03:59:35.580983+0000","last_clean":"2024-03-26T03:59:35.580983+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T03:59:35.580983+0000","last_undegraded":"2024-03-26T03:59:35.580983+0000","last_fullsized":"2024-03-26T03:59:35.580983+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.561017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.561017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.161594+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":568,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595647+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T03:59:35.595647+0000","last_peered":"2024-03-26T03:59:35.595647+0000","last_clean":"2024-03-26T03:59:35.595647+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T03:59:35.595647+0000","last_undegraded":"2024-03-26T03:59:35.595647+0000","last_fullsized":"2024-03-26T03:59:35.595647+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":106,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584015+0000","last_change":"2024-03-26T03:59:27.171191+0000","last_active":"2024-03-26T03:59:35.584015+0000","last_peered":"2024-03-26T03:59:35.584015+0000","last_clean":"2024-03-26T03:59:35.584015+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T03:59:35.584015+0000","last_undegraded":"2024-03-26T03:59:35.584015+0000","last_fullsized":"2024-03-26T03:59:35.584015+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.171113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.171113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:50.875823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":514,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.590920+0000","last_change":"2024-03-26T03:58:42.470991+0000","last_active":"2024-03-26T03:59:35.590920+0000","last_peered":"2024-03-26T03:59:35.590920+0000","last_clean":"2024-03-26T03:59:35.590920+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T03:59:35.590920+0000","last_undegraded":"2024-03-26T03:59:35.590920+0000","last_fullsized":"2024-03-26T03:59:35.590920+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:42.470857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.470857+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:45.803581+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"259'1171","reported_seq":1191,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.275126+0000","last_change":"2024-03-26T03:59:23.523826+0000","last_active":"2024-03-26T03:59:39.275126+0000","last_peered":"2024-03-26T03:59:39.275126+0000","last_clean":"2024-03-26T03:59:39.275126+0000","last_became_active":"2024-03-26T03:59:23.523509+0000","last_became_peered":"2024-03-26T03:59:23.523509+0000","last_unstale":"2024-03-26T03:59:39.275126+0000","last_undegraded":"2024-03-26T03:59:39.275126+0000","last_fullsized":"2024-03-26T03:59:39.275126+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:50.788562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":76742656,"num_objects":1171,"num_object_clones":0,"num_object_copies":3513,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1171,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1171,"num_write_kb":74944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.584825+0000","last_change":"2024-03-26T03:57:49.373101+0000","last_active":"2024-03-26T03:59:35.584825+0000","last_peered":"2024-03-26T03:59:35.584825+0000","last_clean":"2024-03-26T03:59:35.584825+0000","last_became_active":"2024-03-26T03:57:49.372744+0000","last_became_peered":"2024-03-26T03:57:49.372744+0000","last_unstale":"2024-03-26T03:59:35.584825+0000","last_undegraded":"2024-03-26T03:59:35.584825+0000","last_fullsized":"2024-03-26T03:59:35.584825+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":383,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.581558+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T03:59:35.581558+0000","last_peered":"2024-03-26T03:59:35.581558+0000","last_clean":"2024-03-26T03:59:35.581558+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T03:59:35.581558+0000","last_undegraded":"2024-03-26T03:59:35.581558+0000","last_fullsized":"2024-03-26T03:59:35.581558+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.421400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":426,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594349+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T03:59:35.594349+0000","last_peered":"2024-03-26T03:59:35.594349+0000","last_clean":"2024-03-26T03:59:35.594349+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T03:59:35.594349+0000","last_undegraded":"2024-03-26T03:59:35.594349+0000","last_fullsized":"2024-03-26T03:59:35.594349+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":541,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595395+0000","last_change":"2024-03-26T03:59:14.317073+0000","last_active":"2024-03-26T03:59:35.595395+0000","last_peered":"2024-03-26T03:59:35.595395+0000","last_clean":"2024-03-26T03:59:35.595395+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T03:59:35.595395+0000","last_undegraded":"2024-03-26T03:59:35.595395+0000","last_fullsized":"2024-03-26T03:59:35.595395+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T03:59:14.316934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.316934+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.183304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"259'1326","reported_seq":1348,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:40.338019+0000","last_change":"2024-03-26T03:59:23.931746+0000","last_active":"2024-03-26T03:59:40.338019+0000","last_peered":"2024-03-26T03:59:40.338019+0000","last_clean":"2024-03-26T03:59:40.338019+0000","last_became_active":"2024-03-26T03:59:23.931459+0000","last_became_peered":"2024-03-26T03:59:23.931459+0000","last_unstale":"2024-03-26T03:59:40.338019+0000","last_undegraded":"2024-03-26T03:59:40.338019+0000","last_fullsized":"2024-03-26T03:59:40.338019+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1326,"log_dups_size":0,"ondisk_log_size":1326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:29.777517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86900736,"num_objects":1326,"num_object_clones":0,"num_object_copies":3978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1326,"num_write_kb":84864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":449,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587867+0000","last_change":"2024-03-26T03:58:31.314191+0000","last_active":"2024-03-26T03:59:35.587867+0000","last_peered":"2024-03-26T03:59:35.587867+0000","last_clean":"2024-03-26T03:59:35.587867+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T03:59:35.587867+0000","last_undegraded":"2024-03-26T03:59:35.587867+0000","last_fullsized":"2024-03-26T03:59:35.587867+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:31.314073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:31.314073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:57.668269+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":423,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.592829+0000","last_change":"2024-03-26T03:59:19.316596+0000","last_active":"2024-03-26T03:59:35.592829+0000","last_peered":"2024-03-26T03:59:35.592829+0000","last_clean":"2024-03-26T03:59:35.592829+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T03:59:35.592829+0000","last_undegraded":"2024-03-26T03:59:35.592829+0000","last_fullsized":"2024-03-26T03:59:35.592829+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:19.316523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.316523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:44.794322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":385,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.594417+0000","last_change":"2024-03-26T03:59:27.313345+0000","last_active":"2024-03-26T03:59:35.594417+0000","last_peered":"2024-03-26T03:59:35.594417+0000","last_clean":"2024-03-26T03:59:35.594417+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T03:59:35.594417+0000","last_undegraded":"2024-03-26T03:59:35.594417+0000","last_fullsized":"2024-03-26T03:59:35.594417+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.313255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.313255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:46.779259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":478,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595668+0000","last_change":"2024-03-26T03:59:26.340091+0000","last_active":"2024-03-26T03:59:35.595668+0000","last_peered":"2024-03-26T03:59:35.595668+0000","last_clean":"2024-03-26T03:59:35.595668+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T03:59:35.595668+0000","last_undegraded":"2024-03-26T03:59:35.595668+0000","last_fullsized":"2024-03-26T03:59:35.595668+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:26.339621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.339621+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:46.853603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"259'1336","reported_seq":1357,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:39.912160+0000","last_change":"2024-03-26T03:59:23.930540+0000","last_active":"2024-03-26T03:59:39.912160+0000","last_peered":"2024-03-26T03:59:39.912160+0000","last_clean":"2024-03-26T03:59:39.912160+0000","last_became_active":"2024-03-26T03:59:23.930238+0000","last_became_peered":"2024-03-26T03:59:23.930238+0000","last_unstale":"2024-03-26T03:59:39.912160+0000","last_undegraded":"2024-03-26T03:59:39.912160+0000","last_fullsized":"2024-03-26T03:59:39.912160+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":1336,"log_dups_size":0,"ondisk_log_size":1336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.352378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":87556096,"num_objects":1336,"num_object_clones":0,"num_object_copies":4008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1336,"num_write_kb":85504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":413,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595677+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T03:59:35.595677+0000","last_peered":"2024-03-26T03:59:35.595677+0000","last_clean":"2024-03-26T03:59:35.595677+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T03:59:35.595677+0000","last_undegraded":"2024-03-26T03:59:35.595677+0000","last_fullsized":"2024-03-26T03:59:35.595677+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341202+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":584,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.587347+0000","last_change":"2024-03-26T03:58:33.177842+0000","last_active":"2024-03-26T03:59:35.587347+0000","last_peered":"2024-03-26T03:59:35.587347+0000","last_clean":"2024-03-26T03:59:35.587347+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T03:59:35.587347+0000","last_undegraded":"2024-03-26T03:59:35.587347+0000","last_fullsized":"2024-03-26T03:59:35.587347+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:58:33.177720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:33.177720+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:39.489885+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":98,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.583891+0000","last_change":"2024-03-26T03:57:49.360595+0000","last_active":"2024-03-26T03:59:35.583891+0000","last_peered":"2024-03-26T03:59:35.583891+0000","last_clean":"2024-03-26T03:59:35.583891+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T03:59:35.583891+0000","last_undegraded":"2024-03-26T03:59:35.583891+0000","last_fullsized":"2024-03-26T03:59:35.583891+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":505,"reported_epoch":259,"state":"active+clean","last_fresh":"2024-03-26T03:59:35.595778+0000","last_change":"2024-03-26T03:57:49.507032+0000","last_active":"2024-03-26T03:59:35.595778+0000","last_peered":"2024-03-26T03:59:35.595778+0000","last_clean":"2024-03-26T03:59:35.595778+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T03:59:35.595778+0000","last_undegraded":"2024-03-26T03:59:35.595778+0000","last_fullsized":"2024-03-26T03:59:35.595778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T03:59:40.341594+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1484783616,"num_objects":22656,"num_object_clones":0,"num_object_copies":67968,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22656,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22656,"num_write_kb":1449984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4343332864,"data_stored":4343332864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22656,"ondisk_log_size":22656,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":647,"num_read_kb":642,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1408,"num_read_kb":1133,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070599,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":959272,"kb_used_data":679584,"kb_used_omap":1,"kb_used_meta":279678,"kb_avail":92785880,"statfs":{"total":95995035648,"available":95012741120,"internally_reserved":0,"allocated":695894016,"data_stored":695495439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":286390730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364941,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575014,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1067968,"kb_used_data":707768,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92677184,"statfs":{"total":95995035648,"available":94901436416,"internally_reserved":0,"allocated":724754432,"data_stored":724330432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738539,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":772856,"kb_used_data":404336,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92972296,"statfs":{"total":95995035648,"available":95203631104,"internally_reserved":0,"allocated":414040064,"data_stored":413625225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411489,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":713928,"kb_used_data":313472,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":93031224,"statfs":{"total":95995035648,"available":95263973376,"internally_reserved":0,"allocated":320995328,"data_stored":320626801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182555,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949368,"kb_used_data":621104,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92795784,"statfs":{"total":95995035648,"available":95022882816,"internally_reserved":0,"allocated":636010496,"data_stored":635595037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":336132551},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,0,0,0,0,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953622,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":662568,"kb_used_data":306528,"kb_used_omap":1,"kb_used_meta":356030,"kb_avail":93082584,"statfs":{"total":95995035648,"available":95316566016,"internally_reserved":0,"allocated":313884672,"data_stored":313483867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":364575173},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113326,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1266264,"kb_used_data":957840,"kb_used_omap":4,"kb_used_meta":308411,"kb_avail":92478888,"statfs":{"total":95995035648,"available":94698381312,"internally_reserved":0,"allocated":980828160,"data_stored":980443777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":315813536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":436469760,"data_stored":436469760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":795607040,"data_stored":795607040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":342097920,"data_stored":342097920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":681639936,"data_stored":681639936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":319684608,"data_stored":319684608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1048903680,"data_stored":1048903680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":718929920,"data_stored":718929920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T03:59:47.875 INFO:tasks.thrashosds.thrasher:Setting 5.2 to [7, 4, 6] 2024-03-26T03:59:47.876 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.2', '7', '4', '6'] 2024-03-26T03:59:47.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pg-upmap 5.2 7 4 6 2024-03-26T03:59:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:47.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:47.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:47.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:48.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:48.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 16 16 36254 36238 141.54 112.562 0.0100319 0.0070586 2024-03-26T03:59:48.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:48.183 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:48.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:48.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:48.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:48.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:48.510 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T03:59:48.510 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T03:59:48.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:48.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:48.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:48.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:48.740 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:48 smithi006 ceph-mon[37501]: pgmap v496: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 866 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T03:59:48.740 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:48 smithi006 ceph-mon[29446]: pgmap v496: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 866 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T03:59:48.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:48.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:48 smithi086 ceph-mon[32965]: pgmap v496: 148 pgs: 148 active+clean; 1.8 GiB data, 7.7 GiB used, 618 GiB / 626 GiB avail; 866 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-26T03:59:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:48.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:48.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:48.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:49.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:49.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 17 15 37928 37913 139.372 104.688 0.00700138 0.00716965 2024-03-26T03:59:49.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:49.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:49.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:49.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T03:59:49.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:49.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:49.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:49.460 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T03:59:49.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:49.615 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:49.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:49.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:49.775 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:49 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:49.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:49 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:49 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T03:59:49.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:49.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:49.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:49.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:50.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:50.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 18 16 39492 39476 137.056 97.6875 0.00578372 0.00729067 2024-03-26T03:59:50.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:50.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:50.224 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:50.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:50.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:50.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:50.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:50.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.455 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:50.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:50.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[29446]: pgmap v497: 148 pgs: 148 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[29446]: osdmap e260: 8 total, 8 up, 7 in 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[37501]: pgmap v497: 148 pgs: 148 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:50.577 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:50 smithi006 ceph-mon[37501]: osdmap e260: 8 total, 8 up, 7 in 2024-03-26T03:59:50.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:50.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:50.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:50.727 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:50.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:50.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:50 smithi086 ceph-mon[32965]: pgmap v497: 148 pgs: 148 active+clean; 2.1 GiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.2 KiB/s rd, 137 MiB/s wr, 2.19k op/s 2024-03-26T03:59:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:50 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T03:59:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1856960155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T03:59:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:50 smithi086 ceph-mon[32965]: osdmap e260: 8 total, 8 up, 7 in 2024-03-26T03:59:50.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:50.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:50.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:51.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.010 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:51.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:51.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 19 16 41230 41214 135.559 108.625 0.00778483 0.00737126 2024-03-26T03:59:51.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:51.153 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:51.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:51.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:51.371 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:51.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:51.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:51.487 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T03:59:51.537 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T03:59:51.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:51.580 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T03:59:51.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:51.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:51.624 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T03:59:51.662 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T03:59:51.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:51.706 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T03:59:51.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:51.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:51.747 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T03:59:51.748 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/223565906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:51.748 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/223565906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:51.788 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T03:59:51.793 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T03:59:51.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/223565906' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T03:59:51.858 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:51.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:52.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T03:59:52.071903+0000 min lat: 0.00120616 max lat: 0.11982 avg lat: 0.00745749 2024-03-26T03:59:52.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T03:59:52.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 20 16 42892 42876 133.974 103.875 0.00800142 0.00745749 2024-03-26T03:59:52.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:52.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:52.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:52.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:52.425 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:52.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.564 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:52.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:52.708 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:52.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:52 smithi086 ceph-mon[32965]: pgmap v499: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T03:59:52.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:52.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:52 smithi006 ceph-mon[37501]: pgmap v499: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T03:59:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:52 smithi006 ceph-mon[29446]: pgmap v499: 148 pgs: 148 active+clean; 2.4 GiB data, 9.2 GiB used, 617 GiB / 626 GiB avail; 1023 B/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-26T03:59:52.946 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:52.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T03:59:52.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:52.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:52.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:53.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 21 16 44617 44601 132.728 107.812 0.0114874 0.00752912 2024-03-26T03:59:53.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:53.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:53.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:53.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:53.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:53.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:53.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:53.381 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:53.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:53.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:53.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:53.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:53.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:53.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:53.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:53.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:53.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:53.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:53.935 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:53.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:54.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:54.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:54.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:54.071 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 22 16 46122 46106 130.97 94.0625 0.00566045 0.00763017 2024-03-26T03:59:54.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:54.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:54.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:54.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:54.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:54.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:54.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:54.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:54.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:54.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T03:59:54.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:54.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:54 smithi006 ceph-mon[37501]: pgmap v500: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-26T03:59:54.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:54.641 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 03:59:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:03:59:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T03:59:54.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:54 smithi006 ceph-mon[29446]: pgmap v500: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-26T03:59:54.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:54.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:54.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:54.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:54.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:54 smithi086 ceph-mon[32965]: pgmap v500: 148 pgs: 148 active+clean; 2.5 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 113 MiB/s wr, 1.80k op/s 2024-03-26T03:59:54.823 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T03:59:54.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:54.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:54.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:54.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:54.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:55.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.006 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:55.072 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 23 16 47690 47674 129.533 98 0.00619703 0.00771419 2024-03-26T03:59:55.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 03:59:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T03:59:55.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:55.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:55.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:55.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:55.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:55.419 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:55.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:55.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.577 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:55.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:55.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:55.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:55.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:55.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:55.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T03:59:55.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:55.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:55.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:55.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:56.070 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:56.072 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 24 16 49271 49255 128.252 98.8125 0.0126219 0.00779141 2024-03-26T03:59:56.075 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T03:59:56.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:56.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:56.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:56.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:56.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:56.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:56.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:56.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.516 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:56.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:56.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:56.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:56.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:56.707 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:56 smithi006 ceph-mon[29446]: pgmap v501: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:56.707 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:56 smithi006 ceph-mon[37501]: pgmap v501: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:56.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:56.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:56 smithi086 ceph-mon[32965]: pgmap v501: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:56.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:56.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:56.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:57.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:57.072 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 25 16 50771 50755 126.872 93.75 0.0149854 0.00787659 2024-03-26T03:59:57.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:57.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:57.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:57.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:57.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:57.347 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:57.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:57.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:57.486 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:57.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:57.557 INFO:teuthology.orchestra.run.smithi006.stderr:set 5.2 pg_upmap mapping to [7,4,6] 2024-03-26T03:59:57.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:57.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:57.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:57.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.764 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:57.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:57.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]: dispatch 2024-03-26T03:59:57.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:57.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]: dispatch 2024-03-26T03:59:57.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]: dispatch 2024-03-26T03:59:57.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:57.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T03:59:58.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:58.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:58.072 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 26 16 52304 52288 125.676 95.8125 0.0063587 0.0079516 2024-03-26T03:59:58.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:58.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:58.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T03:59:58.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:58.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T03:59:58.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:58.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:58.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[29446]: pgmap v502: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:58.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]': finished 2024-03-26T03:59:58.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[29446]: osdmap e261: 8 total, 8 up, 7 in 2024-03-26T03:59:58.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:59:58.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[37501]: pgmap v502: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:58.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]': finished 2024-03-26T03:59:58.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[37501]: osdmap e261: 8 total, 8 up, 7 in 2024-03-26T03:59:58.609 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:58 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:59:58.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:58.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:58.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:58.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T03:59:58.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:58 smithi086 ceph-mon[32965]: pgmap v502: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T03:59:58.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1950655992' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.2", "id": [7, 4, 6]}]': finished 2024-03-26T03:59:58.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:58 smithi086 ceph-mon[32965]: osdmap e261: 8 total, 8 up, 7 in 2024-03-26T03:59:58.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:58 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T03:59:58.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T03:59:58.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:58.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:58.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:58.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:59.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T03:59:59.073 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 27 16 53676 53660 124.194 85.75 0.00966577 0.00804716 2024-03-26T03:59:59.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T03:59:59.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T03:59:59.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T03:59:59.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T03:59:59.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:59.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T03:59:59.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:59.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:59.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:59.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T03:59:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T03:59:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T03:59:59.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T03:59:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi086 to 3248M 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: Adjusting osd_memory_target on smithi006 to 3249M 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:59:59.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 03:59:59 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.840 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi086 to 3248M 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: Adjusting osd_memory_target on smithi006 to 3249M 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:59:59.841 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi086 to 3248M 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: Adjusting osd_memory_target on smithi006 to 3249M 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T03:59:59.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 03:59:59 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T03:59:59.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T03:59:59.941 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T03:59:59.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:59.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:00.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:00.073 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 28 16 55175 55159 123.105 93.6875 0.012232 0.00811734 2024-03-26T04:00:00.083 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:00.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:00.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:00.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:00.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:00.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:00.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:00.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:00.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:00.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:00.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[37501]: pgmap v504: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[37501]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[37501]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[29446]: pgmap v504: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[29446]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:00 smithi006 ceph-mon[29446]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:00.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:00.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:00 smithi086 ceph-mon[32965]: pgmap v504: 148 pgs: 148 active+clean; 3.1 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-03-26T04:00:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:00 smithi086 ceph-mon[32965]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:00 smithi086 ceph-mon[32965]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-26T04:00:00.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:00.900 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:00.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:01.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:01.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:01.073 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 29 16 56662 56646 122.064 92.9375 0.0114115 0.00818662 2024-03-26T04:00:01.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:01.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:01.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:01.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:01.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:01.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:01.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:01.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3881933814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:01.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3881933814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:01.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:01.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:01.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:01 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:01 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3881933814' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:01.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:01.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:01.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:02.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:02.073 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 30 16 57932 57916 120.641 79.375 0.00302687 0.00827546 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:00:02.076 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:00:02.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:00:02.095 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:02.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:02.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:02.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:02.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:02.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:02.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:02.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:02.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:02.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:02.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:02.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:02.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:02.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:02 smithi086 ceph-mon[32965]: pgmap v505: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T04:00:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:02 smithi086 ceph-mon[32965]: osdmap e262: 8 total, 8 up, 7 in 2024-03-26T04:00:02.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:02 smithi006 ceph-mon[29446]: pgmap v505: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T04:00:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:02 smithi006 ceph-mon[29446]: osdmap e262: 8 total, 8 up, 7 in 2024-03-26T04:00:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:02 smithi006 ceph-mon[37501]: pgmap v505: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-03-26T04:00:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:02 smithi006 ceph-mon[37501]: osdmap e262: 8 total, 8 up, 7 in 2024-03-26T04:00:02.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:02.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:02.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:03.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:03.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:03.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:03.073 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 31 16 59356 59340 119.62 89 0.00512075 0.00835425 2024-03-26T04:00:03.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:03.194 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:03.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:03.335 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:03.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:03.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:03.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:03.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:03.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:03.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:03.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:03.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:03.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:03.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:03.864 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:03.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:03.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:04.017 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:04.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:00:04.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:00:04.060 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-26T04:00:04.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:00:04.074 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 32 16 60694 60678 118.494 83.625 0.0147973 0.00843414 2024-03-26T04:00:04.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:04.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:04.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:04.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:04.309 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:04.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:04.383 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:00:04.383 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:00:04.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:04.472 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:04.574 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:04.608 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:04.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[29446]: pgmap v507: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T04:00:04.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1713126913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:04.608 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:04.624 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:00:04.625 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:04.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:04.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:04.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:04.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[37501]: pgmap v507: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T04:00:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1713126913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:04.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:04.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:04.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:05.027 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:05.074 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 33 16 62040 62024 117.452 84.125 0.00818795 0.00850907 2024-03-26T04:00:05.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:04 smithi086 ceph-mon[32965]: pgmap v507: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-26T04:00:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1713126913' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:05.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:00:05.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:05.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.163 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:05.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:05.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:05.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:05.438 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:05.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:05.568 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:05.587 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T04:00:05.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:05.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:05.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:00:05.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:05.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:05.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:05.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:00:05.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[29446]: osdmap e263: 8 total, 8 up, 7 in 2024-03-26T04:00:05.875 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:05.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:05.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:00:05.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[37501]: osdmap e263: 8 total, 8 up, 7 in 2024-03-26T04:00:05.875 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:05.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:05.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:05.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:06.055 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:06.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 34 16 63389 63373 116.464 84.3125 0.0196162 0.00858078 2024-03-26T04:00:06.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:05 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:05 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:00:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:05 smithi086 ceph-mon[32965]: osdmap e263: 8 total, 8 up, 7 in 2024-03-26T04:00:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:06.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:06.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:06.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:06.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:06.295 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:06.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:06.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:06.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:06.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:06.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:06.663 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T04:00:06.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T04:00:06.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[37501]: pgmap v509: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T04:00:06.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T04:00:06.664 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T04:00:06.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T04:00:06.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T04:00:06.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[29446]: pgmap v509: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T04:00:06.665 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T04:00:06.666 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:06 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T04:00:06.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:06.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:06.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:06.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:06.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:06.863 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:06.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:06.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:07.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:07.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 35 16 64619 64603 115.327 76.875 0.0135593 0.00866583 2024-03-26T04:00:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:06 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T04:00:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:06 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T04:00:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:06 smithi086 ceph-mon[32965]: pgmap v509: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 115 MiB/s wr, 1.84k op/s 2024-03-26T04:00:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:06 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T04:00:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:06 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T04:00:07.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:07.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:07.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:07.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:07.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:07.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:07.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:07.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:07.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:07.550 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:07.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:07.678 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:07.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:07.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:07 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T04:00:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:07 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T04:00:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:07 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T04:00:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:07 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T04:00:07.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:07.952 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:07.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:07.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:08.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:08.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:00:08.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 36 16 65766 65750 114.115 71.6875 0.0027247 0.00875692 2024-03-26T04:00:08.086 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:07 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T04:00:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:07 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T04:00:08.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:08.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:08.319 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:08.361 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:08.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:08.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:08.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:08.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:08.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:08.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:08.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:08.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: pgmap v510: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[37501]: osdmap e264: 8 total, 8 up, 7 in 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:08.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: pgmap v510: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T04:00:08.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T04:00:08.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T04:00:08.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:08 smithi006 ceph-mon[29446]: osdmap e264: 8 total, 8 up, 7 in 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: pgmap v510: 148 pgs: 148 active+clean; 3.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T04:00:08.893 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T04:00:08.894 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:08 smithi086 ceph-mon[32965]: osdmap e264: 8 total, 8 up, 7 in 2024-03-26T04:00:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:08.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:08.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:09.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:09.031 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:09.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 37 15 66979 66964 113.081 75.875 0.0144296 0.00883831 2024-03-26T04:00:09.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:09.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:09.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:09.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:09.423 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:09.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.484 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:09.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:09.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:09.625 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:09.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:09.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:09.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:09.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:09.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:09.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:09.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:09.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:10.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:10.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:10.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 38 15 68277 68262 112.24 81.125 0.00664629 0.00890433 2024-03-26T04:00:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:10.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:10.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:10.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:10.317 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:10.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:10.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:10.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:10.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:10.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:10.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:10.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:10.726 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:10.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:10.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:10.864 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:10.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: pgmap v512: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T04:00:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: pgmap v512: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T04:00:10.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:10 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T04:00:10.966 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:11.005 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:11.080 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 39 15 69546 69531 111.396 79.3125 0.0102611 0.00897187 2024-03-26T04:00:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T04:00:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T04:00:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T04:00:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: pgmap v512: 148 pgs: 148 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.3 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-03-26T04:00:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T04:00:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T04:00:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:10 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T04:00:11.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:11.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:11.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:11.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:11.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:11.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:11.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:11.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:11.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:11.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:11.595 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 6.2 scrub starts 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 6.2 scrub ok 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T04:00:11.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T04:00:11.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:11.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:11.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:11.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:11.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.845 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:11.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 6.2 scrub starts 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 6.2 scrub ok 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T04:00:11.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:11 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T04:00:11.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:11.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:11.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:12.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:12.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:00:12.081975+0000 min lat: 0.00120616 max lat: 0.11982 avg lat: 0.00904303 2024-03-26T04:00:12.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:00:12.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 40 16 70761 70745 110.506 75.875 0.00522006 0.00904303 2024-03-26T04:00:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 6.2 scrub starts 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 6.2 scrub ok 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T04:00:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T04:00:12.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:11 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T04:00:12.101 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:00:12.104 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:00:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:12.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:12.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:12.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:12.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:12.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.399 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:12.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:12.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:12.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:12.551 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:00:12.552 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:00:12.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:12.672 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:12.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:12.811 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:12.880 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: pgmap v513: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T04:00:12.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:00:12.882 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: pgmap v513: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T04:00:12.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T04:00:12.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T04:00:12.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T04:00:12.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:12 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T04:00:12.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:12.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:12.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:13.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:13.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:13.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 41 16 72048 72032 109.772 80.4375 0.00835896 0.00910458 2024-03-26T04:00:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: pgmap v513: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T04:00:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:12 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T04:00:13.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:13.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:13.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:13.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:13.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:13.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:13.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:13.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:13.745 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:13.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:13.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:13.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:13.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:13.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:14.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:14.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:14.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 42 16 73420 73404 109.2 85.75 0.00718866 0.00915251 2024-03-26T04:00:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:13 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T04:00:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:13 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T04:00:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:13 smithi086 ceph-mon[32965]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:00:14.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:14.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T04:00:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T04:00:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[29446]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:00:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T04:00:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T04:00:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:13 smithi006 ceph-mon[37501]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:00:14.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:14.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:14.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:14.294 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:14.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:14.366 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-03-26T04:00:11.413936+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4427749800,"num_objects":67783,"num_object_clones":0,"num_object_copies":203349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67783,"num_whiteouts":0,"num_read":2321,"num_read_kb":2135,"num_write":68646,"num_write_kb":4325853,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":2368758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68292,"ondisk_log_size":68292,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":15419072,"kb_used_data":12839744,"kb_used_omap":13,"kb_used_meta":2579250,"kb_avail":640796992,"statfs":{"total":671965249536,"available":656176119808,"internally_reserved":0,"allocated":13147897856,"data_stored":13144909336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14011,"internal_metadata":2641152325},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":745668608,"num_objects":11378,"num_object_clones":0,"num_object_copies":34134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11378,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11378,"num_write_kb":728192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002284"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":432,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629845+0000","last_change":"2024-03-26T03:59:12.322291+0000","last_active":"2024-03-26T04:00:07.629845+0000","last_peered":"2024-03-26T04:00:07.629845+0000","last_clean":"2024-03-26T04:00:07.629845+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T04:00:07.629845+0000","last_undegraded":"2024-03-26T04:00:07.629845+0000","last_fullsized":"2024-03-26T04:00:07.629845+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.322148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.322148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:12.821832+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":434,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670267+0000","last_change":"2024-03-26T03:58:37.457198+0000","last_active":"2024-03-26T04:00:07.670267+0000","last_peered":"2024-03-26T04:00:07.670267+0000","last_clean":"2024-03-26T04:00:07.670267+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T04:00:07.670267+0000","last_undegraded":"2024-03-26T04:00:07.670267+0000","last_fullsized":"2024-03-26T04:00:07.670267+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T03:58:37.457112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671357+0000","last_change":"2024-03-26T04:00:07.415630+0000","last_active":"2024-03-26T04:00:07.671357+0000","last_peered":"2024-03-26T04:00:07.671357+0000","last_clean":"2024-03-26T04:00:07.671357+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T04:00:07.671357+0000","last_undegraded":"2024-03-26T04:00:07.671357+0000","last_fullsized":"2024-03-26T04:00:07.671357+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:07.415513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:07.415513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:09.932147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":522,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629701+0000","last_change":"2024-03-26T03:57:49.470002+0000","last_active":"2024-03-26T04:00:07.629701+0000","last_peered":"2024-03-26T04:00:07.629701+0000","last_clean":"2024-03-26T04:00:07.629701+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T04:00:07.629701+0000","last_undegraded":"2024-03-26T04:00:07.629701+0000","last_fullsized":"2024-03-26T04:00:07.629701+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235648+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":108,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639097+0000","last_change":"2024-03-26T03:57:49.377568+0000","last_active":"2024-03-26T04:00:07.639097+0000","last_peered":"2024-03-26T04:00:07.639097+0000","last_clean":"2024-03-26T04:00:07.639097+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T04:00:07.639097+0000","last_undegraded":"2024-03-26T04:00:07.639097+0000","last_fullsized":"2024-03-26T04:00:07.639097+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:20.739949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.739949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629695+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T04:00:07.629695+0000","last_peered":"2024-03-26T04:00:07.629695+0000","last_clean":"2024-03-26T04:00:07.629695+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T04:00:07.629695+0000","last_undegraded":"2024-03-26T04:00:07.629695+0000","last_fullsized":"2024-03-26T04:00:07.629695+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235664+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638979+0000","last_change":"2024-03-26T03:57:49.382349+0000","last_active":"2024-03-26T04:00:07.638979+0000","last_peered":"2024-03-26T04:00:07.638979+0000","last_clean":"2024-03-26T04:00:07.638979+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T04:00:07.638979+0000","last_undegraded":"2024-03-26T04:00:07.638979+0000","last_fullsized":"2024-03-26T04:00:07.638979+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:56.852179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:56.852179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":158,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638406+0000","last_change":"2024-03-26T03:59:19.277977+0000","last_active":"2024-03-26T04:00:07.638406+0000","last_peered":"2024-03-26T04:00:07.638406+0000","last_clean":"2024-03-26T04:00:07.638406+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T04:00:07.638406+0000","last_undegraded":"2024-03-26T04:00:07.638406+0000","last_fullsized":"2024-03-26T04:00:07.638406+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T03:59:19.277872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.277872+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.852334+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":423,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.640036+0000","last_change":"2024-03-26T03:55:31.182181+0000","last_active":"2024-03-26T04:00:07.640036+0000","last_peered":"2024-03-26T04:00:07.640036+0000","last_clean":"2024-03-26T04:00:07.640036+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:00:07.640036+0000","last_undegraded":"2024-03-26T04:00:07.640036+0000","last_fullsized":"2024-03-26T04:00:07.640036+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:31.181879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:31.181879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":416,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638469+0000","last_change":"2024-03-26T03:55:59.311528+0000","last_active":"2024-03-26T04:00:07.638469+0000","last_peered":"2024-03-26T04:00:07.638469+0000","last_clean":"2024-03-26T04:00:07.638469+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T04:00:07.638469+0000","last_undegraded":"2024-03-26T04:00:07.638469+0000","last_fullsized":"2024-03-26T04:00:07.638469+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.213485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.213485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":374,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638624+0000","last_change":"2024-03-26T03:59:20.650716+0000","last_active":"2024-03-26T04:00:07.638624+0000","last_peered":"2024-03-26T04:00:07.638624+0000","last_clean":"2024-03-26T04:00:07.638624+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T04:00:07.638624+0000","last_undegraded":"2024-03-26T04:00:07.638624+0000","last_fullsized":"2024-03-26T04:00:07.638624+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.717054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":529,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629961+0000","last_change":"2024-03-26T03:59:21.358672+0000","last_active":"2024-03-26T04:00:07.629961+0000","last_peered":"2024-03-26T04:00:07.629961+0000","last_clean":"2024-03-26T04:00:07.629961+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T04:00:07.629961+0000","last_undegraded":"2024-03-26T04:00:07.629961+0000","last_fullsized":"2024-03-26T04:00:07.629961+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T03:59:21.358504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.358504+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:39.467201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":390,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638600+0000","last_change":"2024-03-26T03:57:49.381007+0000","last_active":"2024-03-26T04:00:07.638600+0000","last_peered":"2024-03-26T04:00:07.638600+0000","last_clean":"2024-03-26T04:00:07.638600+0000","last_became_active":"2024-03-26T03:57:49.380178+0000","last_became_peered":"2024-03-26T03:57:49.380178+0000","last_unstale":"2024-03-26T04:00:07.638600+0000","last_undegraded":"2024-03-26T04:00:07.638600+0000","last_fullsized":"2024-03-26T04:00:07.638600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:21.709622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:21.709622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482171+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":435,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636392+0000","last_change":"2024-03-26T03:58:28.050117+0000","last_active":"2024-03-26T04:00:07.636392+0000","last_peered":"2024-03-26T04:00:07.636392+0000","last_clean":"2024-03-26T04:00:07.636392+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:00:07.636392+0000","last_undegraded":"2024-03-26T04:00:07.636392+0000","last_fullsized":"2024-03-26T04:00:07.636392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:28.050003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.050003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":391,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638085+0000","last_change":"2024-03-26T03:55:59.296339+0000","last_active":"2024-03-26T04:00:07.638085+0000","last_peered":"2024-03-26T04:00:07.638085+0000","last_clean":"2024-03-26T04:00:07.638085+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T04:00:07.638085+0000","last_undegraded":"2024-03-26T04:00:07.638085+0000","last_fullsized":"2024-03-26T04:00:07.638085+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.784861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.784861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":487,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630156+0000","last_change":"2024-03-26T03:57:49.485971+0000","last_active":"2024-03-26T04:00:07.630156+0000","last_peered":"2024-03-26T04:00:07.630156+0000","last_clean":"2024-03-26T04:00:07.630156+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T04:00:07.630156+0000","last_undegraded":"2024-03-26T04:00:07.630156+0000","last_fullsized":"2024-03-26T04:00:07.630156+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235595+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":451,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630928+0000","last_change":"2024-03-26T03:59:11.325209+0000","last_active":"2024-03-26T04:00:07.630928+0000","last_peered":"2024-03-26T04:00:07.630928+0000","last_clean":"2024-03-26T04:00:07.630928+0000","last_became_active":"2024-03-26T03:57:49.360508+0000","last_became_peered":"2024-03-26T03:57:49.360508+0000","last_unstale":"2024-03-26T04:00:07.630928+0000","last_undegraded":"2024-03-26T04:00:07.630928+0000","last_fullsized":"2024-03-26T04:00:07.630928+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:11.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:11.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:21.774440+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":425,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.680622+0000","last_change":"2024-03-26T03:55:28.882335+0000","last_active":"2024-03-26T04:00:07.680622+0000","last_peered":"2024-03-26T04:00:07.680622+0000","last_clean":"2024-03-26T04:00:07.680622+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:00:07.680622+0000","last_undegraded":"2024-03-26T04:00:07.680622+0000","last_fullsized":"2024-03-26T04:00:07.680622+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:28.882226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:28.882226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":443,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671931+0000","last_change":"2024-03-26T03:55:57.218080+0000","last_active":"2024-03-26T04:00:07.671931+0000","last_peered":"2024-03-26T04:00:07.671931+0000","last_clean":"2024-03-26T04:00:07.671931+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:00:07.671931+0000","last_undegraded":"2024-03-26T04:00:07.671931+0000","last_fullsized":"2024-03-26T04:00:07.671931+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:57.217987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:57.217987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":542,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.625107+0000","last_change":"2024-03-26T03:58:28.238146+0000","last_active":"2024-03-26T04:00:07.625107+0000","last_peered":"2024-03-26T04:00:07.625107+0000","last_clean":"2024-03-26T04:00:07.625107+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T04:00:07.625107+0000","last_undegraded":"2024-03-26T04:00:07.625107+0000","last_fullsized":"2024-03-26T04:00:07.625107+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:28.237979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:28.237979+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641293+0000","last_change":"2024-03-26T03:59:22.373594+0000","last_active":"2024-03-26T04:00:07.641293+0000","last_peered":"2024-03-26T04:00:07.641293+0000","last_clean":"2024-03-26T04:00:07.641293+0000","last_became_active":"2024-03-26T03:57:49.376975+0000","last_became_peered":"2024-03-26T03:57:49.376975+0000","last_unstale":"2024-03-26T04:00:07.641293+0000","last_undegraded":"2024-03-26T04:00:07.641293+0000","last_fullsized":"2024-03-26T04:00:07.641293+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.373494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.373494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:35.705499+0000","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":439,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630484+0000","last_change":"2024-03-26T03:57:49.360798+0000","last_active":"2024-03-26T04:00:07.630484+0000","last_peered":"2024-03-26T04:00:07.630484+0000","last_clean":"2024-03-26T04:00:07.630484+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T04:00:07.630484+0000","last_undegraded":"2024-03-26T04:00:07.630484+0000","last_fullsized":"2024-03-26T04:00:07.630484+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:11.388597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:11.388597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235459+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":443,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630043+0000","last_change":"2024-03-26T03:59:23.332947+0000","last_active":"2024-03-26T04:00:07.630043+0000","last_peered":"2024-03-26T04:00:07.630043+0000","last_clean":"2024-03-26T04:00:07.630043+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T04:00:07.630043+0000","last_undegraded":"2024-03-26T04:00:07.630043+0000","last_fullsized":"2024-03-26T04:00:07.630043+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:23.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.172598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":139,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638377+0000","last_change":"2024-03-26T03:59:13.198844+0000","last_active":"2024-03-26T04:00:07.638377+0000","last_peered":"2024-03-26T04:00:07.638377+0000","last_clean":"2024-03-26T04:00:07.638377+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T04:00:07.638377+0000","last_undegraded":"2024-03-26T04:00:07.638377+0000","last_fullsized":"2024-03-26T04:00:07.638377+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T03:59:13.198735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.198735+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.242387+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":118,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638241+0000","last_change":"2024-03-26T04:00:05.475301+0000","last_active":"2024-03-26T04:00:07.638241+0000","last_peered":"2024-03-26T04:00:07.638241+0000","last_clean":"2024-03-26T04:00:07.638241+0000","last_became_active":"2024-03-26T03:58:40.496528+0000","last_became_peered":"2024-03-26T03:58:40.496528+0000","last_unstale":"2024-03-26T04:00:07.638241+0000","last_undegraded":"2024-03-26T04:00:07.638241+0000","last_fullsized":"2024-03-26T04:00:07.638241+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:00:05.475152+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:00:05.475152+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:31.911000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":514,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630850+0000","last_change":"2024-03-26T03:55:20.194705+0000","last_active":"2024-03-26T04:00:07.630850+0000","last_peered":"2024-03-26T04:00:07.630850+0000","last_clean":"2024-03-26T04:00:07.630850+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T04:00:07.630850+0000","last_undegraded":"2024-03-26T04:00:07.630850+0000","last_fullsized":"2024-03-26T04:00:07.630850+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:55:20.194580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.194580+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":463,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670173+0000","last_change":"2024-03-26T03:59:26.325088+0000","last_active":"2024-03-26T04:00:07.670173+0000","last_peered":"2024-03-26T04:00:07.670173+0000","last_clean":"2024-03-26T04:00:07.670173+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T04:00:07.670173+0000","last_undegraded":"2024-03-26T04:00:07.670173+0000","last_fullsized":"2024-03-26T04:00:07.670173+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.324998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:43.900699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"241'26","reported_seq":514,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671574+0000","last_change":"2024-03-26T03:57:49.470100+0000","last_active":"2024-03-26T04:00:07.671574+0000","last_peered":"2024-03-26T04:00:07.671574+0000","last_clean":"2024-03-26T04:00:07.671574+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T04:00:07.671574+0000","last_undegraded":"2024-03-26T04:00:07.671574+0000","last_fullsized":"2024-03-26T04:00:07.671574+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'16","last_scrub_stamp":"2024-03-26T03:54:36.814666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:36.814666+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":430,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.680293+0000","last_change":"2024-03-26T03:58:39.479856+0000","last_active":"2024-03-26T04:00:07.680293+0000","last_peered":"2024-03-26T04:00:07.680293+0000","last_clean":"2024-03-26T04:00:07.680293+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T04:00:07.680293+0000","last_undegraded":"2024-03-26T04:00:07.680293+0000","last_fullsized":"2024-03-26T04:00:07.680293+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:39.479775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.479775+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":428,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671836+0000","last_change":"2024-03-26T03:54:20.809675+0000","last_active":"2024-03-26T04:00:07.671836+0000","last_peered":"2024-03-26T04:00:07.671836+0000","last_clean":"2024-03-26T04:00:07.671836+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T04:00:07.671836+0000","last_undegraded":"2024-03-26T04:00:07.671836+0000","last_fullsized":"2024-03-26T04:00:07.671836+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:20.809553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T03:54:20.809553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":108,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641258+0000","last_change":"2024-03-26T03:57:49.399354+0000","last_active":"2024-03-26T04:00:07.641258+0000","last_peered":"2024-03-26T04:00:07.641258+0000","last_clean":"2024-03-26T04:00:07.641258+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T04:00:07.641258+0000","last_undegraded":"2024-03-26T04:00:07.641258+0000","last_fullsized":"2024-03-26T04:00:07.641258+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:31.825379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:31.825379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":433,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638714+0000","last_change":"2024-03-26T03:55:59.315416+0000","last_active":"2024-03-26T04:00:07.638714+0000","last_peered":"2024-03-26T04:00:07.638714+0000","last_clean":"2024-03-26T04:00:07.638714+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T04:00:07.638714+0000","last_undegraded":"2024-03-26T04:00:07.638714+0000","last_fullsized":"2024-03-26T04:00:07.638714+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:55:17.124834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.124834+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":431,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636539+0000","last_change":"2024-03-26T03:58:38.043643+0000","last_active":"2024-03-26T04:00:07.636539+0000","last_peered":"2024-03-26T04:00:07.636539+0000","last_clean":"2024-03-26T04:00:07.636539+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T04:00:07.636539+0000","last_undegraded":"2024-03-26T04:00:07.636539+0000","last_fullsized":"2024-03-26T04:00:07.636539+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:38.043527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:38.043527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":366,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629810+0000","last_change":"2024-03-26T03:54:23.295001+0000","last_active":"2024-03-26T04:00:07.629810+0000","last_peered":"2024-03-26T04:00:07.629810+0000","last_clean":"2024-03-26T04:00:07.629810+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:00:07.629810+0000","last_undegraded":"2024-03-26T04:00:07.629810+0000","last_fullsized":"2024-03-26T04:00:07.629810+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.294920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.294920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":456,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.672117+0000","last_change":"2024-03-26T03:57:49.385528+0000","last_active":"2024-03-26T04:00:07.672117+0000","last_peered":"2024-03-26T04:00:07.672117+0000","last_clean":"2024-03-26T04:00:07.672117+0000","last_became_active":"2024-03-26T03:57:49.384847+0000","last_became_peered":"2024-03-26T03:57:49.384847+0000","last_unstale":"2024-03-26T04:00:07.672117+0000","last_undegraded":"2024-03-26T04:00:07.672117+0000","last_fullsized":"2024-03-26T04:00:07.672117+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:14.041793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.041793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":129,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641274+0000","last_change":"2024-03-26T03:59:16.277806+0000","last_active":"2024-03-26T04:00:07.641274+0000","last_peered":"2024-03-26T04:00:07.641274+0000","last_clean":"2024-03-26T04:00:07.641274+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T04:00:07.641274+0000","last_undegraded":"2024-03-26T04:00:07.641274+0000","last_fullsized":"2024-03-26T04:00:07.641274+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T03:59:16.277717+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.277717+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.141535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639010+0000","last_change":"2024-03-26T03:57:49.383008+0000","last_active":"2024-03-26T04:00:07.639010+0000","last_peered":"2024-03-26T04:00:07.639010+0000","last_clean":"2024-03-26T04:00:07.639010+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T04:00:07.639010+0000","last_undegraded":"2024-03-26T04:00:07.639010+0000","last_fullsized":"2024-03-26T04:00:07.639010+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T03:55:29.311918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":441,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636655+0000","last_change":"2024-03-26T03:58:42.072686+0000","last_active":"2024-03-26T04:00:07.636655+0000","last_peered":"2024-03-26T04:00:07.636655+0000","last_clean":"2024-03-26T04:00:07.636655+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:00:07.636655+0000","last_undegraded":"2024-03-26T04:00:07.636655+0000","last_fullsized":"2024-03-26T04:00:07.636655+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:42.072569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.072569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":399,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638339+0000","last_change":"2024-03-26T03:55:59.305819+0000","last_active":"2024-03-26T04:00:07.638339+0000","last_peered":"2024-03-26T04:00:07.638339+0000","last_clean":"2024-03-26T04:00:07.638339+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T04:00:07.638339+0000","last_undegraded":"2024-03-26T04:00:07.638339+0000","last_fullsized":"2024-03-26T04:00:07.638339+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:35.798378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:35.798378+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":143,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638619+0000","last_change":"2024-03-26T03:59:14.247318+0000","last_active":"2024-03-26T04:00:07.638619+0000","last_peered":"2024-03-26T04:00:07.638619+0000","last_clean":"2024-03-26T04:00:07.638619+0000","last_became_active":"2024-03-26T03:57:49.373634+0000","last_became_peered":"2024-03-26T03:57:49.373634+0000","last_unstale":"2024-03-26T04:00:07.638619+0000","last_undegraded":"2024-03-26T04:00:07.638619+0000","last_fullsized":"2024-03-26T04:00:07.638619+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:14.247185+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.247185+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.620476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":440,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.430618+0000","last_change":"2024-03-26T04:00:09.430654+0000","last_active":"2024-03-26T04:00:09.430618+0000","last_peered":"2024-03-26T04:00:09.430618+0000","last_clean":"2024-03-26T04:00:09.430618+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T04:00:09.430618+0000","last_undegraded":"2024-03-26T04:00:09.430618+0000","last_fullsized":"2024-03-26T04:00:09.430618+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:09.430539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:09.430539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:27.699407+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638910+0000","last_change":"2024-03-26T03:57:49.377475+0000","last_active":"2024-03-26T04:00:07.638910+0000","last_peered":"2024-03-26T04:00:07.638910+0000","last_clean":"2024-03-26T04:00:07.638910+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T04:00:07.638910+0000","last_undegraded":"2024-03-26T04:00:07.638910+0000","last_fullsized":"2024-03-26T04:00:07.638910+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:56:00.569877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:56:00.569877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":254,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":455,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.637316+0000","last_change":"2024-03-26T03:58:11.047294+0000","last_active":"2024-03-26T04:00:07.637316+0000","last_peered":"2024-03-26T04:00:07.637316+0000","last_clean":"2024-03-26T04:00:07.637316+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T04:00:07.637316+0000","last_undegraded":"2024-03-26T04:00:07.637316+0000","last_fullsized":"2024-03-26T04:00:07.637316+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:11.047184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:11.047184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":490,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.677207+0000","last_change":"2024-03-26T03:59:27.366321+0000","last_active":"2024-03-26T04:00:07.677207+0000","last_peered":"2024-03-26T04:00:07.677207+0000","last_clean":"2024-03-26T04:00:07.677207+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T04:00:07.677207+0000","last_undegraded":"2024-03-26T04:00:07.677207+0000","last_fullsized":"2024-03-26T04:00:07.677207+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:27.366230+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.366230+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:52.869773+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638348+0000","last_change":"2024-03-26T03:59:28.149466+0000","last_active":"2024-03-26T04:00:07.638348+0000","last_peered":"2024-03-26T04:00:07.638348+0000","last_clean":"2024-03-26T04:00:07.638348+0000","last_became_active":"2024-03-26T03:57:49.370042+0000","last_became_peered":"2024-03-26T03:57:49.370042+0000","last_unstale":"2024-03-26T04:00:07.638348+0000","last_undegraded":"2024-03-26T04:00:07.638348+0000","last_fullsized":"2024-03-26T04:00:07.638348+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.149370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.149370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.085072+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.677024+0000","last_change":"2024-03-26T03:58:20.461781+0000","last_active":"2024-03-26T04:00:07.677024+0000","last_peered":"2024-03-26T04:00:07.677024+0000","last_clean":"2024-03-26T04:00:07.677024+0000","last_became_active":"2024-03-26T03:57:49.379985+0000","last_became_peered":"2024-03-26T03:57:49.379985+0000","last_unstale":"2024-03-26T04:00:07.677024+0000","last_undegraded":"2024-03-26T04:00:07.677024+0000","last_fullsized":"2024-03-26T04:00:07.677024+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:20.461665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:20.461665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630403+0000","last_change":"2024-03-26T03:59:24.336092+0000","last_active":"2024-03-26T04:00:07.630403+0000","last_peered":"2024-03-26T04:00:07.630403+0000","last_clean":"2024-03-26T04:00:07.630403+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T04:00:07.630403+0000","last_undegraded":"2024-03-26T04:00:07.630403+0000","last_fullsized":"2024-03-26T04:00:07.630403+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:24.336010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:24.336010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.514695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630338+0000","last_change":"2024-03-26T03:59:18.367293+0000","last_active":"2024-03-26T04:00:07.630338+0000","last_peered":"2024-03-26T04:00:07.630338+0000","last_clean":"2024-03-26T04:00:07.630338+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T04:00:07.630338+0000","last_undegraded":"2024-03-26T04:00:07.630338+0000","last_fullsized":"2024-03-26T04:00:07.630338+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:18.367180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.367180+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:25.855644+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":435,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671114+0000","last_change":"2024-03-26T03:59:17.820332+0000","last_active":"2024-03-26T04:00:07.671114+0000","last_peered":"2024-03-26T04:00:07.671114+0000","last_clean":"2024-03-26T04:00:07.671114+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T04:00:07.671114+0000","last_undegraded":"2024-03-26T04:00:07.671114+0000","last_fullsized":"2024-03-26T04:00:07.671114+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:17.820216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.820216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.723506+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":552,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638517+0000","last_change":"2024-03-26T03:55:59.341726+0000","last_active":"2024-03-26T04:00:07.638517+0000","last_peered":"2024-03-26T04:00:07.638517+0000","last_clean":"2024-03-26T04:00:07.638517+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T04:00:07.638517+0000","last_undegraded":"2024-03-26T04:00:07.638517+0000","last_fullsized":"2024-03-26T04:00:07.638517+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:55:16.090158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:16.090158+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":451,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636536+0000","last_change":"2024-03-26T03:59:17.888655+0000","last_active":"2024-03-26T04:00:07.636536+0000","last_peered":"2024-03-26T04:00:07.636536+0000","last_clean":"2024-03-26T04:00:07.636536+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T04:00:07.636536+0000","last_undegraded":"2024-03-26T04:00:07.636536+0000","last_fullsized":"2024-03-26T04:00:07.636536+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:17.888578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.888578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:31.797567+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":140,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.640094+0000","last_change":"2024-03-26T03:57:49.435763+0000","last_active":"2024-03-26T04:00:07.640094+0000","last_peered":"2024-03-26T04:00:07.640094+0000","last_clean":"2024-03-26T04:00:07.640094+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T04:00:07.640094+0000","last_undegraded":"2024-03-26T04:00:07.640094+0000","last_fullsized":"2024-03-26T04:00:07.640094+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'22","last_scrub_stamp":"2024-03-26T03:54:32.911285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.911285+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":432,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630899+0000","last_change":"2024-03-26T03:55:59.298979+0000","last_active":"2024-03-26T04:00:07.630899+0000","last_peered":"2024-03-26T04:00:07.630899+0000","last_clean":"2024-03-26T04:00:07.630899+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T04:00:07.630899+0000","last_undegraded":"2024-03-26T04:00:07.630899+0000","last_fullsized":"2024-03-26T04:00:07.630899+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:15.344379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:15.344379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235482+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":142,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.209845+0000","last_change":"2024-03-26T03:57:49.512060+0000","last_active":"2024-03-26T04:00:09.209845+0000","last_peered":"2024-03-26T04:00:09.209845+0000","last_clean":"2024-03-26T04:00:09.209845+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T04:00:09.209845+0000","last_undegraded":"2024-03-26T04:00:09.209845+0000","last_fullsized":"2024-03-26T04:00:09.209845+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-26T03:55:13.804862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:13.804862+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":473,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.640052+0000","last_change":"2024-03-26T03:57:49.480593+0000","last_active":"2024-03-26T04:00:07.640052+0000","last_peered":"2024-03-26T04:00:07.640052+0000","last_clean":"2024-03-26T04:00:07.640052+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T04:00:07.640052+0000","last_undegraded":"2024-03-26T04:00:07.640052+0000","last_fullsized":"2024-03-26T04:00:07.640052+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T03:55:53.234652+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T03:55:53.234652+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":474,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638604+0000","last_change":"2024-03-26T03:59:18.719422+0000","last_active":"2024-03-26T04:00:07.638604+0000","last_peered":"2024-03-26T04:00:07.638604+0000","last_clean":"2024-03-26T04:00:07.638604+0000","last_became_active":"2024-03-26T03:57:49.382991+0000","last_became_peered":"2024-03-26T03:57:49.382991+0000","last_unstale":"2024-03-26T04:00:07.638604+0000","last_undegraded":"2024-03-26T04:00:07.638604+0000","last_fullsized":"2024-03-26T04:00:07.638604+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T03:59:18.719326+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.719326+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:18.748523+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":449,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.632201+0000","last_change":"2024-03-26T03:59:17.399536+0000","last_active":"2024-03-26T04:00:07.632201+0000","last_peered":"2024-03-26T04:00:07.632201+0000","last_clean":"2024-03-26T04:00:07.632201+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T04:00:07.632201+0000","last_undegraded":"2024-03-26T04:00:07.632201+0000","last_fullsized":"2024-03-26T04:00:07.632201+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:59:17.399444+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.399444+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.422968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":108,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641509+0000","last_change":"2024-03-26T03:57:49.368813+0000","last_active":"2024-03-26T04:00:07.641509+0000","last_peered":"2024-03-26T04:00:07.641509+0000","last_clean":"2024-03-26T04:00:07.641509+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T04:00:07.641509+0000","last_undegraded":"2024-03-26T04:00:07.641509+0000","last_fullsized":"2024-03-26T04:00:07.641509+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:19.712285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.712285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":144,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629944+0000","last_change":"2024-03-26T03:58:40.497439+0000","last_active":"2024-03-26T04:00:07.629944+0000","last_peered":"2024-03-26T04:00:07.629944+0000","last_clean":"2024-03-26T04:00:07.629944+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:00:07.629944+0000","last_undegraded":"2024-03-26T04:00:07.629944+0000","last_fullsized":"2024-03-26T04:00:07.629944+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:15.134741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.134741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235681+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":520,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671678+0000","last_change":"2024-03-26T03:57:49.445142+0000","last_active":"2024-03-26T04:00:07.671678+0000","last_peered":"2024-03-26T04:00:07.671678+0000","last_clean":"2024-03-26T04:00:07.671678+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T04:00:07.671678+0000","last_undegraded":"2024-03-26T04:00:07.671678+0000","last_fullsized":"2024-03-26T04:00:07.671678+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'21","last_scrub_stamp":"2024-03-26T03:55:25.160214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.160214+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.637059+0000","last_change":"2024-03-26T03:59:28.005851+0000","last_active":"2024-03-26T04:00:07.637059+0000","last_peered":"2024-03-26T04:00:07.637059+0000","last_clean":"2024-03-26T04:00:07.637059+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T04:00:07.637059+0000","last_undegraded":"2024-03-26T04:00:07.637059+0000","last_fullsized":"2024-03-26T04:00:07.637059+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.005705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.005705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.608573+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":404,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638530+0000","last_change":"2024-03-26T03:59:15.662289+0000","last_active":"2024-03-26T04:00:07.638530+0000","last_peered":"2024-03-26T04:00:07.638530+0000","last_clean":"2024-03-26T04:00:07.638530+0000","last_became_active":"2024-03-26T03:57:49.380167+0000","last_became_peered":"2024-03-26T03:57:49.380167+0000","last_unstale":"2024-03-26T04:00:07.638530+0000","last_undegraded":"2024-03-26T04:00:07.638530+0000","last_fullsized":"2024-03-26T04:00:07.638530+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:15.662211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.662211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.275532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":395,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639951+0000","last_change":"2024-03-26T03:59:20.739789+0000","last_active":"2024-03-26T04:00:07.639951+0000","last_peered":"2024-03-26T04:00:07.639951+0000","last_clean":"2024-03-26T04:00:07.639951+0000","last_became_active":"2024-03-26T03:57:49.377869+0000","last_became_peered":"2024-03-26T03:57:49.377869+0000","last_unstale":"2024-03-26T04:00:07.639951+0000","last_undegraded":"2024-03-26T04:00:07.639951+0000","last_fullsized":"2024-03-26T04:00:07.639951+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:43.234343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":127,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641322+0000","last_change":"2024-03-26T03:57:49.511256+0000","last_active":"2024-03-26T04:00:07.641322+0000","last_peered":"2024-03-26T04:00:07.641322+0000","last_clean":"2024-03-26T04:00:07.641322+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T04:00:07.641322+0000","last_undegraded":"2024-03-26T04:00:07.641322+0000","last_fullsized":"2024-03-26T04:00:07.641322+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-03-26T03:55:19.697070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.697070+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"264'4260","reported_seq":4292,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:10.214692+0000","last_change":"2024-03-26T03:59:23.518905+0000","last_active":"2024-03-26T04:00:10.214692+0000","last_peered":"2024-03-26T04:00:10.214692+0000","last_clean":"2024-03-26T04:00:10.214692+0000","last_became_active":"2024-03-26T03:59:23.518616+0000","last_became_peered":"2024-03-26T03:59:23.518616+0000","last_unstale":"2024-03-26T04:00:10.214692+0000","last_undegraded":"2024-03-26T04:00:10.214692+0000","last_fullsized":"2024-03-26T04:00:10.214692+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4260,"log_dups_size":0,"ondisk_log_size":4260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.143028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279183360,"num_objects":4260,"num_object_clones":0,"num_object_copies":12780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4260,"num_write_kb":272640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":391,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638451+0000","last_change":"2024-03-26T03:59:31.507063+0000","last_active":"2024-03-26T04:00:07.638451+0000","last_peered":"2024-03-26T04:00:07.638451+0000","last_clean":"2024-03-26T04:00:07.638451+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T04:00:07.638451+0000","last_undegraded":"2024-03-26T04:00:07.638451+0000","last_fullsized":"2024-03-26T04:00:07.638451+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:31.506978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:31.506978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.951142+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":456,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636737+0000","last_change":"2024-03-26T03:58:29.964096+0000","last_active":"2024-03-26T04:00:07.636737+0000","last_peered":"2024-03-26T04:00:07.636737+0000","last_clean":"2024-03-26T04:00:07.636737+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:00:07.636737+0000","last_undegraded":"2024-03-26T04:00:07.636737+0000","last_fullsized":"2024-03-26T04:00:07.636737+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T03:58:29.964005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.964005+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":474,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.637923+0000","last_change":"2024-03-26T03:55:59.293102+0000","last_active":"2024-03-26T04:00:07.637923+0000","last_peered":"2024-03-26T04:00:07.637923+0000","last_clean":"2024-03-26T04:00:07.637923+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T04:00:07.637923+0000","last_undegraded":"2024-03-26T04:00:07.637923+0000","last_fullsized":"2024-03-26T04:00:07.637923+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'27","last_scrub_stamp":"2024-03-26T03:54:19.799171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:19.799171+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482139+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":600,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638255+0000","last_change":"2024-03-26T03:59:11.877979+0000","last_active":"2024-03-26T04:00:07.638255+0000","last_peered":"2024-03-26T04:00:07.638255+0000","last_clean":"2024-03-26T04:00:07.638255+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:00:07.638255+0000","last_undegraded":"2024-03-26T04:00:07.638255+0000","last_fullsized":"2024-03-26T04:00:07.638255+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T03:59:11.877821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:11.877821+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.352750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"264'4156","reported_seq":4188,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:10.292869+0000","last_change":"2024-03-26T03:59:23.931890+0000","last_active":"2024-03-26T04:00:10.292869+0000","last_peered":"2024-03-26T04:00:10.292869+0000","last_clean":"2024-03-26T04:00:10.292869+0000","last_became_active":"2024-03-26T03:59:23.931611+0000","last_became_peered":"2024-03-26T03:59:23.931611+0000","last_unstale":"2024-03-26T04:00:10.292869+0000","last_undegraded":"2024-03-26T04:00:10.292869+0000","last_fullsized":"2024-03-26T04:00:10.292869+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.332800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272367616,"num_objects":4156,"num_object_clones":0,"num_object_copies":12468,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4156,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4156,"num_write_kb":265984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638778+0000","last_change":"2024-03-26T03:59:30.159583+0000","last_active":"2024-03-26T04:00:07.638778+0000","last_peered":"2024-03-26T04:00:07.638778+0000","last_clean":"2024-03-26T04:00:07.638778+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T04:00:07.638778+0000","last_undegraded":"2024-03-26T04:00:07.638778+0000","last_fullsized":"2024-03-26T04:00:07.638778+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.159481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.159481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.184050+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":444,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638085+0000","last_change":"2024-03-26T03:58:25.086917+0000","last_active":"2024-03-26T04:00:07.638085+0000","last_peered":"2024-03-26T04:00:07.638085+0000","last_clean":"2024-03-26T04:00:07.638085+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:00:07.638085+0000","last_undegraded":"2024-03-26T04:00:07.638085+0000","last_fullsized":"2024-03-26T04:00:07.638085+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:25.086805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:25.086805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":532,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630048+0000","last_change":"2024-03-26T03:57:49.408864+0000","last_active":"2024-03-26T04:00:07.630048+0000","last_peered":"2024-03-26T04:00:07.630048+0000","last_clean":"2024-03-26T04:00:07.630048+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T04:00:07.630048+0000","last_undegraded":"2024-03-26T04:00:07.630048+0000","last_fullsized":"2024-03-26T04:00:07.630048+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'24","last_scrub_stamp":"2024-03-26T03:55:14.285331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:14.285331+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235557+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":452,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.624856+0000","last_change":"2024-03-26T03:59:22.387740+0000","last_active":"2024-03-26T04:00:07.624856+0000","last_peered":"2024-03-26T04:00:07.624856+0000","last_clean":"2024-03-26T04:00:07.624856+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T04:00:07.624856+0000","last_undegraded":"2024-03-26T04:00:07.624856+0000","last_fullsized":"2024-03-26T04:00:07.624856+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.387664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.387664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.979624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"264'4248","reported_seq":4280,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:08.295854+0000","last_change":"2024-03-26T03:59:23.521792+0000","last_active":"2024-03-26T04:00:08.295854+0000","last_peered":"2024-03-26T04:00:08.295854+0000","last_clean":"2024-03-26T04:00:08.295854+0000","last_became_active":"2024-03-26T03:59:23.521399+0000","last_became_peered":"2024-03-26T03:59:23.521399+0000","last_unstale":"2024-03-26T04:00:08.295854+0000","last_undegraded":"2024-03-26T04:00:08.295854+0000","last_fullsized":"2024-03-26T04:00:08.295854+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.327375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278396928,"num_objects":4248,"num_object_clones":0,"num_object_copies":12744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4248,"num_write_kb":271872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":402,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.625059+0000","last_change":"2024-03-26T03:59:30.465692+0000","last_active":"2024-03-26T04:00:07.625059+0000","last_peered":"2024-03-26T04:00:07.625059+0000","last_clean":"2024-03-26T04:00:07.625059+0000","last_became_active":"2024-03-26T03:58:40.502368+0000","last_became_peered":"2024-03-26T03:58:40.502368+0000","last_unstale":"2024-03-26T04:00:07.625059+0000","last_undegraded":"2024-03-26T04:00:07.625059+0000","last_fullsized":"2024-03-26T04:00:07.625059+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.465607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.465607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.813985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.625044+0000","last_change":"2024-03-26T03:58:16.032385+0000","last_active":"2024-03-26T04:00:07.625044+0000","last_peered":"2024-03-26T04:00:07.625044+0000","last_clean":"2024-03-26T04:00:07.625044+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T04:00:07.625044+0000","last_undegraded":"2024-03-26T04:00:07.625044+0000","last_fullsized":"2024-03-26T04:00:07.625044+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:16.032069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:16.032069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":420,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638838+0000","last_change":"2024-03-26T03:58:40.500096+0000","last_active":"2024-03-26T04:00:07.638838+0000","last_peered":"2024-03-26T04:00:07.638838+0000","last_clean":"2024-03-26T04:00:07.638838+0000","last_became_active":"2024-03-26T03:58:40.499811+0000","last_became_peered":"2024-03-26T03:58:40.499811+0000","last_unstale":"2024-03-26T04:00:07.638838+0000","last_undegraded":"2024-03-26T04:00:07.638838+0000","last_fullsized":"2024-03-26T04:00:07.638838+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:29.010725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.010725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482273+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"241'41","reported_seq":490,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.676859+0000","last_change":"2024-03-26T03:59:23.343922+0000","last_active":"2024-03-26T04:00:07.676859+0000","last_peered":"2024-03-26T04:00:07.676859+0000","last_clean":"2024-03-26T04:00:07.676859+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T04:00:07.676859+0000","last_undegraded":"2024-03-26T04:00:07.676859+0000","last_fullsized":"2024-03-26T04:00:07.676859+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-26T03:59:23.343802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.343802+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.007422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"264'4175","reported_seq":4207,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:08.281112+0000","last_change":"2024-03-26T03:59:23.932187+0000","last_active":"2024-03-26T04:00:08.281112+0000","last_peered":"2024-03-26T04:00:08.281112+0000","last_clean":"2024-03-26T04:00:08.281112+0000","last_became_active":"2024-03-26T03:59:23.931234+0000","last_became_peered":"2024-03-26T03:59:23.931234+0000","last_unstale":"2024-03-26T04:00:08.281112+0000","last_undegraded":"2024-03-26T04:00:08.281112+0000","last_fullsized":"2024-03-26T04:00:08.281112+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4175,"log_dups_size":0,"ondisk_log_size":4175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.549151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273612800,"num_objects":4175,"num_object_clones":0,"num_object_copies":12525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4175,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4175,"num_write_kb":267200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":424,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670068+0000","last_change":"2024-03-26T03:55:24.261371+0000","last_active":"2024-03-26T04:00:07.670068+0000","last_peered":"2024-03-26T04:00:07.670068+0000","last_clean":"2024-03-26T04:00:07.670068+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T04:00:07.670068+0000","last_undegraded":"2024-03-26T04:00:07.670068+0000","last_fullsized":"2024-03-26T04:00:07.670068+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:24.261134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:55:24.261134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":56,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":464,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.625097+0000","last_change":"2024-03-26T03:59:16.377443+0000","last_active":"2024-03-26T04:00:07.625097+0000","last_peered":"2024-03-26T04:00:07.625097+0000","last_clean":"2024-03-26T04:00:07.625097+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T04:00:07.625097+0000","last_undegraded":"2024-03-26T04:00:07.625097+0000","last_fullsized":"2024-03-26T04:00:07.625097+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:16.377329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.377329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:36.622396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":418,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.637991+0000","last_change":"2024-03-26T03:57:49.368242+0000","last_active":"2024-03-26T04:00:07.637991+0000","last_peered":"2024-03-26T04:00:07.637991+0000","last_clean":"2024-03-26T04:00:07.637991+0000","last_became_active":"2024-03-26T03:57:49.367885+0000","last_became_peered":"2024-03-26T03:57:49.367885+0000","last_unstale":"2024-03-26T04:00:07.637991+0000","last_undegraded":"2024-03-26T04:00:07.637991+0000","last_fullsized":"2024-03-26T04:00:07.637991+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:18.820654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T03:54:18.820654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482080+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":504,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.672195+0000","last_change":"2024-03-26T03:59:22.808667+0000","last_active":"2024-03-26T04:00:07.672195+0000","last_peered":"2024-03-26T04:00:07.672195+0000","last_clean":"2024-03-26T04:00:07.672195+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T04:00:07.672195+0000","last_undegraded":"2024-03-26T04:00:07.672195+0000","last_fullsized":"2024-03-26T04:00:07.672195+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T03:59:22.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.808562+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.465010+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"264'4245","reported_seq":4277,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:10.205973+0000","last_change":"2024-03-26T03:59:23.520900+0000","last_active":"2024-03-26T04:00:10.205973+0000","last_peered":"2024-03-26T04:00:10.205973+0000","last_clean":"2024-03-26T04:00:10.205973+0000","last_became_active":"2024-03-26T03:59:23.520581+0000","last_became_peered":"2024-03-26T03:59:23.520581+0000","last_unstale":"2024-03-26T04:00:10.205973+0000","last_undegraded":"2024-03-26T04:00:10.205973+0000","last_fullsized":"2024-03-26T04:00:10.205973+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.337899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278200320,"num_objects":4245,"num_object_clones":0,"num_object_copies":12735,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4245,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4245,"num_write_kb":271680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638879+0000","last_change":"2024-03-26T03:57:49.361707+0000","last_active":"2024-03-26T04:00:07.638879+0000","last_peered":"2024-03-26T04:00:07.638879+0000","last_clean":"2024-03-26T04:00:07.638879+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T04:00:07.638879+0000","last_undegraded":"2024-03-26T04:00:07.638879+0000","last_fullsized":"2024-03-26T04:00:07.638879+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:32.411608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:32.411608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":396,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638383+0000","last_change":"2024-03-26T03:59:27.539611+0000","last_active":"2024-03-26T04:00:07.638383+0000","last_peered":"2024-03-26T04:00:07.638383+0000","last_clean":"2024-03-26T04:00:07.638383+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T04:00:07.638383+0000","last_undegraded":"2024-03-26T04:00:07.638383+0000","last_fullsized":"2024-03-26T04:00:07.638383+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.539528+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.024809+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":442,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630045+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T04:00:07.630045+0000","last_peered":"2024-03-26T04:00:07.630045+0000","last_clean":"2024-03-26T04:00:07.630045+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:00:07.630045+0000","last_undegraded":"2024-03-26T04:00:07.630045+0000","last_fullsized":"2024-03-26T04:00:07.630045+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235611+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"241'48","reported_seq":714,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.676213+0000","last_change":"2024-03-26T03:59:18.299385+0000","last_active":"2024-03-26T04:00:07.676213+0000","last_peered":"2024-03-26T04:00:07.676213+0000","last_clean":"2024-03-26T04:00:07.676213+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T04:00:07.676213+0000","last_undegraded":"2024-03-26T04:00:07.676213+0000","last_fullsized":"2024-03-26T04:00:07.676213+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T03:59:18.299233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.299233+0000","objects_scrubbed":11,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.944708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":436,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.672301+0000","last_change":"2024-03-26T03:59:29.728348+0000","last_active":"2024-03-26T04:00:07.672301+0000","last_peered":"2024-03-26T04:00:07.672301+0000","last_clean":"2024-03-26T04:00:07.672301+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:00:07.672301+0000","last_undegraded":"2024-03-26T04:00:07.672301+0000","last_fullsized":"2024-03-26T04:00:07.672301+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.728266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.728266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:32.334402+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"264'4044","reported_seq":4076,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:08.289778+0000","last_change":"2024-03-26T03:59:23.522695+0000","last_active":"2024-03-26T04:00:08.289778+0000","last_peered":"2024-03-26T04:00:08.289778+0000","last_clean":"2024-03-26T04:00:08.289778+0000","last_became_active":"2024-03-26T03:59:23.522433+0000","last_became_peered":"2024-03-26T03:59:23.522433+0000","last_unstale":"2024-03-26T04:00:08.289778+0000","last_undegraded":"2024-03-26T04:00:08.289778+0000","last_fullsized":"2024-03-26T04:00:08.289778+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.498587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":258816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":459,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.637403+0000","last_change":"2024-03-26T03:58:41.051187+0000","last_active":"2024-03-26T04:00:07.637403+0000","last_peered":"2024-03-26T04:00:07.637403+0000","last_clean":"2024-03-26T04:00:07.637403+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T04:00:07.637403+0000","last_undegraded":"2024-03-26T04:00:07.637403+0000","last_fullsized":"2024-03-26T04:00:07.637403+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.050966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.050966+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":456,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.674071+0000","last_change":"2024-03-26T03:59:30.302063+0000","last_active":"2024-03-26T04:00:07.674071+0000","last_peered":"2024-03-26T04:00:07.674071+0000","last_clean":"2024-03-26T04:00:07.674071+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T04:00:07.674071+0000","last_undegraded":"2024-03-26T04:00:07.674071+0000","last_fullsized":"2024-03-26T04:00:07.674071+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.301949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.301949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.722337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":481,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.624806+0000","last_change":"2024-03-26T04:00:05.018260+0000","last_active":"2024-03-26T04:00:07.624806+0000","last_peered":"2024-03-26T04:00:07.624806+0000","last_clean":"2024-03-26T04:00:07.624806+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T04:00:07.624806+0000","last_undegraded":"2024-03-26T04:00:07.624806+0000","last_fullsized":"2024-03-26T04:00:07.624806+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:05.018129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:00:05.018129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:24.981088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'23","reported_seq":484,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670831+0000","last_change":"2024-03-26T03:58:29.467581+0000","last_active":"2024-03-26T04:00:07.670831+0000","last_peered":"2024-03-26T04:00:07.670831+0000","last_clean":"2024-03-26T04:00:07.670831+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T04:00:07.670831+0000","last_undegraded":"2024-03-26T04:00:07.670831+0000","last_fullsized":"2024-03-26T04:00:07.670831+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-03-26T03:58:29.467279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:29.467279+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":453,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.634261+0000","last_change":"2024-03-26T04:00:06.045528+0000","last_active":"2024-03-26T04:00:07.634261+0000","last_peered":"2024-03-26T04:00:07.634261+0000","last_clean":"2024-03-26T04:00:07.634261+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T04:00:07.634261+0000","last_undegraded":"2024-03-26T04:00:07.634261+0000","last_fullsized":"2024-03-26T04:00:07.634261+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:06.045409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:06.045409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:25.286379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"264'4271","reported_seq":4303,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.936324+0000","last_change":"2024-03-26T03:59:23.518791+0000","last_active":"2024-03-26T04:00:09.936324+0000","last_peered":"2024-03-26T04:00:09.936324+0000","last_clean":"2024-03-26T04:00:09.936324+0000","last_became_active":"2024-03-26T03:59:23.518459+0000","last_became_peered":"2024-03-26T03:59:23.518459+0000","last_unstale":"2024-03-26T04:00:09.936324+0000","last_undegraded":"2024-03-26T04:00:09.936324+0000","last_fullsized":"2024-03-26T04:00:09.936324+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:48.705785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":279904256,"num_objects":4271,"num_object_clones":0,"num_object_copies":12813,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4271,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4271,"num_write_kb":273344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":400,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638808+0000","last_change":"2024-03-26T03:59:21.654528+0000","last_active":"2024-03-26T04:00:07.638808+0000","last_peered":"2024-03-26T04:00:07.638808+0000","last_clean":"2024-03-26T04:00:07.638808+0000","last_became_active":"2024-03-26T03:57:49.367895+0000","last_became_peered":"2024-03-26T03:57:49.367895+0000","last_unstale":"2024-03-26T04:00:07.638808+0000","last_undegraded":"2024-03-26T04:00:07.638808+0000","last_fullsized":"2024-03-26T04:00:07.638808+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:21.654428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.654428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:23.596896+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":708,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636388+0000","last_change":"2024-03-26T03:58:39.055551+0000","last_active":"2024-03-26T04:00:07.636388+0000","last_peered":"2024-03-26T04:00:07.636388+0000","last_clean":"2024-03-26T04:00:07.636388+0000","last_became_active":"2024-03-26T03:50:48.723236+0000","last_became_peered":"2024-03-26T03:50:48.723236+0000","last_unstale":"2024-03-26T04:00:07.636388+0000","last_undegraded":"2024-03-26T04:00:07.636388+0000","last_fullsized":"2024-03-26T04:00:07.636388+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T03:58:39.055403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T03:58:39.055403+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638321+0000","last_change":"2024-03-26T03:59:29.150185+0000","last_active":"2024-03-26T04:00:07.638321+0000","last_peered":"2024-03-26T04:00:07.638321+0000","last_clean":"2024-03-26T04:00:07.638321+0000","last_became_active":"2024-03-26T03:57:49.369463+0000","last_became_peered":"2024-03-26T03:57:49.369463+0000","last_unstale":"2024-03-26T04:00:07.638321+0000","last_undegraded":"2024-03-26T04:00:07.638321+0000","last_fullsized":"2024-03-26T04:00:07.638321+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.150101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.150101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:38.172188+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":558,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630611+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T04:00:07.630611+0000","last_peered":"2024-03-26T04:00:07.630611+0000","last_clean":"2024-03-26T04:00:07.630611+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:00:07.630611+0000","last_undegraded":"2024-03-26T04:00:07.630611+0000","last_fullsized":"2024-03-26T04:00:07.630611+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235630+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":334,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.632302+0000","last_change":"2024-03-26T03:59:25.418140+0000","last_active":"2024-03-26T04:00:07.632302+0000","last_peered":"2024-03-26T04:00:07.632302+0000","last_clean":"2024-03-26T04:00:07.632302+0000","last_became_active":"2024-03-26T03:58:40.497499+0000","last_became_peered":"2024-03-26T03:58:40.497499+0000","last_unstale":"2024-03-26T04:00:07.632302+0000","last_undegraded":"2024-03-26T04:00:07.632302+0000","last_fullsized":"2024-03-26T04:00:07.632302+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.418009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.418009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:52.675582+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.b","version":"264'4366","reported_seq":4398,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:10.215013+0000","last_change":"2024-03-26T03:59:23.931889+0000","last_active":"2024-03-26T04:00:10.215013+0000","last_peered":"2024-03-26T04:00:10.215013+0000","last_clean":"2024-03-26T04:00:10.215013+0000","last_became_active":"2024-03-26T03:59:23.931557+0000","last_became_peered":"2024-03-26T03:59:23.931557+0000","last_unstale":"2024-03-26T04:00:10.215013+0000","last_undegraded":"2024-03-26T04:00:10.215013+0000","last_fullsized":"2024-03-26T04:00:10.215013+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:34.030948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":286130176,"num_objects":4366,"num_object_clones":0,"num_object_copies":13098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":279424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":443,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.675847+0000","last_change":"2024-03-26T03:58:40.462393+0000","last_active":"2024-03-26T04:00:07.675847+0000","last_peered":"2024-03-26T04:00:07.675847+0000","last_clean":"2024-03-26T04:00:07.675847+0000","last_became_active":"2024-03-26T03:57:49.379569+0000","last_became_peered":"2024-03-26T03:57:49.379569+0000","last_unstale":"2024-03-26T04:00:07.675847+0000","last_undegraded":"2024-03-26T04:00:07.675847+0000","last_fullsized":"2024-03-26T04:00:07.675847+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:40.462316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:40.462316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":465,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.640001+0000","last_change":"2024-03-26T03:57:49.364364+0000","last_active":"2024-03-26T04:00:07.640001+0000","last_peered":"2024-03-26T04:00:07.640001+0000","last_clean":"2024-03-26T04:00:07.640001+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T04:00:07.640001+0000","last_undegraded":"2024-03-26T04:00:07.640001+0000","last_fullsized":"2024-03-26T04:00:07.640001+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:10.117840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:55:10.117840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1141,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.674245+0000","last_change":"2024-03-26T03:59:09.978226+0000","last_active":"2024-03-26T04:00:07.674245+0000","last_peered":"2024-03-26T04:00:07.674245+0000","last_clean":"2024-03-26T04:00:07.674245+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:00:07.674245+0000","last_undegraded":"2024-03-26T04:00:07.674245+0000","last_fullsized":"2024-03-26T04:00:07.674245+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T03:59:09.978111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T03:59:09.978111+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:15.087807+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":677,"num_read_kb":672,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":468,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671374+0000","last_change":"2024-03-26T03:55:20.120400+0000","last_active":"2024-03-26T04:00:07.671374+0000","last_peered":"2024-03-26T04:00:07.671374+0000","last_clean":"2024-03-26T04:00:07.671374+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T04:00:07.671374+0000","last_undegraded":"2024-03-26T04:00:07.671374+0000","last_fullsized":"2024-03-26T04:00:07.671374+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-03-26T03:55:20.120302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:20.120302+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":381,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638912+0000","last_change":"2024-03-26T03:59:32.538750+0000","last_active":"2024-03-26T04:00:07.638912+0000","last_peered":"2024-03-26T04:00:07.638912+0000","last_clean":"2024-03-26T04:00:07.638912+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T04:00:07.638912+0000","last_undegraded":"2024-03-26T04:00:07.638912+0000","last_fullsized":"2024-03-26T04:00:07.638912+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:32.538628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:32.538628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:49.022689+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"264'4034","reported_seq":4066,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.290379+0000","last_change":"2024-03-26T03:59:23.930380+0000","last_active":"2024-03-26T04:00:09.290379+0000","last_peered":"2024-03-26T04:00:09.290379+0000","last_clean":"2024-03-26T04:00:09.290379+0000","last_became_active":"2024-03-26T03:59:23.930042+0000","last_became_peered":"2024-03-26T03:59:23.930042+0000","last_unstale":"2024-03-26T04:00:09.290379+0000","last_undegraded":"2024-03-26T04:00:09.290379+0000","last_fullsized":"2024-03-26T04:00:09.290379+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4034,"log_dups_size":0,"ondisk_log_size":4034,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:38.160641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":264372224,"num_objects":4034,"num_object_clones":0,"num_object_copies":12102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4034,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4034,"num_write_kb":258176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":436,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670344+0000","last_change":"2024-03-26T03:59:24.317142+0000","last_active":"2024-03-26T04:00:07.670344+0000","last_peered":"2024-03-26T04:00:07.670344+0000","last_clean":"2024-03-26T04:00:07.670344+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T04:00:07.670344+0000","last_undegraded":"2024-03-26T04:00:07.670344+0000","last_fullsized":"2024-03-26T04:00:07.670344+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:24.317061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:24.317061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.745057+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":451,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.676093+0000","last_change":"2024-03-26T03:58:41.426711+0000","last_active":"2024-03-26T04:00:07.676093+0000","last_peered":"2024-03-26T04:00:07.676093+0000","last_clean":"2024-03-26T04:00:07.676093+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T04:00:07.676093+0000","last_undegraded":"2024-03-26T04:00:07.676093+0000","last_fullsized":"2024-03-26T04:00:07.676093+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:41.426593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:41.426593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"241'35","reported_seq":512,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.672034+0000","last_change":"2024-03-26T03:59:25.757129+0000","last_active":"2024-03-26T04:00:07.672034+0000","last_peered":"2024-03-26T04:00:07.672034+0000","last_clean":"2024-03-26T04:00:07.672034+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T04:00:07.672034+0000","last_undegraded":"2024-03-26T04:00:07.672034+0000","last_fullsized":"2024-03-26T04:00:07.672034+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'35","last_scrub_stamp":"2024-03-26T03:59:25.756981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.756981+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:26.517629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":383,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639951+0000","last_change":"2024-03-26T03:59:27.734291+0000","last_active":"2024-03-26T04:00:07.639951+0000","last_peered":"2024-03-26T04:00:07.639951+0000","last_clean":"2024-03-26T04:00:07.639951+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T04:00:07.639951+0000","last_undegraded":"2024-03-26T04:00:07.639951+0000","last_fullsized":"2024-03-26T04:00:07.639951+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:51.534460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.f","version":"264'4148","reported_seq":4180,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:08.306301+0000","last_change":"2024-03-26T03:59:23.523599+0000","last_active":"2024-03-26T04:00:08.306301+0000","last_peered":"2024-03-26T04:00:08.306301+0000","last_clean":"2024-03-26T04:00:08.306301+0000","last_became_active":"2024-03-26T03:59:23.523293+0000","last_became_peered":"2024-03-26T03:59:23.523293+0000","last_unstale":"2024-03-26T04:00:08.306301+0000","last_undegraded":"2024-03-26T04:00:08.306301+0000","last_fullsized":"2024-03-26T04:00:08.306301+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4148,"log_dups_size":0,"ondisk_log_size":4148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.604819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":271843328,"num_objects":4148,"num_object_clones":0,"num_object_copies":12444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4148,"num_write_kb":265472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":445,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639727+0000","last_change":"2024-03-26T03:59:13.839565+0000","last_active":"2024-03-26T04:00:07.639727+0000","last_peered":"2024-03-26T04:00:07.639727+0000","last_clean":"2024-03-26T04:00:07.639727+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T04:00:07.639727+0000","last_undegraded":"2024-03-26T04:00:07.639727+0000","last_fullsized":"2024-03-26T04:00:07.639727+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:13.839415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.839415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:28.685298+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":405,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638854+0000","last_change":"2024-03-26T03:57:49.368358+0000","last_active":"2024-03-26T04:00:07.638854+0000","last_peered":"2024-03-26T04:00:07.638854+0000","last_clean":"2024-03-26T04:00:07.638854+0000","last_became_active":"2024-03-26T03:57:49.367990+0000","last_became_peered":"2024-03-26T03:57:49.367990+0000","last_unstale":"2024-03-26T04:00:07.638854+0000","last_undegraded":"2024-03-26T04:00:07.638854+0000","last_fullsized":"2024-03-26T04:00:07.638854+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:14.805020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:14.805020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482155+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.677109+0000","last_change":"2024-03-26T03:59:15.299800+0000","last_active":"2024-03-26T04:00:07.677109+0000","last_peered":"2024-03-26T04:00:07.677109+0000","last_clean":"2024-03-26T04:00:07.677109+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T04:00:07.677109+0000","last_undegraded":"2024-03-26T04:00:07.677109+0000","last_fullsized":"2024-03-26T04:00:07.677109+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T03:59:15.299680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.299680+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:30.178450+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"215'13","reported_seq":615,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671826+0000","last_change":"2024-03-26T03:59:16.761537+0000","last_active":"2024-03-26T04:00:07.671826+0000","last_peered":"2024-03-26T04:00:07.671826+0000","last_clean":"2024-03-26T04:00:07.671826+0000","last_became_active":"2024-03-26T03:57:49.361026+0000","last_became_peered":"2024-03-26T03:57:49.361026+0000","last_unstale":"2024-03-26T04:00:07.671826+0000","last_undegraded":"2024-03-26T04:00:07.671826+0000","last_fullsized":"2024-03-26T04:00:07.671826+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'13","last_scrub_stamp":"2024-03-26T03:59:16.761422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.761422+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.563005+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"264'4344","reported_seq":4376,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.947264+0000","last_change":"2024-03-26T03:59:23.930568+0000","last_active":"2024-03-26T04:00:09.947264+0000","last_peered":"2024-03-26T04:00:09.947264+0000","last_clean":"2024-03-26T04:00:09.947264+0000","last_became_active":"2024-03-26T03:59:23.930275+0000","last_became_peered":"2024-03-26T03:59:23.930275+0000","last_unstale":"2024-03-26T04:00:09.947264+0000","last_undegraded":"2024-03-26T04:00:09.947264+0000","last_fullsized":"2024-03-26T04:00:09.947264+0000","mapping_epoch":253,"log_start":"258'100","ondisk_log_start":"258'100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4244,"log_dups_size":0,"ondisk_log_size":4244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:42.507204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284688384,"num_objects":4344,"num_object_clones":0,"num_object_copies":13032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4344,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4344,"num_write_kb":278016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":446,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.636846+0000","last_change":"2024-03-26T04:00:07.116295+0000","last_active":"2024-03-26T04:00:07.636846+0000","last_peered":"2024-03-26T04:00:07.636846+0000","last_clean":"2024-03-26T04:00:07.636846+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T04:00:07.636846+0000","last_undegraded":"2024-03-26T04:00:07.636846+0000","last_fullsized":"2024-03-26T04:00:07.636846+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:07.116198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:00:07.116198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:23.405005+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":445,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.671474+0000","last_change":"2024-03-26T03:59:14.764278+0000","last_active":"2024-03-26T04:00:07.671474+0000","last_peered":"2024-03-26T04:00:07.671474+0000","last_clean":"2024-03-26T04:00:07.671474+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:00:07.671474+0000","last_undegraded":"2024-03-26T04:00:07.671474+0000","last_fullsized":"2024-03-26T04:00:07.671474+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:59:14.764155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.764155+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:37.349897+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":519,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630041+0000","last_change":"2024-03-26T03:55:59.302271+0000","last_active":"2024-03-26T04:00:07.630041+0000","last_peered":"2024-03-26T04:00:07.630041+0000","last_clean":"2024-03-26T04:00:07.630041+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T04:00:07.630041+0000","last_undegraded":"2024-03-26T04:00:07.630041+0000","last_fullsized":"2024-03-26T04:00:07.630041+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'20","last_scrub_stamp":"2024-03-26T03:54:16.339480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:54:16.339480+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235501+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":142,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641186+0000","last_change":"2024-03-26T03:57:49.461921+0000","last_active":"2024-03-26T04:00:07.641186+0000","last_peered":"2024-03-26T04:00:07.641186+0000","last_clean":"2024-03-26T04:00:07.641186+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T04:00:07.641186+0000","last_undegraded":"2024-03-26T04:00:07.641186+0000","last_fullsized":"2024-03-26T04:00:07.641186+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T03:54:34.943428+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"264'4222","reported_seq":4254,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.278725+0000","last_change":"2024-03-26T03:59:23.525085+0000","last_active":"2024-03-26T04:00:09.278725+0000","last_peered":"2024-03-26T04:00:09.278725+0000","last_clean":"2024-03-26T04:00:09.278725+0000","last_became_active":"2024-03-26T03:59:23.524729+0000","last_became_peered":"2024-03-26T03:59:23.524729+0000","last_unstale":"2024-03-26T04:00:09.278725+0000","last_undegraded":"2024-03-26T04:00:09.278725+0000","last_fullsized":"2024-03-26T04:00:09.278725+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4222,"log_dups_size":0,"ondisk_log_size":4222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:35.776916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":276692992,"num_objects":4222,"num_object_clones":0,"num_object_copies":12666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4222,"num_write_kb":270208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":428,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629891+0000","last_change":"2024-03-26T03:57:49.360270+0000","last_active":"2024-03-26T04:00:07.629891+0000","last_peered":"2024-03-26T04:00:07.629891+0000","last_clean":"2024-03-26T04:00:07.629891+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T04:00:07.629891+0000","last_undegraded":"2024-03-26T04:00:07.629891+0000","last_fullsized":"2024-03-26T04:00:07.629891+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:17.321485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:17.321485+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235356+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641562+0000","last_change":"2024-03-26T03:59:12.184503+0000","last_active":"2024-03-26T04:00:07.641562+0000","last_peered":"2024-03-26T04:00:07.641562+0000","last_clean":"2024-03-26T04:00:07.641562+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T04:00:07.641562+0000","last_undegraded":"2024-03-26T04:00:07.641562+0000","last_fullsized":"2024-03-26T04:00:07.641562+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.184357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.184357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.537755+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":157,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641350+0000","last_change":"2024-03-26T03:59:21.245414+0000","last_active":"2024-03-26T04:00:07.641350+0000","last_peered":"2024-03-26T04:00:07.641350+0000","last_clean":"2024-03-26T04:00:07.641350+0000","last_became_active":"2024-03-26T03:57:49.398824+0000","last_became_peered":"2024-03-26T03:57:49.398824+0000","last_unstale":"2024-03-26T04:00:07.641350+0000","last_undegraded":"2024-03-26T04:00:07.641350+0000","last_fullsized":"2024-03-26T04:00:07.641350+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T03:59:21.245251+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.245251+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:31.393654+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":70,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":440,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629775+0000","last_change":"2024-03-26T03:57:49.360193+0000","last_active":"2024-03-26T04:00:07.629775+0000","last_peered":"2024-03-26T04:00:07.629775+0000","last_clean":"2024-03-26T04:00:07.629775+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T04:00:07.629775+0000","last_undegraded":"2024-03-26T04:00:07.629775+0000","last_fullsized":"2024-03-26T04:00:07.629775+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:22.211573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T03:55:22.211573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235518+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"264'4275","reported_seq":4306,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.962499+0000","last_change":"2024-03-26T03:59:23.932941+0000","last_active":"2024-03-26T04:00:09.962499+0000","last_peered":"2024-03-26T04:00:09.962499+0000","last_clean":"2024-03-26T04:00:09.962499+0000","last_became_active":"2024-03-26T03:59:23.932638+0000","last_became_peered":"2024-03-26T03:59:23.932638+0000","last_unstale":"2024-03-26T04:00:09.962499+0000","last_undegraded":"2024-03-26T04:00:09.962499+0000","last_fullsized":"2024-03-26T04:00:09.962499+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4275,"log_dups_size":0,"ondisk_log_size":4275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:41.461400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":280166400,"num_objects":4275,"num_object_clones":0,"num_object_copies":12825,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4275,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4275,"num_write_kb":273600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":359,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638724+0000","last_change":"2024-03-26T03:59:25.561119+0000","last_active":"2024-03-26T04:00:07.638724+0000","last_peered":"2024-03-26T04:00:07.638724+0000","last_clean":"2024-03-26T04:00:07.638724+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T04:00:07.638724+0000","last_undegraded":"2024-03-26T04:00:07.638724+0000","last_fullsized":"2024-03-26T04:00:07.638724+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.561017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.561017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:33.161594+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":585,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.206080+0000","last_change":"2024-03-26T03:55:59.295990+0000","last_active":"2024-03-26T04:00:09.206080+0000","last_peered":"2024-03-26T04:00:09.206080+0000","last_clean":"2024-03-26T04:00:09.206080+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T04:00:09.206080+0000","last_undegraded":"2024-03-26T04:00:09.206080+0000","last_fullsized":"2024-03-26T04:00:09.206080+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-03-26T03:54:33.386275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.386275+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":117,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.641460+0000","last_change":"2024-03-26T03:59:27.171191+0000","last_active":"2024-03-26T04:00:07.641460+0000","last_peered":"2024-03-26T04:00:07.641460+0000","last_clean":"2024-03-26T04:00:07.641460+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T04:00:07.641460+0000","last_undegraded":"2024-03-26T04:00:07.641460+0000","last_fullsized":"2024-03-26T04:00:07.641460+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.171113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.171113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:50.875823+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":525,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.670938+0000","last_change":"2024-03-26T03:58:42.470991+0000","last_active":"2024-03-26T04:00:07.670938+0000","last_peered":"2024-03-26T04:00:07.670938+0000","last_clean":"2024-03-26T04:00:07.670938+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T04:00:07.670938+0000","last_undegraded":"2024-03-26T04:00:07.670938+0000","last_fullsized":"2024-03-26T04:00:07.670938+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:58:42.470857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:58:42.470857+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"264'4044","reported_seq":4076,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.267858+0000","last_change":"2024-03-26T03:59:23.523826+0000","last_active":"2024-03-26T04:00:09.267858+0000","last_peered":"2024-03-26T04:00:09.267858+0000","last_clean":"2024-03-26T04:00:09.267858+0000","last_became_active":"2024-03-26T03:59:23.523509+0000","last_became_peered":"2024-03-26T03:59:23.523509+0000","last_unstale":"2024-03-26T04:00:09.267858+0000","last_undegraded":"2024-03-26T04:00:09.267858+0000","last_fullsized":"2024-03-26T04:00:09.267858+0000","mapping_epoch":253,"log_start":"0'0","ondisk_log_start":"0'0","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4044,"log_dups_size":0,"ondisk_log_size":4044,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:50.788562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265027584,"num_objects":4044,"num_object_clones":0,"num_object_copies":12132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4044,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4044,"num_write_kb":258816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.639081+0000","last_change":"2024-03-26T03:57:49.373101+0000","last_active":"2024-03-26T04:00:07.639081+0000","last_peered":"2024-03-26T04:00:07.639081+0000","last_clean":"2024-03-26T04:00:07.639081+0000","last_became_active":"2024-03-26T03:57:49.372744+0000","last_became_peered":"2024-03-26T03:57:49.372744+0000","last_unstale":"2024-03-26T04:00:07.639081+0000","last_undegraded":"2024-03-26T04:00:07.639081+0000","last_fullsized":"2024-03-26T04:00:07.639081+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:23.683370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:54:23.683370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":393,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638804+0000","last_change":"2024-03-26T03:55:59.302453+0000","last_active":"2024-03-26T04:00:07.638804+0000","last_peered":"2024-03-26T04:00:07.638804+0000","last_clean":"2024-03-26T04:00:07.638804+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T04:00:07.638804+0000","last_undegraded":"2024-03-26T04:00:07.638804+0000","last_fullsized":"2024-03-26T04:00:07.638804+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:24.803430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T03:54:24.803430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:08.482105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":436,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629875+0000","last_change":"2024-03-26T03:54:25.335352+0000","last_active":"2024-03-26T04:00:07.629875+0000","last_peered":"2024-03-26T04:00:07.629875+0000","last_clean":"2024-03-26T04:00:07.629875+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T04:00:07.629875+0000","last_undegraded":"2024-03-26T04:00:07.629875+0000","last_fullsized":"2024-03-26T04:00:07.629875+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:25.335267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T03:54:25.335267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235394+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":552,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630740+0000","last_change":"2024-03-26T03:59:14.317073+0000","last_active":"2024-03-26T04:00:07.630740+0000","last_peered":"2024-03-26T04:00:07.630740+0000","last_clean":"2024-03-26T04:00:07.630740+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T04:00:07.630740+0000","last_undegraded":"2024-03-26T04:00:07.630740+0000","last_fullsized":"2024-03-26T04:00:07.630740+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T03:59:14.316934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.316934+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:40.183304+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"264'4321","reported_seq":4351,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:10.372492+0000","last_change":"2024-03-26T03:59:23.931746+0000","last_active":"2024-03-26T04:00:10.372492+0000","last_peered":"2024-03-26T04:00:10.372492+0000","last_clean":"2024-03-26T04:00:10.372492+0000","last_became_active":"2024-03-26T03:59:23.931459+0000","last_became_peered":"2024-03-26T03:59:23.931459+0000","last_unstale":"2024-03-26T04:00:10.372492+0000","last_undegraded":"2024-03-26T04:00:10.372492+0000","last_fullsized":"2024-03-26T04:00:10.372492+0000","mapping_epoch":253,"log_start":"258'200","ondisk_log_start":"258'200","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4121,"log_dups_size":0,"ondisk_log_size":4121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:29.777517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":283181056,"num_objects":4321,"num_object_clones":0,"num_object_copies":12963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4321,"num_write_kb":276544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":460,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.631720+0000","last_change":"2024-03-26T03:58:31.314191+0000","last_active":"2024-03-26T04:00:07.631720+0000","last_peered":"2024-03-26T04:00:07.631720+0000","last_clean":"2024-03-26T04:00:07.631720+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T04:00:07.631720+0000","last_undegraded":"2024-03-26T04:00:07.631720+0000","last_fullsized":"2024-03-26T04:00:07.631720+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:31.314073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:58:31.314073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":434,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.676314+0000","last_change":"2024-03-26T03:59:19.316596+0000","last_active":"2024-03-26T04:00:07.676314+0000","last_peered":"2024-03-26T04:00:07.676314+0000","last_clean":"2024-03-26T04:00:07.676314+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:00:07.676314+0000","last_undegraded":"2024-03-26T04:00:07.676314+0000","last_fullsized":"2024-03-26T04:00:07.676314+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:19.316523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.316523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:44.794322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":395,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.629960+0000","last_change":"2024-03-26T03:59:27.313345+0000","last_active":"2024-03-26T04:00:07.629960+0000","last_peered":"2024-03-26T04:00:07.629960+0000","last_clean":"2024-03-26T04:00:07.629960+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T04:00:07.629960+0000","last_undegraded":"2024-03-26T04:00:07.629960+0000","last_fullsized":"2024-03-26T04:00:07.629960+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.313255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.313255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:46.779259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":489,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630647+0000","last_change":"2024-03-26T03:59:26.340091+0000","last_active":"2024-03-26T04:00:07.630647+0000","last_peered":"2024-03-26T04:00:07.630647+0000","last_clean":"2024-03-26T04:00:07.630647+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T04:00:07.630647+0000","last_undegraded":"2024-03-26T04:00:07.630647+0000","last_fullsized":"2024-03-26T04:00:07.630647+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:26.339621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.339621+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:46.853603+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"264'4400","reported_seq":4432,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:09.968961+0000","last_change":"2024-03-26T03:59:23.930540+0000","last_active":"2024-03-26T04:00:09.968961+0000","last_peered":"2024-03-26T04:00:09.968961+0000","last_clean":"2024-03-26T04:00:09.968961+0000","last_became_active":"2024-03-26T03:59:23.930238+0000","last_became_peered":"2024-03-26T03:59:23.930238+0000","last_unstale":"2024-03-26T04:00:09.968961+0000","last_undegraded":"2024-03-26T04:00:09.968961+0000","last_fullsized":"2024-03-26T04:00:09.968961+0000","mapping_epoch":253,"log_start":"257'100","ondisk_log_start":"257'100","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4300,"log_dups_size":0,"ondisk_log_size":4300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:27.352378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288358400,"num_objects":4400,"num_object_clones":0,"num_object_copies":13200,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4400,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4400,"num_write_kb":281600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":424,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.631155+0000","last_change":"2024-03-26T03:54:28.365105+0000","last_active":"2024-03-26T04:00:07.631155+0000","last_peered":"2024-03-26T04:00:07.631155+0000","last_clean":"2024-03-26T04:00:07.631155+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T04:00:07.631155+0000","last_undegraded":"2024-03-26T04:00:07.631155+0000","last_fullsized":"2024-03-26T04:00:07.631155+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T03:54:28.365009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":601,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.631880+0000","last_change":"2024-03-26T03:58:33.177842+0000","last_active":"2024-03-26T04:00:07.631880+0000","last_peered":"2024-03-26T04:00:07.631880+0000","last_clean":"2024-03-26T04:00:07.631880+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T04:00:07.631880+0000","last_undegraded":"2024-03-26T04:00:07.631880+0000","last_fullsized":"2024-03-26T04:00:07.631880+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T03:58:33.177720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:58:33.177720+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":109,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.638690+0000","last_change":"2024-03-26T03:57:49.360595+0000","last_active":"2024-03-26T04:00:07.638690+0000","last_peered":"2024-03-26T04:00:07.638690+0000","last_clean":"2024-03-26T04:00:07.638690+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T04:00:07.638690+0000","last_undegraded":"2024-03-26T04:00:07.638690+0000","last_fullsized":"2024-03-26T04:00:07.638690+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:54:33.815111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:54:33.815111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":516,"reported_epoch":264,"state":"active+clean","last_fresh":"2024-03-26T04:00:07.630482+0000","last_change":"2024-03-26T03:57:49.507032+0000","last_active":"2024-03-26T04:00:07.630482+0000","last_peered":"2024-03-26T04:00:07.630482+0000","last_clean":"2024-03-26T04:00:07.630482+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T04:00:07.630482+0000","last_undegraded":"2024-03-26T04:00:07.630482+0000","last_fullsized":"2024-03-26T04:00:07.630482+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-03-26T03:55:17.192116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:17.192116+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:09.235575+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4427153408,"num_objects":67553,"num_object_clones":0,"num_object_copies":202659,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67553,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67553,"num_write_kb":4323392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13297909760,"data_stored":13297909760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67153,"ondisk_log_size":67153,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":677,"num_read_kb":672,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1408,"num_read_kb":1133,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":4500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070605,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2722020,"kb_used_data":2406236,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":91023132,"statfs":{"total":95995035648,"available":93207687168,"internally_reserved":0,"allocated":2463985664,"data_stored":2463560337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":323353034},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575020,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2806652,"kb_used_data":2406196,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":90938500,"statfs":{"total":95995035648,"available":93121024000,"internally_reserved":0,"allocated":2463944704,"data_stored":2463493954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738545,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728628,"kb_used_data":1354540,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92016524,"statfs":{"total":95995035648,"available":94224920576,"internally_reserved":0,"allocated":1387048960,"data_stored":1386607371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411495,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497476,"kb_used_data":1083132,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92247676,"statfs":{"total":95995035648,"available":94461620224,"internally_reserved":0,"allocated":1109127168,"data_stored":1108731891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182561,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2530040,"kb_used_data":2165676,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":91215112,"statfs":{"total":95995035648,"available":93404274688,"internally_reserved":0,"allocated":2217652224,"data_stored":2217210015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953628,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1440808,"kb_used_data":1070876,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92304344,"statfs":{"total":95995035648,"available":94519648256,"internally_reserved":0,"allocated":1096577024,"data_stored":1096149469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113331,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2693448,"kb_used_data":2353088,"kb_used_omap":4,"kb_used_meta":340347,"kb_avail":91051704,"statfs":{"total":95995035648,"available":93236944896,"internally_reserved":0,"allocated":2409562112,"data_stored":2409156299,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":348516000},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1407582208,"data_stored":1407582208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2502950912,"data_stored":2502950912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1127350272,"data_stored":1127350272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2239430656,"data_stored":2239430656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1099694080,"data_stored":1099694080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2446065664,"data_stored":2446065664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2474835968,"data_stored":2474835968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:00:14.373 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-26T04:00:14.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-26T04:00:14.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:14.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:14.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:14.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:14.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:14.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:14.611 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:14.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:14.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:14.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:14.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:14.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:14.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:14.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:14.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[29446]: pgmap v514: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 580 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[37501]: pgmap v514: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 580 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T04:00:14.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:14 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T04:00:14.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:14.981 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:00:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:15.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:15.031 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:00:15.074 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:00:15.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 43 16 74582 74566 108.35 72.625 0.0124718 0.00922445 2024-03-26T04:00:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T04:00:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T04:00:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-mon[32965]: pgmap v514: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 580 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:00:15.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T04:00:15.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T04:00:15.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:15.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:15.123 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:00:15.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:15.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:15.163 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:00:15.203 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:00:15.246 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:00:15.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:15.288 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:00:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:15.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:00:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:00:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:00:15.292 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:00:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:00:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:00:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:00:15.293 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:00:15.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:15.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:15.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T04:00:15.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:15.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:15.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:15.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:15.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:15.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:15.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:15.872 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:15.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: osdmap e265: 8 total, 8 up, 7 in 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2725684965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T04:00:15.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: osdmap e265: 8 total, 8 up, 7 in 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2725684965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T04:00:15.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:15 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T04:00:15.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.000 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:16.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 44 16 75779 75763 107.588 74.8125 0.00990713 0.00929012 2024-03-26T04:00:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T04:00:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: osdmap e265: 8 total, 8 up, 7 in 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2725684965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T04:00:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:15 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T04:00:16.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:16.139 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:16.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:16.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:16.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:16.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:16.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:16.701 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:16.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:16.844 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:16.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:16.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:16.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:16.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:17.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 45 15 77103 77088 107.037 82.8125 0.0136249 0.00933783 2024-03-26T04:00:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-mon[32965]: pgmap v516: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-26T04:00:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T04:00:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T04:00:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T04:00:17.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T04:00:17.088 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:00:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=cleanup t=2024-03-26T04:00:16.967909423Z level=info msg="Completed cleanup jobs" duration=1.47932ms 2024-03-26T04:00:17.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:17.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:17.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:17.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[29446]: pgmap v516: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[37501]: pgmap v516: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T04:00:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:16 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T04:00:17.220 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:17.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:17.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:17.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:17.305 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:17.328 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:17.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:17.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:17.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:17.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:17.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:17.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.699 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:17.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:17.800 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:17.842 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:17.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:17.944 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:17.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:17.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: 4.14 scrub starts 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[29446]: 4.14 scrub ok 2024-03-26T04:00:18.011 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T04:00:18.012 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T04:00:18.012 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:18.012 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:18.012 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: 4.14 scrub starts 2024-03-26T04:00:18.012 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:17 smithi006 ceph-mon[37501]: 4.14 scrub ok 2024-03-26T04:00:18.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 46 16 78286 78270 106.316 73.875 0.00415458 0.0094004 2024-03-26T04:00:18.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T04:00:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T04:00:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:18.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: 4.14 scrub starts 2024-03-26T04:00:18.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:17 smithi086 ceph-mon[32965]: 4.14 scrub ok 2024-03-26T04:00:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:18.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:18.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:18.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:18.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:18.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:18.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:18.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:18.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:18.544 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:18.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:18.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:18.793 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:18.831 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:18.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: pgmap v517: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: osdmap e266: 8 total, 8 up, 7 in 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:00:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: pgmap v517: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: osdmap e266: 8 total, 8 up, 7 in 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T04:00:18.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:18 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T04:00:18.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:18.973 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:18.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:19.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:19.081 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 47 16 79586 79570 105.783 81.25 0.00822774 0.00944795 2024-03-26T04:00:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T04:00:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T04:00:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: pgmap v517: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: osdmap e266: 8 total, 8 up, 7 in 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T04:00:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:18 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T04:00:19.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:19.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:00:19.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:19.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:19.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:19.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:19.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:19.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:19.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:19.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:19.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:19.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:19.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:19.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:20.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:20.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:20.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 48 16 80862 80846 105.24 79.75 0.00708177 0.00949702 2024-03-26T04:00:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T04:00:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T04:00:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T04:00:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T04:00:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:00:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:00:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: osdmap e267: 8 total, 8 up, 7 in 2024-03-26T04:00:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 6.6 scrub starts 2024-03-26T04:00:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:19 smithi086 ceph-mon[32965]: 6.6 scrub ok 2024-03-26T04:00:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T04:00:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T04:00:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T04:00:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T04:00:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: osdmap e267: 8 total, 8 up, 7 in 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 6.6 scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[29446]: 6.6 scrub ok 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: osdmap e267: 8 total, 8 up, 7 in 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 6.6 scrub starts 2024-03-26T04:00:20.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:19 smithi006 ceph-mon[37501]: 6.6 scrub ok 2024-03-26T04:00:20.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:20.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:20.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:20.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:20.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:20.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:20.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:20.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:20.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:20.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:20.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:20.734 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T04:00:20.748 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:20.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:20.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:20.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:20.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:20.991 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:21.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:21.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 5.14 scrub starts 2024-03-26T04:00:21.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[29446]: 5.14 scrub ok 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 5.14 scrub starts 2024-03-26T04:00:21.086 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:20 smithi006 ceph-mon[37501]: 5.14 scrub ok 2024-03-26T04:00:21.086 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 49 16 82156 82140 104.743 80.875 0.0135984 0.00954164 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: pgmap v520: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T04:00:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T04:00:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 5.14 scrub starts 2024-03-26T04:00:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:20 smithi086 ceph-mon[32965]: 5.14 scrub ok 2024-03-26T04:00:21.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:21.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:21.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:21.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:21.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:21.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.467 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:21.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:21.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:21.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:21.700 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:21.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T04:00:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T04:00:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T04:00:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:00:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:00:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T04:00:21.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:21 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T04:00:21.902 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:21.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:22.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:22.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:22.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 50 16 83407 83391 104.212 78.1875 0.0161163 0.00959072 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T04:00:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:21 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T04:00:22.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:22.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.324 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:22.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:22.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.492 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:22.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:22.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:22.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:22.779 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:22.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:22.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:22.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:22.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:23.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: pgmap v521: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:23.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T04:00:23.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: pgmap v521: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T04:00:23.015 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T04:00:23.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T04:00:23.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T04:00:23.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T04:00:23.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:22 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T04:00:23.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:23.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:23.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 51 16 84619 84603 103.654 75.75 0.00509661 0.0096427 2024-03-26T04:00:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: pgmap v521: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 762 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T04:00:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T04:00:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T04:00:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T04:00:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T04:00:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:22 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T04:00:23.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:23.194 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:23.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:23.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:23.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:23.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:23.445 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:23.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:23.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:23.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:23.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:23.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:23.904 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T04:00:23.904 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T04:00:23.904 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:23.904 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T04:00:23.905 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:23 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T04:00:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:23.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:23.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:23.997 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:24.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:24.085 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 52 16 85636 85620 102.883 63.5625 0.0335846 0.00971428 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T04:00:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:23 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T04:00:24.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.090 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:24.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:24.114 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1969 5.3 5:c609908c:::notify.5:head [watch ping cookie 94232629225728 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e152)", 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136387+0000", 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 354.97318280000002, 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.30655227899999998, 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.115 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1969 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136387+0000", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136387+0000", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136389+0000", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2550000000000001e-06 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136394+0000", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4750000000000001e-06 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136408+0000", 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4248e-05 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136421+0000", 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2261e-05 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136554+0000", 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013327 2024-03-26T04:00:24.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136642+0000", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8004999999999995e-05 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401102+0000", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.26446012499999999 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401407+0000", 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030506799999999997 2024-03-26T04:00:24.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442774+0000", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041367379000000003 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442858+0000", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4009000000000002e-05 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442940+0000", 2024-03-26T04:00:24.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1184000000000001e-05 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:1979 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94232629225728 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e153)", 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:54:29.136815+0000", 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 354.97275508500002, 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.32657255899999998, 2024-03-26T04:00:24.121 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 1979 2024-03-26T04:00:24.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136815+0000", 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136815+0000", 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136816+0000", 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7899999999999997e-07 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136820+0000", 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3789999999999999e-06 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136829+0000", 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7630000000000006e-06 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136836+0000", 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3540000000000002e-06 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136897+0000", 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0992999999999997e-05 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.136910+0000", 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2948999999999999e-05 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401439+0000", 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.26452896599999998 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.401489+0000", 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9790000000000003e-05 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.442966+0000", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041476987 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.443039+0000", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2958000000000002e-05 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:54:29.463388+0000", 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020348641000000001 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.127 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215917 8.c 8:328b5576:::benchmark_data_smithi006_103169_object85986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 155.633672698, 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9868534169999998, 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:24.128 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215917 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:00:24.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3900000000000003e-07 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475904+0000", 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.344e-06 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475915+0000", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1496e-05 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475935+0000", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9800999999999999e-05 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476186+0000", 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025108200000000002 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476303+0000", 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117278 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476416+0000", 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011205999999999999 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.500056+0000", 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023640960999999999 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.500146+0000", 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9951999999999995e-05 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.516444+0000", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016297282999999999 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.516463+0000", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9071000000000001e-05 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337701+0000", 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82123782300000003 2024-03-26T04:00:24.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337727+0000", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6482000000000002e-05 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.462673+0000", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1249464010000001 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.462751+0000", 2024-03-26T04:00:24.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7443999999999999e-05 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215934 8.9 8.4e2021d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.534329+0000", 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 155.57524084100001, 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.804382771, 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.136 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215934 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534329+0000", 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534329+0000", 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534331+0000", 2024-03-26T04:00:24.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5239999999999999e-06 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534338+0000", 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7440000000000001e-06 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534350+0000", 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2514000000000001e-05 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534359+0000", 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8349999999999993e-06 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534612+0000", 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025262899999999998 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.534622+0000", 2024-03-26T04:00:24.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0861e-05 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338700+0000", 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80407781099999998 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338712+0000", 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1853e-05 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215940 8.c 8:3db160f6:::benchmark_data_smithi006_103169_object86009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 155.567508578, 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9264022000000001, 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215940 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.144 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.145 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542063+0000", 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6509999999999999e-06 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542070+0000", 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.737e-06 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542086+0000", 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6189999999999999e-05 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542096+0000", 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5009999999999993e-06 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542168+0000", 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1810999999999994e-05 2024-03-26T04:00:24.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542179+0000", 2024-03-26T04:00:24.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1406000000000001e-05 2024-03-26T04:00:24.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337766+0000", 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79558757499999999 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337776+0000", 2024-03-26T04:00:24.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7059999999999999e-06 2024-03-26T04:00:24.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463110+0000", 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.125333892 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463217+0000", 2024-03-26T04:00:24.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010647899999999999 2024-03-26T04:00:24.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T04:00:24.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463319+0000", 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102587 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.467958+0000", 2024-03-26T04:00:24.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046384590000000002 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468011+0000", 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3446000000000003e-05 2024-03-26T04:00:24.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468172+0000", 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016070799999999999 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.156 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468290+0000", 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118597 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468349+0000", 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8641000000000003e-05 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.157 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468464+0000", 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114814 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.158 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215951 8.9 8.176579d9 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.563757+0000", 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 155.545813383, 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.77499025600000004, 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.159 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215951 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563757+0000", 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563757+0000", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563758+0000", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.71e-07 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563762+0000", 2024-03-26T04:00:24.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5759999999999999e-06 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563771+0000", 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8950000000000004e-06 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563777+0000", 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6160000000000001e-06 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563849+0000", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2342999999999995e-05 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.563862+0000", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2883e-05 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338737+0000", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.77487521199999998 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338747+0000", 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8600000000000005e-06 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215954 8.c 8:39dff541:::benchmark_data_smithi006_103169_object86023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 155.53735756899999, 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.898915927, 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.164 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215954 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572214+0000", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1519999999999999e-06 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572219+0000", 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1200000000000001e-06 2024-03-26T04:00:24.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572230+0000", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0709e-05 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572236+0000", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9800000000000003e-06 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572296+0000", 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9914999999999998e-05 2024-03-26T04:00:24.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572308+0000", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2189e-05 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338781+0000", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.76647347099999996 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338791+0000", 2024-03-26T04:00:24.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6840000000000008e-06 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.465697+0000", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.126906169 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.465834+0000", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000136608 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.466026+0000", 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019242099999999999 2024-03-26T04:00:24.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468233+0000", 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022071669999999999 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468277+0000", 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3503e-05 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468472+0000", 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019535200000000001 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471025+0000", 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025533539999999999 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471063+0000", 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7477e-05 2024-03-26T04:00:24.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471129+0000", 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5655999999999996e-05 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:2388 5.3 5:c609908c:::notify.5:head [watch ping cookie 94232629225728 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e212)", 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.177166+0000", 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 154.93240457799999, 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22496470599999999, 2024-03-26T04:00:24.172 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 2388 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177166+0000", 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177166+0000", 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177168+0000", 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0559999999999999e-06 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177177+0000", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3300000000000005e-06 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177202+0000", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4579999999999998e-05 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177212+0000", 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0475000000000001e-05 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177348+0000", 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013599599999999999 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177366+0000", 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7949999999999999e-05 2024-03-26T04:00:24.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.363124+0000", 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18575846500000001 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.363450+0000", 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032554099999999999 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.401970+0000", 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.038519682 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.402062+0000", 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1809000000000001e-05 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.402130+0000", 2024-03-26T04:00:24.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8821999999999999e-05 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.24349.0:2398 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 94232629225728 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.177720+0000", 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 154.931849911, 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.22556026600000001, 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.24349", 2024-03-26T04:00:24.179 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2414067140", 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 2398 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177720+0000", 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177720+0000", 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177721+0000", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2600000000000001e-07 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177726+0000", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8919999999999999e-06 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177741+0000", 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4515000000000001e-05 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177749+0000", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5520000000000005e-06 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177808+0000", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9281999999999999e-05 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.177818+0000", 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8779999999999993e-06 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.363559+0000", 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18574053700000001 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.363611+0000", 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2411999999999999e-05 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.402156+0000", 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.038544952 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.402218+0000", 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1526999999999996e-05 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.403281+0000", 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0010628930000000001 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222643 8.c 8:3896b36c:::benchmark_data_smithi006_103169_object92712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.73350256399999, 2024-03-26T04:00:24.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0007649670000001, 2024-03-26T04:00:24.186 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.186 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.186 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.186 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.186 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222643 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376069+0000", 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.703e-06 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.189 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376077+0000", 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0260000000000007e-06 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376095+0000", 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7682000000000001e-05 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376103+0000", 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7430000000000002e-06 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376154+0000", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1616000000000002e-05 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376161+0000", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.075e-06 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.373847+0000", 2024-03-26T04:00:24.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99768531100000002 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.373971+0000", 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012385099999999999 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.374098+0000", 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012708399999999999 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376704+0000", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026066800000000001 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376717+0000", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2388999999999999e-05 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376761+0000", 2024-03-26T04:00:24.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3745999999999998e-05 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376793+0000", 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2396e-05 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376833+0000", 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9665000000000001e-05 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222659 8.c 8:3d3c1162:::benchmark_data_smithi006_103169_object92728:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.510563+0000", 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.599007597, 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87378867999999998, 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.196 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222659 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510563+0000", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510563+0000", 2024-03-26T04:00:24.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510564+0000", 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6169999999999999e-06 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510571+0000", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8449999999999997e-06 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510586+0000", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.525e-05 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510596+0000", 2024-03-26T04:00:24.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3689999999999996e-06 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510650+0000", 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4060000000000001e-05 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.510658+0000", 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7789999999999996e-06 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376884+0000", 2024-03-26T04:00:24.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.86622657000000003 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376990+0000", 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105596 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.377120+0000", 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013003099999999999 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.203 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:24.206 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384229+0000", 2024-03-26T04:00:24.206 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0071093659999999998 2024-03-26T04:00:24.206 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384242+0000", 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2778e-05 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.207 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384280+0000", 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8374000000000002e-05 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.208 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384311+0000", 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0836000000000001e-05 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384351+0000", 2024-03-26T04:00:24.209 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0209000000000001e-05 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.210 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222676 8.c 8:3d314ef6:::benchmark_data_smithi006_103169_object92745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.211 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.571217+0000", 2024-03-26T04:00:24.211 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.538353132, 2024-03-26T04:00:24.211 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81333487400000004, 2024-03-26T04:00:24.211 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.212 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222676 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571217+0000", 2024-03-26T04:00:24.213 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571217+0000", 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.214 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571219+0000", 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.57e-06 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.215 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571224+0000", 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7559999999999996e-06 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.216 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571235+0000", 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0446e-05 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571243+0000", 2024-03-26T04:00:24.217 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7780000000000007e-06 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571296+0000", 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3643999999999997e-05 2024-03-26T04:00:24.218 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.571305+0000", 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9449999999999999e-06 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.219 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.379122+0000", 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80781717399999997 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.379225+0000", 2024-03-26T04:00:24.220 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102192 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.379325+0000", 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9940999999999997e-05 2024-03-26T04:00:24.221 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384247+0000", 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049227910000000001 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.222 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384424+0000", 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017621999999999999 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.223 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384464+0000", 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0343000000000003e-05 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384514+0000", 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0575000000000003e-05 2024-03-26T04:00:24.224 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384552+0000", 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7499000000000001e-05 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.225 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222681 8.c 8:3bfa25ed:::benchmark_data_smithi006_103169_object92750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.583177+0000", 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.526393609, 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80783078500000005, 2024-03-26T04:00:24.226 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222681 2024-03-26T04:00:24.227 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583177+0000", 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.228 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583177+0000", 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.229 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583178+0000", 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.482e-06 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583185+0000", 2024-03-26T04:00:24.230 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5169999999999996e-06 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583198+0000", 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3162999999999999e-05 2024-03-26T04:00:24.231 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583206+0000", 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9170000000000006e-06 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.232 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583254+0000", 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8081000000000003e-05 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.583261+0000", 2024-03-26T04:00:24.233 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1550000000000001e-06 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.381841+0000", 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79858008400000002 2024-03-26T04:00:24.234 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.381919+0000", 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8447000000000003e-05 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.235 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.382036+0000", 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011644100000000001 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.236 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384617+0000", 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025811760000000001 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384668+0000", 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0550999999999998e-05 2024-03-26T04:00:24.237 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.390904+0000", 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0062362959999999997 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.238 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.390954+0000", 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9906e-05 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.239 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391007+0000", 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3569000000000002e-05 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.240 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222713 8.c 8:31e966d7:::benchmark_data_smithi006_103169_object92782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.757285+0000", 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.35228488300001, 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.63385585799999999, 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.241 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.242 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.242 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222713 2024-03-26T04:00:24.242 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757285+0000", 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.243 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757285+0000", 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.244 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757286+0000", 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.85e-07 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.245 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757290+0000", 2024-03-26T04:00:24.246 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0319999999999997e-06 2024-03-26T04:00:24.246 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.246 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.246 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.246 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757299+0000", 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2979999999999997e-06 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757305+0000", 2024-03-26T04:00:24.247 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3589999999999999e-06 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757355+0000", 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9710999999999999e-05 2024-03-26T04:00:24.248 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.757364+0000", 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3510000000000008e-06 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.249 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384705+0000", 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.62734156799999996 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.250 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384811+0000", 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010525499999999999 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.384922+0000", 2024-03-26T04:00:24.251 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110853 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.390886+0000", 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059643730000000002 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.252 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391049+0000", 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016298199999999999 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.253 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391060+0000", 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.133e-05 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391098+0000", 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7873999999999997e-05 2024-03-26T04:00:24.254 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391141+0000", 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3087000000000003e-05 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222736 8.c 8:3defffc6:::benchmark_data_smithi006_103169_object92805:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.949094+0000", 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.16047611499999, 2024-03-26T04:00:24.255 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.44724022499999999, 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222736 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949094+0000", 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.256 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949094+0000", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949096+0000", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5409999999999999e-06 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949102+0000", 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.511e-06 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.257 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949118+0000", 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6256000000000002e-05 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949127+0000", 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3699999999999995e-06 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.258 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949180+0000", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3264999999999999e-05 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.949188+0000", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1559999999999995e-06 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.388391+0000", 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.43920245800000002 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.259 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.388514+0000", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012370600000000001 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.388639+0000", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012448 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391151+0000", 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025122600000000001 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.260 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391188+0000", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6621000000000001e-05 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396087+0000", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048988629999999998 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396248+0000", 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016164700000000001 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.261 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396334+0000", 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6091e-05 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222757 8.c 8:362c2025:::benchmark_data_smithi006_103169_object92826:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:14.037222+0000", 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.07234789099999, 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.36436154999999998, 2024-03-26T04:00:24.262 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222757 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037222+0000", 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.263 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037222+0000", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037224+0000", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.621e-06 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037230+0000", 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4869999999999999e-06 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.264 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037245+0000", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4116e-05 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037254+0000", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7119999999999995e-06 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037327+0000", 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2887999999999997e-05 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.265 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.037337+0000", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9140000000000003e-06 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391217+0000", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.35387968800000003 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391319+0000", 2024-03-26T04:00:24.266 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102542 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.391420+0000", 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100295 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401324+0000", 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0099044939999999998 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.267 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401458+0000", 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013432099999999999 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401475+0000", 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6376999999999999e-05 2024-03-26T04:00:24.268 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401528+0000", 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3708999999999997e-05 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.269 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.270 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401584+0000", 2024-03-26T04:00:24.270 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5386000000000001e-05 2024-03-26T04:00:24.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.270 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.270 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222758 8.c 8:3cbeb5d6:::benchmark_data_smithi006_103169_object92827:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:14.039413+0000", 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.07015690700001, 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.36231782200000001, 2024-03-26T04:00:24.271 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222758 2024-03-26T04:00:24.272 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039413+0000", 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.273 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039413+0000", 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039415+0000", 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8640000000000001e-06 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.274 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039421+0000", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9800000000000003e-06 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039435+0000", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3889e-05 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039444+0000", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0170000000000002e-06 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039512+0000", 2024-03-26T04:00:24.275 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7593000000000001e-05 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.039523+0000", 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1041e-05 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.393496+0000", 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.35397335099999999 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.276 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.393625+0000", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012848699999999999 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.393795+0000", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000170834 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401374+0000", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075791369999999997 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.277 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401597+0000", 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022207699999999999 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401654+0000", 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6991999999999999e-05 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401704+0000", 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0519999999999997e-05 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.278 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401731+0000", 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7039999999999999e-05 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222767 8.c 8:339d85a6:::benchmark_data_smithi006_103169_object92836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:14.092217+0000", 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 130.01735332300001, 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.31744159700000002, 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.279 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222767 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092217+0000", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092217+0000", 2024-03-26T04:00:24.280 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092218+0000", 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.093e-06 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092223+0000", 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1340000000000003e-06 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.281 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092233+0000", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0006e-05 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092239+0000", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3470000000000004e-06 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092293+0000", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3795000000000002e-05 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.282 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.092301+0000", 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4839999999999998e-06 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396362+0000", 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.30406147100000003 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396449+0000", 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7239999999999998e-05 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.283 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.396533+0000", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3961000000000005e-05 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401603+0000", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050692510000000003 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401779+0000", 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017653100000000001 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.284 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409551+0000", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0077713849999999996 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409610+0000", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9063e-05 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409658+0000", 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8835999999999998e-05 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.285 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222777 8.c 8:3a069a8e:::benchmark_data_smithi006_103169_object92846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:14.127081+0000", 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 129.982489638, 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.28270132399999998, 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222777 2024-03-26T04:00:24.286 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127081+0000", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127081+0000", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127082+0000", 2024-03-26T04:00:24.287 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5099999999999999e-06 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127087+0000", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4160000000000003e-06 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127098+0000", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.082e-05 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127105+0000", 2024-03-26T04:00:24.288 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1940000000000003e-06 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127152+0000", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7398000000000003e-05 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.127158+0000", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.302e-06 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.398972+0000", 2024-03-26T04:00:24.289 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.27181338500000002 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.399077+0000", 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105401 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.399177+0000", 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9769e-05 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.290 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401816+0000", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026391750000000001 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409670+0000", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.007854527 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409706+0000", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5778999999999997e-05 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.291 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409748+0000", 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1279000000000002e-05 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409782+0000", 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4369000000000002e-05 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222800 8.c 8:33852300:::benchmark_data_smithi006_103169_object92869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:14.200786+0000", 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 129.908784027, 2024-03-26T04:00:24.292 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.209141251, 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222800 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200786+0000", 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.293 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200786+0000", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200788+0000", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7990000000000001e-06 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200794+0000", 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3069999999999999e-06 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.294 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200808+0000", 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3468999999999999e-05 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200816+0000", 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5550000000000003e-06 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.295 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200875+0000", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8226000000000001e-05 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.200885+0000", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0573e-05 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401833+0000", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20094832200000001 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:24.296 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.401922+0000", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8912000000000005e-05 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.402037+0000", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011499 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409790+0000", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0077527159999999998 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409821+0000", 2024-03-26T04:00:24.297 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0769000000000003e-05 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409854+0000", 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3361000000000001e-05 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409894+0000", 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0123999999999997e-05 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:24.298 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.409927+0000", 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3127999999999999e-05 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:24.299 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:00:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.301 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:24.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:24.388 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 1 pg_num to 11 2024-03-26T04:00:24.388 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:24.392 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:00:24.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:24.453 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:24.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:24.593 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:24.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:24.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:24.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:24.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:24.864 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:24.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:24.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:24.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:24.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:25.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:25.042 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T04:00:25.042 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: osdmap e268: 8 total, 8 up, 7 in 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/681690727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2233675200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T04:00:25.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[37501]: osdmap e269: 8 total, 8 up, 7 in 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-26T04:00:25.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: osdmap e268: 8 total, 8 up, 7 in 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/681690727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2233675200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:25.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:24 smithi006 ceph-mon[29446]: osdmap e269: 8 total, 8 up, 7 in 2024-03-26T04:00:25.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 53 15 86826 86811 102.346 74.4375 0.00980725 0.00976597 2024-03-26T04:00:25.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:25.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: pgmap v522: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 62 MiB/s wr, 985 op/s 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: osdmap e268: 8 total, 8 up, 7 in 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/681690727' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2233675200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:00:25.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T04:00:25.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-03-26T04:00:25.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-26T04:00:25.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:25.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:24 smithi086 ceph-mon[32965]: osdmap e269: 8 total, 8 up, 7 in 2024-03-26T04:00:25.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:25.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:25.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:25.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:25.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:25.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:25.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:25.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:25.535 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:25.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:25.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:25.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:25.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:25.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:25.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:25.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T04:00:25.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: osdmap e270: 8 total, 8 up, 7 in 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: osdmap e270: 8 total, 8 up, 7 in 2024-03-26T04:00:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:25.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:25.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:25.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:25.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:26.080 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:26.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 54 16 88093 88077 101.916 79.125 0.00818989 0.0098073 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: osdmap e270: 8 total, 8 up, 7 in 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:26.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.120 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:26.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:26.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.245 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:26.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:26.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:26.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:26.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:26.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:26.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:26.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:26.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:26.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:26.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:26.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:26.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:26.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:26.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:27.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:27.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:27.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 55 16 89302 89286 101.437 75.5625 0.0075824 0.00985402 2024-03-26T04:00:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 5.1b deep-scrub starts 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 5.1b deep-scrub ok 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: pgmap v526: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T04:00:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:26 smithi086 ceph-mon[32965]: osdmap e271: 8 total, 8 up, 7 in 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 5.1b deep-scrub starts 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 5.1b deep-scrub ok 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: pgmap v526: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T04:00:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[37501]: osdmap e271: 8 total, 8 up, 7 in 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 5.1b deep-scrub starts 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 5.1b deep-scrub ok 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: pgmap v526: 148 pgs: 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 682 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-26T04:00:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:26 smithi006 ceph-mon[29446]: osdmap e271: 8 total, 8 up, 7 in 2024-03-26T04:00:27.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:27.196 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:27.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T04:00:27.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:27.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:27.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:27.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:27.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:27.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:27.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:27.767 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:27.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:27.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:27.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:27.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:27.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:28.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:28.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:28.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: osdmap e272: 8 total, 8 up, 7 in 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T04:00:28.047 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: osdmap e272: 8 total, 8 up, 7 in 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T04:00:28.048 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:27 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.082 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 56 16 90540 90524 101.007 77.375 0.00824356 0.00989516 2024-03-26T04:00:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T04:00:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T04:00:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T04:00:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: osdmap e272: 8 total, 8 up, 7 in 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T04:00:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:27 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-26T04:00:28.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:28.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:28.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:28.317 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:28.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:28.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:28.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:28.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:28.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:28.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:28.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:28.750 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:28.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T04:00:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: pgmap v528: 158 pgs: 10 unknown, 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[29446]: osdmap e273: 8 total, 8 up, 7 in 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T04:00:28.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T04:00:28.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: pgmap v528: 158 pgs: 10 unknown, 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T04:00:28.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:00:28.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:00:28.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T04:00:28.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:28 smithi006 ceph-mon[37501]: osdmap e273: 8 total, 8 up, 7 in 2024-03-26T04:00:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:28.893 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:28.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:28.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:29.025 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:29.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 57 16 91901 91885 100.727 85.0625 0.00891614 0.00992327 2024-03-26T04:00:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T04:00:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: pgmap v528: 158 pgs: 10 unknown, 1 peering, 1 active+clean+scrubbing, 146 active+clean; 5.3 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-26T04:00:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:28 smithi086 ceph-mon[32965]: osdmap e273: 8 total, 8 up, 7 in 2024-03-26T04:00:29.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:29.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:29.194 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:29.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:29.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:29.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:29.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:29.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:29.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:29.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:29.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:29.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:29.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:29.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:29.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:29.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:29.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:30.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:30.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 58 16 93193 93177 100.383 80.75 0.0106532 0.00995733 2024-03-26T04:00:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:00:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:00:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T04:00:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T04:00:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T04:00:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T04:00:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: osdmap e274: 8 total, 8 up, 7 in 2024-03-26T04:00:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:29 smithi086 ceph-mon[32965]: 6.13 scrub starts 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T04:00:30.104 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: osdmap e274: 8 total, 8 up, 7 in 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[37501]: 6.13 scrub starts 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: osdmap e274: 8 total, 8 up, 7 in 2024-03-26T04:00:30.105 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:29 smithi006 ceph-mon[29446]: 6.13 scrub starts 2024-03-26T04:00:30.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:30.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:30.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:30.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:30.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:30.417 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:30.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:30.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:30.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:00:30.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:30.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:30.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:30.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.822 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:30.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 6.7 deep-scrub starts 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 6.7 deep-scrub ok 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: pgmap v532: 158 pgs: 10 unknown, 1 peering, 147 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 843 op/s 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 6.13 scrub ok 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: osdmap e275: 8 total, 8 up, 7 in 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T04:00:30.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 6.7 deep-scrub starts 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 6.7 deep-scrub ok 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: pgmap v532: 158 pgs: 10 unknown, 1 peering, 147 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 843 op/s 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 6.13 scrub ok 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: osdmap e275: 8 total, 8 up, 7 in 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T04:00:30.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:30 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T04:00:30.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:30.958 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:30.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:30.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:31.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:31.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 59 16 94412 94396 99.9726 76.1875 0.00674553 0.00999796 2024-03-26T04:00:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 6.7 deep-scrub starts 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 6.7 deep-scrub ok 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: pgmap v532: 158 pgs: 10 unknown, 1 peering, 147 active+clean; 5.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 55 MiB/s wr, 843 op/s 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 6.13 scrub ok 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: osdmap e275: 8 total, 8 up, 7 in 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T04:00:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:30 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T04:00:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:31.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:31.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:31.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:31.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:31.365 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:31.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:31.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:31.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:31.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:31.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:31.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:31.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:31.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:31.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:32.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:32.046 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:32.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:00:32.083697+0000 min lat: 0.00120616 max lat: 0.11982 avg lat: 0.0100473 2024-03-26T04:00:32.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:00:32.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 60 16 95547 95531 99.4886 70.9375 0.0308388 0.0100473 2024-03-26T04:00:32.147 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:32.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:32.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:32.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.336 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:32.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:32.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.497 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[37501]: 3.12 scrub starts 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[37501]: 3.12 scrub ok 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[37501]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[37501]: pgmap v535: 158 pgs: 1 active+undersized+remapped, 8 remapped+peering, 1 unknown, 148 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[29446]: 3.12 scrub starts 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[29446]: 3.12 scrub ok 2024-03-26T04:00:32.530 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[29446]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T04:00:32.531 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:32 smithi006 ceph-mon[29446]: pgmap v535: 158 pgs: 1 active+undersized+remapped, 8 remapped+peering, 1 unknown, 148 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:00:32.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:32.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:32.663 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:00:32.707 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:00:32.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:32.755 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:00:32.775 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:32.805 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:00:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:32 smithi086 ceph-mon[32965]: 3.12 scrub starts 2024-03-26T04:00:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:32 smithi086 ceph-mon[32965]: 3.12 scrub ok 2024-03-26T04:00:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:32 smithi086 ceph-mon[32965]: osdmap e276: 8 total, 8 up, 7 in 2024-03-26T04:00:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:32 smithi086 ceph-mon[32965]: pgmap v535: 158 pgs: 1 active+undersized+remapped, 8 remapped+peering, 1 unknown, 148 active+clean; 5.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:00:32.859 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:00:32.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:32.903 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:00:32.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:32.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:32.944 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:00:32.989 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:00:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:00:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:00:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:00:33.003 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:00:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:00:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:00:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:00:33.004 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:00:33.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.061 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:33.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:33.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 61 16 96720 96704 99.0595 73.3125 0.0105895 0.0100902 2024-03-26T04:00:33.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:00:33.095 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:00:33.095 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-26T04:00:33.096 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T04:00:33.096 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T04:00:33.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-26T04:00:33.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:33.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:33.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:33.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:33.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:33.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:33.461 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:00:33.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:33.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:33.551 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:00:33.551 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:00:33.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:33.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:33.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: Health check update: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: 3.1e deep-scrub starts 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: 3.1e deep-scrub ok 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: 1.3 scrub starts 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: 1.3 scrub ok 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/257314478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: Health check update: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: 3.1e deep-scrub starts 2024-03-26T04:00:33.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: 3.1e deep-scrub ok 2024-03-26T04:00:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: 1.3 scrub starts 2024-03-26T04:00:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: 1.3 scrub ok 2024-03-26T04:00:33.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/257314478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:33.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: Health check update: Reduced data availability: 8 pgs peering (PG_AVAILABILITY) 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: osdmap e277: 8 total, 8 up, 7 in 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: 3.1e deep-scrub starts 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: 3.1e deep-scrub ok 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: 1.3 scrub starts 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: 1.3 scrub ok 2024-03-26T04:00:33.744 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/257314478' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:33.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:33.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:33.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:33.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:34.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.045 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:34.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:34.083 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 62 16 98066 98050 98.8185 84.125 0.00968309 0.0101149 2024-03-26T04:00:34.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:34.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:34.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:34.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:34.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:34.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:34.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.466 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:34.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:34.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:34.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:34.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.9 scrub starts 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.9 scrub ok 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.5 scrub starts 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.5 scrub ok 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.8 scrub starts 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.8 scrub ok 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.6 scrub starts 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.6 scrub ok 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.7 scrub starts 2024-03-26T04:00:34.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 1.7 scrub ok 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: pgmap v537: 158 pgs: 1 active+undersized+remapped, 5 remapped+peering, 1 unknown, 151 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 KiB/s rd, 79 MiB/s wr, 1.33k op/s; 311 KiB/s, 3 objects/s recovering 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 5.19 scrub starts 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: 5.19 scrub ok 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[37501]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.9 scrub starts 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.9 scrub ok 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.5 scrub starts 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.5 scrub ok 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.8 scrub starts 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.8 scrub ok 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.6 scrub starts 2024-03-26T04:00:34.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.6 scrub ok 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.7 scrub starts 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 1.7 scrub ok 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: pgmap v537: 158 pgs: 1 active+undersized+remapped, 5 remapped+peering, 1 unknown, 151 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 KiB/s rd, 79 MiB/s wr, 1.33k op/s; 311 KiB/s, 3 objects/s recovering 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 5.19 scrub starts 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: 5.19 scrub ok 2024-03-26T04:00:34.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:34 smithi006 ceph-mon[29446]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T04:00:34.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:34.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.9 scrub starts 2024-03-26T04:00:34.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.9 scrub ok 2024-03-26T04:00:34.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.5 scrub starts 2024-03-26T04:00:34.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.5 scrub ok 2024-03-26T04:00:34.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.8 scrub starts 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.8 scrub ok 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.6 scrub starts 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.6 scrub ok 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.7 scrub starts 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 1.7 scrub ok 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: pgmap v537: 158 pgs: 1 active+undersized+remapped, 5 remapped+peering, 1 unknown, 151 active+clean; 5.7 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 53 KiB/s rd, 79 MiB/s wr, 1.33k op/s; 311 KiB/s, 3 objects/s recovering 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 5.19 scrub starts 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: 5.19 scrub ok 2024-03-26T04:00:34.733 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-mon[32965]: osdmap e278: 8 total, 8 up, 7 in 2024-03-26T04:00:34.739 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:34.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:34.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:34.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:34.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:34.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T04:00:34.984 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:34.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:35.062 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:35.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 63 16 99416 99400 98.5885 84.375 0.00857528 0.0101386 2024-03-26T04:00:35.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:35.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:35.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:35.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:35.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:35.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:35.416 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:00:35.417 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-26T04:00:35.417 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:00:35.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:35.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:35.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:35.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:35.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 5.1 scrub starts 2024-03-26T04:00:35.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 5.1 scrub ok 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.2 deep-scrub starts 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.2 deep-scrub ok 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.1 scrub starts 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.1 scrub ok 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T04:00:35.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 5.1 scrub starts 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 5.1 scrub ok 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.2 deep-scrub starts 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.2 deep-scrub ok 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.1 scrub starts 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.1 scrub ok 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T04:00:35.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T04:00:35.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:35.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 5.1 scrub starts 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 5.1 scrub ok 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.2 deep-scrub starts 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.2 deep-scrub ok 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.1 scrub starts 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.1 scrub ok 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T04:00:35.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:35.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:35.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:35.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:35.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:35.884 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:35.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:35.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:36.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:36.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 64 16 100704 100688 98.3058 80.5 0.00480185 0.0101677 2024-03-26T04:00:36.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:36.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:36.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:36.305 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:36.356 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:36.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:36.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:00:36.463 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:36.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:36.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T04:00:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T04:00:36.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 1.a deep-scrub starts 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 1.a deep-scrub ok 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: pgmap v539: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 6.1 deep-scrub starts 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[29446]: 6.1 deep-scrub ok 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T04:00:36.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 1.a deep-scrub starts 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 1.a deep-scrub ok 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: pgmap v539: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 6.1 deep-scrub starts 2024-03-26T04:00:36.642 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:36 smithi006 ceph-mon[37501]: 6.1 deep-scrub ok 2024-03-26T04:00:36.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:36.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 1.a deep-scrub starts 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 1.a deep-scrub ok 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: pgmap v539: 158 pgs: 1 active+undersized+remapped, 157 active+clean; 6.0 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 77 MiB/s wr, 1.22k op/s 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs peering) 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 6.1 deep-scrub starts 2024-03-26T04:00:36.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:36 smithi086 ceph-mon[32965]: 6.1 deep-scrub ok 2024-03-26T04:00:36.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:36.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:37.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:37.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 65 16 101895 101879 97.9383 74.4375 0.0145011 0.0102055 2024-03-26T04:00:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.092 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:37.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:37.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:37.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:37.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:37.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:37.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:37.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:37.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:37.538 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T04:00:37.538 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T04:00:37.538 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T04:00:37.538 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T04:00:37.539 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T04:00:37.539 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T04:00:37.539 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T04:00:37.539 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:37 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T04:00:37.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:37.618 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:37.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:37.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:37.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:37 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T04:00:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:37 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T04:00:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:37 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T04:00:37.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:37 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T04:00:37.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:37.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:37.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:37.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:38.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:38.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 66 16 102983 102967 97.4846 68 0.00827063 0.0102535 2024-03-26T04:00:38.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:38.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:38.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:38.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:00:38.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:38.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:38.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:38.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:38.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: pgmap v540: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2037816420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:38 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: pgmap v540: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2037816420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: pgmap v540: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 64 MiB/s wr, 1.01k op/s 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2037816420' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:38 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:00:38.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:38.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:38.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:38.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:39.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:39.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 67 16 104106 104090 97.077 70.1875 0.0135195 0.0102966 2024-03-26T04:00:39.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:39.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:39.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:39.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:39.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:39.379 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:39.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:39.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:39.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: 4.10 scrub ok 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[37501]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: 4.10 scrub ok 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:39.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:39 smithi006 ceph-mon[29446]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T04:00:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:39.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:39.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:39.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: 4.10 scrub ok 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: osdmap e279: 8 total, 8 up, 7 in 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:39 smithi086 ceph-mon[32965]: osdmap e280: 8 total, 8 up, 7 in 2024-03-26T04:00:39.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:39.934 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:39.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:39.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:40.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.085 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:40.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:40.086 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 68 16 105423 105407 96.8598 82.3125 0.0119443 0.010319 2024-03-26T04:00:40.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:40.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:40.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:40.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:40.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:40.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:40.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:40.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:40.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:40.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:40.615 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:40.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:40.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:40.757 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:40.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:40.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:40 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:40 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:40.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:40 smithi086 ceph-mon[32965]: pgmap v543: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 76 MiB/s wr, 1.19k op/s 2024-03-26T04:00:40.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:40.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[37501]: pgmap v543: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 76 MiB/s wr, 1.19k op/s 2024-03-26T04:00:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:40 smithi006 ceph-mon[29446]: pgmap v543: 158 pgs: 158 active+clean; 6.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 76 MiB/s wr, 1.19k op/s 2024-03-26T04:00:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:40.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:41.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:41.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:41.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 69 16 106745 106729 96.6534 82.625 0.0138542 0.0103418 2024-03-26T04:00:41.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:41.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:41.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:41.203 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:00:41.203 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:00:41.203 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-03-26T04:00:41.203 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T04:00:41.203 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T04:00:41.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-03-26T04:00:41.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:41.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:41.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:41.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:41.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:41.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:41.569 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:00:41.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:41.700 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:41.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:41.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:41.876 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:41.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:41.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:41.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:42.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:42.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:42.084 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 70 16 108040 108024 96.4288 80.9375 0.00867466 0.0103655 2024-03-26T04:00:42.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.153 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:42.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:42.254 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.292 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:42.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:42.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.431 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:42.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:42.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:42.565 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:42.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:42.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:42.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:42.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:42.732 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:42 smithi086 ceph-mon[32965]: pgmap v544: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 MiB/s wr, 1.34k op/s 2024-03-26T04:00:42.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:42 smithi006 ceph-mon[37501]: pgmap v544: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 MiB/s wr, 1.34k op/s 2024-03-26T04:00:42.765 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:42 smithi006 ceph-mon[29446]: pgmap v544: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 85 MiB/s wr, 1.34k op/s 2024-03-26T04:00:42.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:42.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:42.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:42.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:42.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:42.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:43.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:00:43.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:43.085 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 71 16 109208 109192 96.0976 73 0.00906967 0.0104009 2024-03-26T04:00:43.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:43.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:43.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:43.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:43.275 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:00:43.275 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:00:43.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:43.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.394 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:43.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:43.477 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:00:43.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:43.498 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:00:43.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:43.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:43.547 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:00:43.594 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3665883169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:43.595 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3665883169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:43.595 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:00:43.633 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:00:43.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:43.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:43.702 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:00:43.744 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:00:43.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:43.788 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:00:43.792 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:00:43.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:43.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3665883169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:43.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:43.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:43.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:43.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:44.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:44.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:44.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:44.085 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 72 16 110192 110176 95.6171 61.5 0.00483225 0.0104538 2024-03-26T04:00:44.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:44.210 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:00:44.210 INFO:teuthology.orchestra.run.smithi086.stdout: "success": "" 2024-03-26T04:00:44.210 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:00:44.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:44.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:44.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T04:00:44.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:44.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:44.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:44.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:44.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.638 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:44.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:44.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:44.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:44.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: pgmap v545: 158 pgs: 158 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-26T04:00:44.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:00:44.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[29446]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:44.643 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: pgmap v545: 158 pgs: 158 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-26T04:00:44.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:00:44.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:00:44.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T04:00:44.644 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:44 smithi006 ceph-mon[37501]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T04:00:44.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:44.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: pgmap v545: 158 pgs: 158 active+clean; 6.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 65 MiB/s wr, 1.04k op/s 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: osdmap e281: 8 total, 8 up, 7 in 2024-03-26T04:00:44.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-mon[32965]: osdmap e282: 8 total, 8 up, 7 in 2024-03-26T04:00:44.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:44.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:44.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:44.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:45.018 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:45.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.068 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:45.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:45.087 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 73 16 111425 111409 95.3613 77.0625 0.0147375 0.0104822 2024-03-26T04:00:45.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:45.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:45.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:45.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:45.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:45.361 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:45.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:45.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:45.504 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:45.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:45.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:45.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:45.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.781 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:45.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:45.818 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:45.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:45.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:45.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:45.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:45.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:45.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:46.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:46.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:46.087 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 74 16 112736 112720 95.1798 81.9375 0.00787178 0.0105019 2024-03-26T04:00:46.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:46.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:46.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:46.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.325 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:46.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:46.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:46.460 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:46.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:46.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:46.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:46.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:46 smithi086 ceph-mon[32965]: pgmap v548: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 847 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-26T04:00:46.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:46.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:46.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:46.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:46 smithi006 ceph-mon[37501]: pgmap v548: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 847 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-26T04:00:46.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:46 smithi006 ceph-mon[29446]: pgmap v548: 158 pgs: 158 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 847 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-03-26T04:00:46.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:46.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:00:47.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:47.087 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 75 16 114078 114062 95.029 83.875 0.0183697 0.0105181 2024-03-26T04:00:47.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:47.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:47.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:47.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.283 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:47.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:47.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:47.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:47.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:47.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:47.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:47.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:47.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:47.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:47.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:47.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:47.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:48.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:48.089 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 76 16 115226 115210 94.7227 71.75 0.0153245 0.0105524 2024-03-26T04:00:48.095 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:48.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:48.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:48.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:48.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:48.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:48.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:48.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:48.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:48.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:48.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:48.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:48.770 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:48 smithi086 ceph-mon[32965]: pgmap v549: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T04:00:48.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:48.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:48 smithi006 ceph-mon[29446]: pgmap v549: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T04:00:48.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:48 smithi006 ceph-mon[37501]: pgmap v549: 158 pgs: 158 active+clean; 6.8 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-26T04:00:48.915 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:48.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:49.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:49.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:49.087 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 77 16 116458 116442 94.4924 77 0.006225 0.0105781 2024-03-26T04:00:49.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:49.193 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:49.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.324 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:49.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:49.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:49.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:49.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:49.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:49.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:49.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:49.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:49.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:49.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:49.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:49.972 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:00:49.972 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:00:49.972 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-26T04:00:49.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:00:50.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:50.087 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 78 16 117667 117651 94.2497 75.5625 0.0167349 0.010606 2024-03-26T04:00:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.130 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:50.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:50.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:50.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:50.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:50.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:50.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:50.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:50.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:50.549 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:50.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:50.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:50.658 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:50 smithi006 ceph-mon[37501]: pgmap v550: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-26T04:00:50.658 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:50 smithi006 ceph-mon[29446]: pgmap v550: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-26T04:00:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:50.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:50.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:50.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:50.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:50 smithi086 ceph-mon[32965]: pgmap v550: 158 pgs: 158 active+clean; 6.9 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s 2024-03-26T04:00:50.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:50.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:50.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:50.977 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:51.042 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:51.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:51.059 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:00:51.059 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:00:51.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215919 8.7 8:e03686cb:::benchmark_data_smithi006_103169_object85988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 182.56714837600001, 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9954844170000001, 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215919 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.060 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478962+0000", 2024-03-26T04:00:51.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.252e-06 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478968+0000", 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6489999999999996e-06 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.062 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478978+0000", 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0757999999999999e-05 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478987+0000", 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6619999999999994e-06 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479605+0000", 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00061837100000000005 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.063 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479713+0000", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107643 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479818+0000", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105418 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484843+0000", 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050241180000000002 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484884+0000", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1573000000000003e-05 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531169+0000", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.046285129000000001 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531182+0000", 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2838e-05 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.350540+0000", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81935782999999995 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.350553+0000", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3091000000000001e-05 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474362+0000", 2024-03-26T04:00:51.066 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.123808623 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474445+0000", 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3461999999999995e-05 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215922 8.7 8:e91e8581:::benchmark_data_smithi006_103169_object85991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:00:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 182.56682194800001, 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9959090370000001, 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215922 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479288+0000", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3799999999999996e-07 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479293+0000", 2024-03-26T04:00:51.069 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1699999999999996e-06 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479304+0000", 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0829000000000001e-05 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479311+0000", 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3499999999999999e-06 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.070 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482220+0000", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029082969999999998 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482335+0000", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114765 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482446+0000", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111179 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484848+0000", 2024-03-26T04:00:51.071 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024021229999999999 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.489800+0000", 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049523019999999996 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531214+0000", 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041413932000000001 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.072 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531223+0000", 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2799999999999992e-06 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351560+0000", 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82033632300000003 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.073 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351570+0000", 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0655999999999999e-05 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475126+0000", 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.123555978 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475196+0000", 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9914999999999997e-05 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.074 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215939 8.0 8:06bae83a:::benchmark_data_smithi006_103169_object86008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 182.50616138500001, 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.999205474, 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215939 2024-03-26T04:00:51.075 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.076 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539949+0000", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2720000000000001e-06 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539957+0000", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7510000000000008e-06 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539970+0000", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3461999999999999e-05 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.077 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539989+0000", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8306000000000001e-05 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.540071+0000", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2101999999999994e-05 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.540082+0000", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1216000000000001e-05 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.078 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.348869+0000", 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80878707500000002 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.348882+0000", 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3156000000000001e-05 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.524064+0000", 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.17518202 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.079 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.524582+0000", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051825700000000005 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528374+0000", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037915850000000001 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528484+0000", 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010967999999999999 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.080 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,4,5", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528634+0000", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015087499999999999 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537225+0000", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085907970000000007 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537255+0000", 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9855000000000001e-05 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.081 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537305+0000", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0235000000000003e-05 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537421+0000", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011579 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537485+0000", 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3402e-05 2024-03-26T04:00:51.082 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539153+0000", 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016686380000000001 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215944 8.7 8:edad7f0e:::benchmark_data_smithi006_103169_object86013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 182.49781239999999, 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0056385689999998, 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.083 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215944 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.084 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548298+0000", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5319999999999999e-06 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548305+0000", 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3049999999999998e-06 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.085 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548318+0000", 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3959e-05 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548327+0000", 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5760000000000006e-06 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.086 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548399+0000", 2024-03-26T04:00:51.087 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1643000000000005e-05 2024-03-26T04:00:51.087 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.087 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.087 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548409+0000", 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0488e-05 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351605+0000", 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80319530400000005 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.088 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351613+0000", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8189999999999998e-06 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539126+0000", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1875127670000001 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539245+0000", 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118879 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.089 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539341+0000", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5586000000000003e-05 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552494+0000", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013153583 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552617+0000", 2024-03-26T04:00:51.090 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012313 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553170+0000", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055303500000000001 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553426+0000", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025522899999999999 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553808+0000", 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038286700000000001 2024-03-26T04:00:51.091 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553935+0000", 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012686699999999999 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215910 8.b 8:da361f87:::benchmark_data_smithi006_103169_object85979:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.70749142, 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.13719901, 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.092 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215910 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.093 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338619+0000", 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.505e-06 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338625+0000", 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.327e-06 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.094 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338635+0000", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0754000000000001e-05 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338646+0000", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1031e-05 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338887+0000", 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024031500000000001 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.095 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338894+0000", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3730000000000004e-06 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.356045+0000", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017151473 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.356056+0000", 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0589e-05 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.096 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474260+0000", 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.118203982 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475817+0000", 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001556661 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215912 8.b 8:d870adcc:::benchmark_data_smithi006_103169_object85981:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.097 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.707438835, 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1374827700000001, 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215912 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.098 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338671+0000", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3000000000000001e-07 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338674+0000", 2024-03-26T04:00:51.099 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6459999999999999e-06 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338681+0000", 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8959999999999997e-06 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338688+0000", 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4049999999999997e-06 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.100 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339027+0000", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033922800000000001 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339032+0000", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5269999999999998e-06 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358921+0000", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019888565 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.101 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358931+0000", 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0400999999999999e-05 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476084+0000", 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1171520420000001 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476153+0000", 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9529999999999993e-05 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.102 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215918 8.b 8:d5bbe624:::benchmark_data_smithi006_103169_object85987:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.707412699, 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1375509960000001, 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215918 2024-03-26T04:00:51.103 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338697+0000", 2024-03-26T04:00:51.104 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.65e-07 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338699+0000", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2280000000000001e-06 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338705+0000", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7270000000000004e-06 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338710+0000", 2024-03-26T04:00:51.105 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3689999999999998e-06 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339061+0000", 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035118299999999997 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339066+0000", 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8579999999999999e-06 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.106 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358952+0000", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019886052000000001 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358958+0000", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.178e-06 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476183+0000", 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.117224792 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.107 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476247+0000", 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4244000000000004e-05 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215923 8.b 8:d3d0d2cc:::benchmark_data_smithi006_103169_object85992:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.70739143500001, 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1376206259999999, 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:00:51.108 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215923 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.109 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4999999999999998e-07 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338721+0000", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4310000000000001e-06 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338726+0000", 2024-03-26T04:00:51.110 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3249999999999998e-06 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338731+0000", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6030000000000004e-06 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339093+0000", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036193599999999999 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339098+0000", 2024-03-26T04:00:51.111 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8559999999999998e-06 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358978+0000", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019879903000000001 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358984+0000", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8200000000000002e-06 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476277+0000", 2024-03-26T04:00:51.112 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1172925170000001 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476338+0000", 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1785000000000006e-05 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215928 8.f 8:f5663f39:::benchmark_data_smithi006_103169_object85997:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.70736985799999, 2024-03-26T04:00:51.113 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.199381638, 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215928 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.114 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:51.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:51.115 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.115 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.115 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.115 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:00:51.115 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338740+0000", 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4099999999999999e-07 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.116 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338742+0000", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8030000000000001e-06 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338748+0000", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7740000000000001e-06 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338754+0000", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8340000000000004e-06 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.117 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339328+0000", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057439999999999998 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339335+0000", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6170000000000003e-06 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351418+0000", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012082881 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.118 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351426+0000", 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8380000000000002e-06 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528886+0000", 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1774600500000001 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.530740+0000", 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001854154 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.119 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.530984+0000", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024428999999999998 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537856+0000", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068720939999999996 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537872+0000", 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5116000000000001e-05 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.120 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537908+0000", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6279999999999998e-05 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537948+0000", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0487e-05 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538014+0000", 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6076999999999998e-05 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.121 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538121+0000", 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106502 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215941 8.b 8:d0ea7a54:::benchmark_data_smithi006_103169_object86010:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.707339219, 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.152127299, 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.122 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215941 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:00:51.123 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9900000000000001e-07 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338773+0000", 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.773e-06 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.124 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338779+0000", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3409999999999999e-06 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338786+0000", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339125+0000", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033892500000000001 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:00:51.125 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339130+0000", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6600000000000003e-06 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.359012+0000", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019882115999999998 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.359018+0000", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1380000000000004e-06 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.126 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479021+0000", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1200031269999999 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479501+0000", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00048008000000000002 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479629+0000", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012821600000000001 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.489981+0000", 2024-03-26T04:00:51.127 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010351816999999999 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490089+0000", 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010812 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.128 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.128 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 79 16 118722 118706 93.8856 65.9375 0.043853 0.0106465 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490173+0000", 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3881999999999994e-05 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490507+0000", 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033375599999999999 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.129 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490569+0000", 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2279999999999993e-05 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490897+0000", 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000327852 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215946 8.f 8:f2c0634b:::benchmark_data_smithi006_103169_object86015:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:00:51.130 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 181.707314731, 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.214591854, 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215946 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:00:51.131 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338795+0000", 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.75e-07 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.132 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338797+0000", 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.65e-06 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338804+0000", 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1210000000000003e-06 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338810+0000", 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0739999999999998e-06 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.133 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339366+0000", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055637999999999998 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339371+0000", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0309999999999998e-06 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351451+0000", 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012079546 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.134 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351457+0000", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.764e-06 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538151+0000", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1866938380000001 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538279+0000", 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127262 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.135 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538419+0000", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014042899999999999 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552469+0000", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014049682000000001 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552483+0000", 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4778e-05 2024-03-26T04:00:51.136 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553042+0000", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055900500000000001 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553209+0000", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016694799999999999 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553270+0000", 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0551999999999999e-05 2024-03-26T04:00:51.137 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553386+0000", 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116319 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:216073 8.3 8:cdc5e149:::benchmark_data_smithi006_103169_object86142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:51.391513+0000", 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 179.65459594199999, 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99432406900000003, 2024-03-26T04:00:51.138 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 216073 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391513+0000", 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.139 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391513+0000", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391515+0000", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7230000000000001e-06 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391524+0000", 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2920000000000001e-06 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.140 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391535+0000", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0764000000000001e-05 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391543+0000", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1349999999999992e-06 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391611+0000", 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8392000000000005e-05 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.141 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.391621+0000", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4970000000000007e-06 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.382732+0000", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99111082699999997 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.382868+0000", 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000136408 2024-03-26T04:00:51.142 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.382964+0000", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5499000000000002e-05 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385696+0000", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027319660000000002 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385709+0000", 2024-03-26T04:00:51.143 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3217000000000001e-05 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385749+0000", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9959999999999997e-05 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385799+0000", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9743000000000001e-05 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385837+0000", 2024-03-26T04:00:51.144 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8646000000000002e-05 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:216087 8.3 8:cabc8322:::benchmark_data_smithi006_103169_object86156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:51.413124+0000", 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 179.63298483700001, 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.97796300700000005, 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.145 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 216087 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413124+0000", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413124+0000", 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.146 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413125+0000", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1799999999999999e-06 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413131+0000", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3279999999999996e-06 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413142+0000", 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0998e-05 2024-03-26T04:00:51.147 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413150+0000", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4489999999999999e-06 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413225+0000", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4783999999999994e-05 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.413236+0000", 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0835e-05 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.148 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385877+0000", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.97264119100000002 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.385967+0000", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0239000000000001e-05 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.386092+0000", 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000124339 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.149 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.390661+0000", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045689859999999997 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.390754+0000", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3728999999999997e-05 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.390814+0000", 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9691000000000002e-05 2024-03-26T04:00:51.150 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391040+0000", 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00022642800000000001 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391087+0000", 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6829999999999997e-05 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.151 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:216099 8.3 8:c9e1bdc3:::benchmark_data_smithi006_103169_object86168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:51.472279+0000", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 179.57383008799999, 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91922069699999998, 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 216099 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.152 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472279+0000", 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472279+0000", 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472280+0000", 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.361e-06 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.153 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472286+0000", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1510000000000003e-06 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472296+0000", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.078e-05 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472305+0000", 2024-03-26T04:00:51.154 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3899999999999993e-06 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472380+0000", 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5483999999999998e-05 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.155 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:51.472389+0000", 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7600000000000008e-06 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.388513+0000", 2024-03-26T04:00:51.158 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91612386899999998 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.388633+0000", 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120272 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,5", 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.388812+0000", 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017911500000000001 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.159 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391107+0000", 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002294609 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391138+0000", 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1266000000000002e-05 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391302+0000", 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000164298 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.160 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391456+0000", 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015331899999999999 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:52.391500+0000", 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4023000000000002e-05 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235177 8.7 8:eff9d2da:::benchmark_data_smithi006_103169_object105246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.161 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.524738996, 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99837947699999996, 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235177 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:00:51.162 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521371+0000", 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.19e-06 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.163 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521376+0000", 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4730000000000002e-06 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521383+0000", 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1409999999999999e-06 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521411+0000", 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7846000000000001e-05 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.164 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521523+0000", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001121 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521566+0000", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2948000000000003e-05 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514095+0000", 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99252881900000001 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.165 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514211+0000", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011636700000000001 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514295+0000", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3829e-05 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519561+0000", 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0052661920000000003 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.166 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519653+0000", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2083999999999995e-05 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519686+0000", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2561000000000002e-05 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519720+0000", 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4196999999999998e-05 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.167 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519750+0000", 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9730000000000002e-05 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235179 8.7 8:e08ecef2:::benchmark_data_smithi006_103169_object105248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.52356117799999, 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99733449600000001, 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.168 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235179 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:00:51.169 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522549+0000", 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3200000000000004e-07 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522553+0000", 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5210000000000004e-06 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.170 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522564+0000", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0729e-05 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522571+0000", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.742e-06 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522619+0000", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8296e-05 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.171 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522627+0000", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8420000000000005e-06 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516438+0000", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99381165900000001 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516533+0000", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4327999999999995e-05 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.172 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516644+0000", 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011116999999999999 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519620+0000", 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029761089999999998 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.173 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519810+0000", 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000190106 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519834+0000", 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4304e-05 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519861+0000", 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6877e-05 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.174 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519882+0000", 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1081e-05 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235192 8.7 8:ebe531e6:::benchmark_data_smithi006_103169_object105261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.533881+0000", 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.512228378, 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98612859500000005, 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.175 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235192 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533881+0000", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533881+0000", 2024-03-26T04:00:51.176 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533882+0000", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2529999999999999e-06 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533888+0000", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6710000000000004e-06 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533900+0000", 2024-03-26T04:00:51.177 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2347000000000001e-05 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533906+0000", 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.404e-06 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533954+0000", 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7314999999999997e-05 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.178 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.533961+0000", 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7200000000000006e-06 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.518042+0000", 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98408059400000003 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.518155+0000", 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000112631 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.179 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.518277+0000", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012231699999999999 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519918+0000", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016412040000000001 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519926+0000", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6590000000000007e-06 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.180 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519955+0000", 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9408e-05 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519980+0000", 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5131e-05 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.520009+0000", 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8941e-05 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.181 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235218 8.7 8:e0a68d84:::benchmark_data_smithi006_103169_object105287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.576058+0000", 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.470051098, 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.95263376600000005, 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235218 2024-03-26T04:00:51.182 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576058+0000", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576058+0000", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576059+0000", 2024-03-26T04:00:51.183 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.02e-06 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576064+0000", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6809999999999997e-06 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576075+0000", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1423999999999999e-05 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576081+0000", 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0909999999999998e-06 2024-03-26T04:00:51.184 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.185 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.185 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.185 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576129+0000", 2024-03-26T04:00:51.185 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7518999999999997e-05 2024-03-26T04:00:51.185 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.576135+0000", 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4620000000000001e-06 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.520043+0000", 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.943907995 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.186 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.520109+0000", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6095999999999999e-05 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.520214+0000", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000104335 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528428+0000", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082148480000000003 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.187 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528491+0000", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3006e-05 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528580+0000", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8590000000000004e-05 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528643+0000", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3260000000000001e-05 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.188 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528692+0000", 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8439000000000002e-05 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235267 8.7 8:e964624f:::benchmark_data_smithi006_103169_object105336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.670344+0000", 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.375765489, 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.85852290799999997, 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.189 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235267 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670344+0000", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.190 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670344+0000", 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670345+0000", 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1739999999999999e-06 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670349+0000", 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6149999999999999e-06 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.191 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670361+0000", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1575e-05 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670367+0000", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5060000000000001e-06 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670417+0000", 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9165000000000001e-05 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.192 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.670424+0000", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.525e-06 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.522132+0000", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.85170735500000005 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.522219+0000", 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7369999999999999e-05 2024-03-26T04:00:51.193 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.522320+0000", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101001 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528434+0000", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061141099999999999 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528743+0000", 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030937500000000003 2024-03-26T04:00:51.194 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528806+0000", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2848000000000005e-05 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528841+0000", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5099999999999999e-05 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528867+0000", 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5188999999999999e-05 2024-03-26T04:00:51.195 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235299 8.7 8:e4661533:::benchmark_data_smithi006_103169_object105368:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.742724+0000", 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 134.303384808, 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.78646655300000001, 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235299 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.196 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742724+0000", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742724+0000", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742725+0000", 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4799999999999997e-07 2024-03-26T04:00:51.197 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742730+0000", 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.685e-06 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742741+0000", 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1637999999999999e-05 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:00:51.198 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742749+0000", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5290000000000003e-06 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742800+0000", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0858e-05 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.742806+0000", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8610000000000003e-06 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:00:51.199 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.524303+0000", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.78149767599999997 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.524410+0000", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010640400000000001 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.524526+0000", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115919 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528544+0000", 2024-03-26T04:00:51.200 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004018293 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.528909+0000", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036538400000000001 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.529123+0000", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000213332 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.529153+0000", 2024-03-26T04:00:51.201 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0329999999999999e-05 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.529191+0000", 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7796000000000002e-05 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:00:51.202 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:00:51.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:51.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:51.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:51.439 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:00:51.445 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:51.446 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:51.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:51.532 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:51.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:51.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:51.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:51.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:51.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:51.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:00:51.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:51.950 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:51.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:52.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:52.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:00:52.093731+0000 min lat: 0.00120616 max lat: 0.150514 avg lat: 0.0106669 2024-03-26T04:00:52.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:00:52.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 80 16 119995 119979 93.7057 79.5625 0.00887486 0.0106669 2024-03-26T04:00:52.098 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:52.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:52.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:52.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:52.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:52.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:52.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:52.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T04:00:52.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:52.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:52.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:52.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[37501]: pgmap v551: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T04:00:52.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[37501]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T04:00:52.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:52.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:52.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[29446]: pgmap v551: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T04:00:52.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:52 smithi006 ceph-mon[29446]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T04:00:52.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:52.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:52.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:52.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:52.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:52 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:00:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2630298544' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:00:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:52 smithi086 ceph-mon[32965]: pgmap v551: 158 pgs: 158 active+clean; 7.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-26T04:00:52.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:52 smithi086 ceph-mon[32965]: osdmap e283: 8 total, 8 up, 7 in 2024-03-26T04:00:52.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:52.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:52.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:53.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:53.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:53.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 81 16 121303 121287 93.5581 81.75 0.0126329 0.0106839 2024-03-26T04:00:53.182 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:53.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:53.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:53.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:53.459 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:53.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:53.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:53.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:53.635 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:53.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:53.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:53.763 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:53.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:53.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:53.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:53.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:53.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:54.036 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:54.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 82 16 122544 122528 93.3629 77.5625 0.00784943 0.0107059 2024-03-26T04:00:54.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:54.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:54.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:54.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:54.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:54.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:54.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:54.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:54.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[29446]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 566 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[29446]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[37501]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 566 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-mon[37501]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T04:00:54.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:00:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:00:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:00:54.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:54.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.720 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:54.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:54.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:54 smithi086 ceph-mon[32965]: pgmap v553: 158 pgs: 158 active+clean; 7.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 566 B/s rd, 77 MiB/s wr, 1.24k op/s 2024-03-26T04:00:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:00:54.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:54 smithi086 ceph-mon[32965]: osdmap e284: 8 total, 8 up, 7 in 2024-03-26T04:00:54.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:54.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:54.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:54.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:55.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 83 15 123892 123877 93.2538 84.3125 0.0121781 0.010719 2024-03-26T04:00:55.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:55.143 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:55.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:55.286 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:55.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:00:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:00:55.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:55.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:55.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:55.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:55.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:55.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:55.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:55.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:55.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.710 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:55.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:55.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:00:55.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:55.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:55.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:55.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:55.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:56.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:56.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 84 16 125217 125201 93.1287 82.75 0.0100315 0.0107333 2024-03-26T04:00:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:56.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:56.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:56.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:56.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:56.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:56.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:56.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:56.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:56.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:56.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:56.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:56.763 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:56 smithi006 ceph-mon[29446]: pgmap v555: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T04:00:56.764 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:56 smithi006 ceph-mon[37501]: pgmap v555: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T04:00:56.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:56.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.811 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:56.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:56 smithi086 ceph-mon[32965]: pgmap v555: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.45k op/s 2024-03-26T04:00:56.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:56.960 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:56.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:56.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:57.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:00:57.093 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 85 16 126574 126558 93.0308 84.8125 0.00831976 0.0107447 2024-03-26T04:00:57.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:00:57.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:57.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:57.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:57.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:57.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:57.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:00:57.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:57.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:00:57.586 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:00:57.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:57.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:57.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:57.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:57.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:57.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:57.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:57.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:57.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:57.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:00:58.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:58.094 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 86 16 127828 127812 92.8594 78.375 0.00852424 0.0107644 2024-03-26T04:00:58.113 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:58.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:58.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:58.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:00:58.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:58.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:00:58.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.531 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:58.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:58.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:00:58.670 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:00:58.702 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:58 smithi006 ceph-mon[37501]: pgmap v556: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T04:00:58.703 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1828904571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:58.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:58 smithi006 ceph-mon[29446]: pgmap v556: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T04:00:58.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1828904571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:58.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:00:58.786 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:58 smithi086 ceph-mon[32965]: pgmap v556: 158 pgs: 158 active+clean; 7.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-26T04:00:58.787 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1828904571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:00:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:58.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:00:58.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:58.947 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:58.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:58.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:59.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:59.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.096 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:59.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:59.096 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 87 16 129009 128993 92.6392 73.8125 0.0188923 0.0107895 2024-03-26T04:00:59.176 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:00:59.177 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:00:59.178 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:00:59.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:59.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:59.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:00:59.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:59.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:00:59.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:59.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:00:59.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.648 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:59.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:59.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:00:59.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:00:59.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:00:59.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:00:59 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:00:59.885 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:00:59 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:00:59.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:00:59 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:00:59.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:00:59.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:00:59.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:00:59.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:00.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:00.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.085 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:00.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:00.096 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 88 16 130036 130020 92.3154 64.1875 0.0118864 0.0108279 2024-03-26T04:01:00.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:00.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:00.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:00.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:00.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:00.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:00.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:00.506 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:00.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[29446]: pgmap v557: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T04:01:00.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:01:00.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:01:00.529 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:00.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:00.634 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:01:00.646 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:01:00.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.661 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:00.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:00.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:00.778 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:01:00.778 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:01:00.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:00.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:00.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[37501]: pgmap v557: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T04:01:00.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:01:00.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:01:00.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:00 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:00 smithi086 ceph-mon[32965]: pgmap v557: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.13k op/s 2024-03-26T04:01:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:00 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:01:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:00 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:01:00.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:00 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:00.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:00.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:00.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:00.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:01.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:01.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:01.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:01.096 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 89 16 131156 131140 92.0646 70 0.00634859 0.0108577 2024-03-26T04:01:01.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:01.218 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:01.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:01.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:01.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:01.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:01.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:01.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:01.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:01.763 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:01 smithi086 ceph-mon[32965]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:01.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:01 smithi006 ceph-mon[37501]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:01 smithi006 ceph-mon[29446]: from='client.25307 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:01.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:01.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:01.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:01.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:01:01.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:02.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:02.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total time run: 90.0079 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total writes made: 132454 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Write size: 65536 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object size: 65536 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Bandwidth (MB/sec): 91.9739 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Bandwidth: 28.1449 2024-03-26T04:01:02.097 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max bandwidth (MB/sec): 198.625 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min bandwidth (MB/sec): 61.5 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average IOPS: 1471 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev IOPS: 450.319 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max IOPS: 3178 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min IOPS: 984 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average Latency(s): 0.0108685 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Latency(s): 0.00716339 2024-03-26T04:01:02.098 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max latency(s): 0.150514 2024-03-26T04:01:02.099 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min latency(s): 0.00120616 2024-03-26T04:01:02.102 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T04:01:02.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:02.191 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:02.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:02.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:02.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:02.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:02.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:02.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:02.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:02.607 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:02.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:02.747 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:02 smithi086 ceph-mon[32965]: pgmap v558: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T04:01:02.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:02.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:02.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:02.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:02.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:02 smithi006 ceph-mon[37501]: pgmap v558: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T04:01:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:02 smithi006 ceph-mon[29446]: pgmap v558: 158 pgs: 158 active+clean; 7.9 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-26T04:01:02.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:03.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:03.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:03.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:03.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:03.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:03.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:03.439 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-03-26T04:00:59.432272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":8261933480,"num_objects":126288,"num_object_clones":0,"num_object_copies":378864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126288,"num_whiteouts":0,"num_read":2158,"num_read_kb":1878,"num_write":126972,"num_write_kb":8067757,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":286,"num_bytes_recovered":7506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72576,"ondisk_log_size":72576,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":26966536,"kb_used_data":24001284,"kb_used_omap":13,"kb_used_meta":2965170,"kb_avail":629249528,"statfs":{"total":671965249536,"available":644351516672,"internally_reserved":0,"allocated":24577314816,"data_stored":24573601915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14011,"internal_metadata":3036334405},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,4,2,13,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":590872576,"num_objects":9016,"num_object_clones":0,"num_object_copies":27048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9016,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9016,"num_write_kb":577024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005175"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394357+0000","last_change":"2024-03-26T03:59:12.322291+0000","last_active":"2024-03-26T04:00:54.394357+0000","last_peered":"2024-03-26T04:00:54.394357+0000","last_clean":"2024-03-26T04:00:54.394357+0000","last_became_active":"2024-03-26T03:55:59.299694+0000","last_became_peered":"2024-03-26T03:55:59.299694+0000","last_unstale":"2024-03-26T04:00:54.394357+0000","last_undegraded":"2024-03-26T04:00:54.394357+0000","last_fullsized":"2024-03-26T04:00:54.394357+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.322148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.322148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009207+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":482,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406411+0000","last_change":"2024-03-26T04:00:33.560239+0000","last_active":"2024-03-26T04:00:54.406411+0000","last_peered":"2024-03-26T04:00:54.406411+0000","last_clean":"2024-03-26T04:00:54.406411+0000","last_became_active":"2024-03-26T03:55:59.291176+0000","last_became_peered":"2024-03-26T03:55:59.291176+0000","last_unstale":"2024-03-26T04:00:54.406411+0000","last_undegraded":"2024-03-26T04:00:54.406411+0000","last_fullsized":"2024-03-26T04:00:54.406411+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:33.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.560099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:45.954973+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":493,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397816+0000","last_change":"2024-03-26T04:00:07.415630+0000","last_active":"2024-03-26T04:00:54.397816+0000","last_peered":"2024-03-26T04:00:54.397816+0000","last_clean":"2024-03-26T04:00:54.397816+0000","last_became_active":"2024-03-26T03:55:59.307451+0000","last_became_peered":"2024-03-26T03:55:59.307451+0000","last_unstale":"2024-03-26T04:00:54.397816+0000","last_undegraded":"2024-03-26T04:00:54.397816+0000","last_fullsized":"2024-03-26T04:00:54.397816+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:07.415513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:07.415513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:09.932147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":563,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393426+0000","last_change":"2024-03-26T03:57:49.470002+0000","last_active":"2024-03-26T04:00:54.393426+0000","last_peered":"2024-03-26T04:00:54.393426+0000","last_clean":"2024-03-26T04:00:54.393426+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T04:00:54.393426+0000","last_undegraded":"2024-03-26T04:00:54.393426+0000","last_fullsized":"2024-03-26T04:00:54.393426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-03-26T03:55:32.246145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:32.246145+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009124+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401495+0000","last_change":"2024-03-26T04:00:26.733786+0000","last_active":"2024-03-26T04:00:54.401495+0000","last_peered":"2024-03-26T04:00:54.401495+0000","last_clean":"2024-03-26T04:00:54.401495+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T04:00:54.401495+0000","last_undegraded":"2024-03-26T04:00:54.401495+0000","last_fullsized":"2024-03-26T04:00:54.401495+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.733284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.733284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:36.395678+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":473,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393220+0000","last_change":"2024-03-26T03:55:58.318481+0000","last_active":"2024-03-26T04:00:54.393220+0000","last_peered":"2024-03-26T04:00:54.393220+0000","last_clean":"2024-03-26T04:00:54.393220+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T04:00:54.393220+0000","last_undegraded":"2024-03-26T04:00:54.393220+0000","last_fullsized":"2024-03-26T04:00:54.393220+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:58.318368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:55:58.318368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009152+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.400192+0000","last_change":"2024-03-26T04:00:32.445510+0000","last_active":"2024-03-26T04:00:54.400192+0000","last_peered":"2024-03-26T04:00:54.400192+0000","last_clean":"2024-03-26T04:00:54.400192+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T04:00:54.400192+0000","last_undegraded":"2024-03-26T04:00:54.400192+0000","last_fullsized":"2024-03-26T04:00:54.400192+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:00:32.445390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:43.841897+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":199,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398676+0000","last_change":"2024-03-26T03:59:19.277977+0000","last_active":"2024-03-26T04:00:54.398676+0000","last_peered":"2024-03-26T04:00:54.398676+0000","last_clean":"2024-03-26T04:00:54.398676+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T04:00:54.398676+0000","last_undegraded":"2024-03-26T04:00:54.398676+0000","last_fullsized":"2024-03-26T04:00:54.398676+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T03:59:19.277872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.277872+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404254+0000","last_change":"2024-03-26T04:00:26.435067+0000","last_active":"2024-03-26T04:00:54.404254+0000","last_peered":"2024-03-26T04:00:54.404254+0000","last_clean":"2024-03-26T04:00:54.404254+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:00:54.404254+0000","last_undegraded":"2024-03-26T04:00:54.404254+0000","last_fullsized":"2024-03-26T04:00:54.404254+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.434899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.434899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:44.138049+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":465,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405461+0000","last_change":"2024-03-26T04:00:25.328679+0000","last_active":"2024-03-26T04:00:54.405461+0000","last_peered":"2024-03-26T04:00:54.405461+0000","last_clean":"2024-03-26T04:00:54.405461+0000","last_became_active":"2024-03-26T03:55:59.309490+0000","last_became_peered":"2024-03-26T03:55:59.309490+0000","last_unstale":"2024-03-26T04:00:54.405461+0000","last_undegraded":"2024-03-26T04:00:54.405461+0000","last_fullsized":"2024-03-26T04:00:54.405461+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:00:25.328566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:39.892563+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":415,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.415244+0000","last_change":"2024-03-26T03:59:20.650716+0000","last_active":"2024-03-26T04:00:54.415244+0000","last_peered":"2024-03-26T04:00:54.415244+0000","last_clean":"2024-03-26T04:00:54.415244+0000","last_became_active":"2024-03-26T03:55:59.297862+0000","last_became_peered":"2024-03-26T03:55:59.297862+0000","last_unstale":"2024-03-26T04:00:54.415244+0000","last_undegraded":"2024-03-26T04:00:54.415244+0000","last_fullsized":"2024-03-26T04:00:54.415244+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.650577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.650577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.170022+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":569,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394657+0000","last_change":"2024-03-26T03:59:21.358672+0000","last_active":"2024-03-26T04:00:54.394657+0000","last_peered":"2024-03-26T04:00:54.394657+0000","last_clean":"2024-03-26T04:00:54.394657+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T04:00:54.394657+0000","last_undegraded":"2024-03-26T04:00:54.394657+0000","last_fullsized":"2024-03-26T04:00:54.394657+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T03:59:21.358504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.358504+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":439,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405633+0000","last_change":"2024-03-26T04:00:17.396986+0000","last_active":"2024-03-26T04:00:54.405633+0000","last_peered":"2024-03-26T04:00:54.405633+0000","last_clean":"2024-03-26T04:00:54.405633+0000","last_became_active":"2024-03-26T03:57:49.380178+0000","last_became_peered":"2024-03-26T03:57:49.380178+0000","last_unstale":"2024-03-26T04:00:54.405633+0000","last_undegraded":"2024-03-26T04:00:54.405633+0000","last_fullsized":"2024-03-26T04:00:54.405633+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.396869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.396869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:24.162537+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":483,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406032+0000","last_change":"2024-03-26T04:00:10.992186+0000","last_active":"2024-03-26T04:00:54.406032+0000","last_peered":"2024-03-26T04:00:54.406032+0000","last_clean":"2024-03-26T04:00:54.406032+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:00:54.406032+0000","last_undegraded":"2024-03-26T04:00:54.406032+0000","last_fullsized":"2024-03-26T04:00:54.406032+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.991964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.991964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:24.861983+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":440,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405559+0000","last_change":"2024-03-26T04:00:18.326741+0000","last_active":"2024-03-26T04:00:54.405559+0000","last_peered":"2024-03-26T04:00:54.405559+0000","last_clean":"2024-03-26T04:00:54.405559+0000","last_became_active":"2024-03-26T03:55:59.294899+0000","last_became_peered":"2024-03-26T03:55:59.294899+0000","last_unstale":"2024-03-26T04:00:54.405559+0000","last_undegraded":"2024-03-26T04:00:54.405559+0000","last_fullsized":"2024-03-26T04:00:54.405559+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:18.326616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.326616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:34.001664+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":527,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394716+0000","last_change":"2024-03-26T03:57:49.485971+0000","last_active":"2024-03-26T04:00:54.394716+0000","last_peered":"2024-03-26T04:00:54.394716+0000","last_clean":"2024-03-26T04:00:54.394716+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T04:00:54.394716+0000","last_undegraded":"2024-03-26T04:00:54.394716+0000","last_fullsized":"2024-03-26T04:00:54.394716+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14","last_scrub_stamp":"2024-03-26T03:55:18.237800+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T03:55:18.237800+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009027+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395187+0000","last_change":"2024-03-26T03:59:11.325209+0000","last_active":"2024-03-26T04:00:54.395187+0000","last_peered":"2024-03-26T04:00:54.395187+0000","last_clean":"2024-03-26T04:00:54.395187+0000","last_became_active":"2024-03-26T03:57:49.360508+0000","last_became_peered":"2024-03-26T03:57:49.360508+0000","last_unstale":"2024-03-26T04:00:54.395187+0000","last_undegraded":"2024-03-26T04:00:54.395187+0000","last_fullsized":"2024-03-26T04:00:54.395187+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:11.325089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:11.325089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009234+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":473,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406847+0000","last_change":"2024-03-26T04:00:10.517958+0000","last_active":"2024-03-26T04:00:54.406847+0000","last_peered":"2024-03-26T04:00:54.406847+0000","last_clean":"2024-03-26T04:00:54.406847+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:00:54.406847+0000","last_undegraded":"2024-03-26T04:00:54.406847+0000","last_fullsized":"2024-03-26T04:00:54.406847+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:14.940739+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":491,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398167+0000","last_change":"2024-03-26T04:00:27.363899+0000","last_active":"2024-03-26T04:00:54.398167+0000","last_peered":"2024-03-26T04:00:54.398167+0000","last_clean":"2024-03-26T04:00:54.398167+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:00:54.398167+0000","last_undegraded":"2024-03-26T04:00:54.398167+0000","last_fullsized":"2024-03-26T04:00:54.398167+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.363567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.363567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:30.587040+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":590,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402600+0000","last_change":"2024-03-26T04:00:12.045147+0000","last_active":"2024-03-26T04:00:54.402600+0000","last_peered":"2024-03-26T04:00:54.402600+0000","last_clean":"2024-03-26T04:00:54.402600+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T04:00:54.402600+0000","last_undegraded":"2024-03-26T04:00:54.402600+0000","last_fullsized":"2024-03-26T04:00:54.402600+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:12.044640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:12.044640+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:39.921406+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402062+0000","last_change":"2024-03-26T03:59:22.373594+0000","last_active":"2024-03-26T04:00:54.402062+0000","last_peered":"2024-03-26T04:00:54.402062+0000","last_clean":"2024-03-26T04:00:54.402062+0000","last_became_active":"2024-03-26T03:57:49.376975+0000","last_became_peered":"2024-03-26T03:57:49.376975+0000","last_unstale":"2024-03-26T04:00:54.402062+0000","last_undegraded":"2024-03-26T04:00:54.402062+0000","last_fullsized":"2024-03-26T04:00:54.402062+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.373494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.373494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":85,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":487,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395702+0000","last_change":"2024-03-26T04:00:26.341422+0000","last_active":"2024-03-26T04:00:54.395702+0000","last_peered":"2024-03-26T04:00:54.395702+0000","last_clean":"2024-03-26T04:00:54.395702+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T04:00:54.395702+0000","last_undegraded":"2024-03-26T04:00:54.395702+0000","last_fullsized":"2024-03-26T04:00:54.395702+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.341297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.341297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:32.381250+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":483,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394786+0000","last_change":"2024-03-26T03:59:23.332947+0000","last_active":"2024-03-26T04:00:54.394786+0000","last_peered":"2024-03-26T04:00:54.394786+0000","last_clean":"2024-03-26T04:00:54.394786+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T04:00:54.394786+0000","last_undegraded":"2024-03-26T04:00:54.394786+0000","last_fullsized":"2024-03-26T04:00:54.394786+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:23.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":180,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.400469+0000","last_change":"2024-03-26T03:59:13.198844+0000","last_active":"2024-03-26T04:00:54.400469+0000","last_peered":"2024-03-26T04:00:54.400469+0000","last_clean":"2024-03-26T04:00:54.400469+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T04:00:54.400469+0000","last_undegraded":"2024-03-26T04:00:54.400469+0000","last_fullsized":"2024-03-26T04:00:54.400469+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T03:59:13.198735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.198735+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":158,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406542+0000","last_change":"2024-03-26T04:00:05.475301+0000","last_active":"2024-03-26T04:00:54.406542+0000","last_peered":"2024-03-26T04:00:54.406542+0000","last_clean":"2024-03-26T04:00:54.406542+0000","last_became_active":"2024-03-26T03:58:40.496528+0000","last_became_peered":"2024-03-26T03:58:40.496528+0000","last_unstale":"2024-03-26T04:00:54.406542+0000","last_undegraded":"2024-03-26T04:00:54.406542+0000","last_fullsized":"2024-03-26T04:00:54.406542+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:00:05.475152+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:00:05.475152+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:31.911000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":571,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394992+0000","last_change":"2024-03-26T04:00:35.094959+0000","last_active":"2024-03-26T04:00:54.394992+0000","last_peered":"2024-03-26T04:00:54.394992+0000","last_clean":"2024-03-26T04:00:54.394992+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T04:00:54.394992+0000","last_undegraded":"2024-03-26T04:00:54.394992+0000","last_fullsized":"2024-03-26T04:00:54.394992+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:35.094793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.094793+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:42.444909+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":504,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406357+0000","last_change":"2024-03-26T03:59:26.325088+0000","last_active":"2024-03-26T04:00:54.406357+0000","last_peered":"2024-03-26T04:00:54.406357+0000","last_clean":"2024-03-26T04:00:54.406357+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T04:00:54.406357+0000","last_undegraded":"2024-03-26T04:00:54.406357+0000","last_fullsized":"2024-03-26T04:00:54.406357+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.324998+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"241'26","reported_seq":563,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398057+0000","last_change":"2024-03-26T04:00:11.288905+0000","last_active":"2024-03-26T04:00:54.398057+0000","last_peered":"2024-03-26T04:00:54.398057+0000","last_clean":"2024-03-26T04:00:54.398057+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T04:00:54.398057+0000","last_undegraded":"2024-03-26T04:00:54.398057+0000","last_fullsized":"2024-03-26T04:00:54.398057+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:00:11.288774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.288774+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:37.528884+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":478,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406463+0000","last_change":"2024-03-26T04:00:21.634056+0000","last_active":"2024-03-26T04:00:54.406463+0000","last_peered":"2024-03-26T04:00:54.406463+0000","last_clean":"2024-03-26T04:00:54.406463+0000","last_became_active":"2024-03-26T03:55:59.343310+0000","last_became_peered":"2024-03-26T03:55:59.343310+0000","last_unstale":"2024-03-26T04:00:54.406463+0000","last_undegraded":"2024-03-26T04:00:54.406463+0000","last_fullsized":"2024-03-26T04:00:54.406463+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:21.633920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.633920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.986309+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":477,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399625+0000","last_change":"2024-03-26T04:00:10.317133+0000","last_active":"2024-03-26T04:00:54.399625+0000","last_peered":"2024-03-26T04:00:54.399625+0000","last_clean":"2024-03-26T04:00:54.399625+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T04:00:54.399625+0000","last_undegraded":"2024-03-26T04:00:54.399625+0000","last_fullsized":"2024-03-26T04:00:54.399625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.317007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.317007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:15.935633+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397818+0000","last_change":"2024-03-26T04:00:12.417129+0000","last_active":"2024-03-26T04:00:54.397818+0000","last_peered":"2024-03-26T04:00:54.397818+0000","last_clean":"2024-03-26T04:00:54.397818+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T04:00:54.397818+0000","last_undegraded":"2024-03-26T04:00:54.397818+0000","last_fullsized":"2024-03-26T04:00:54.397818+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:12.416999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:12.416999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:36.793912+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407490+0000","last_change":"2024-03-26T04:00:20.349063+0000","last_active":"2024-03-26T04:00:54.407490+0000","last_peered":"2024-03-26T04:00:54.407490+0000","last_clean":"2024-03-26T04:00:54.407490+0000","last_became_active":"2024-03-26T03:55:59.304531+0000","last_became_peered":"2024-03-26T03:55:59.304531+0000","last_unstale":"2024-03-26T04:00:54.407490+0000","last_undegraded":"2024-03-26T04:00:54.407490+0000","last_fullsized":"2024-03-26T04:00:54.407490+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:20.348805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:20.348805+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:21.025907+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":40,"num_read_kb":32,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406566+0000","last_change":"2024-03-26T04:00:19.993808+0000","last_active":"2024-03-26T04:00:54.406566+0000","last_peered":"2024-03-26T04:00:54.406566+0000","last_clean":"2024-03-26T04:00:54.406566+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T04:00:54.406566+0000","last_undegraded":"2024-03-26T04:00:54.406566+0000","last_fullsized":"2024-03-26T04:00:54.406566+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.993684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.993684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:25.121383+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":414,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393538+0000","last_change":"2024-03-26T04:00:23.343567+0000","last_active":"2024-03-26T04:00:54.393538+0000","last_peered":"2024-03-26T04:00:54.393538+0000","last_clean":"2024-03-26T04:00:54.393538+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:00:54.393538+0000","last_undegraded":"2024-03-26T04:00:54.393538+0000","last_fullsized":"2024-03-26T04:00:54.393538+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:23.343421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.343421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:49.204689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":505,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398047+0000","last_change":"2024-03-26T04:00:16.271711+0000","last_active":"2024-03-26T04:00:54.398047+0000","last_peered":"2024-03-26T04:00:54.398047+0000","last_clean":"2024-03-26T04:00:54.398047+0000","last_became_active":"2024-03-26T03:57:49.384847+0000","last_became_peered":"2024-03-26T03:57:49.384847+0000","last_unstale":"2024-03-26T04:00:54.398047+0000","last_undegraded":"2024-03-26T04:00:54.398047+0000","last_fullsized":"2024-03-26T04:00:54.398047+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:16.271586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:16.271586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:38.846915+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":177,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397716+0000","last_change":"2024-03-26T04:00:38.478725+0000","last_active":"2024-03-26T04:00:54.397716+0000","last_peered":"2024-03-26T04:00:54.397716+0000","last_clean":"2024-03-26T04:00:54.397716+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T04:00:54.397716+0000","last_undegraded":"2024-03-26T04:00:54.397716+0000","last_fullsized":"2024-03-26T04:00:54.397716+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:00:38.458453+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.458453+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:44.586749+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401406+0000","last_change":"2024-03-26T04:00:29.635294+0000","last_active":"2024-03-26T04:00:54.401406+0000","last_peered":"2024-03-26T04:00:54.401406+0000","last_clean":"2024-03-26T04:00:54.401406+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T04:00:54.401406+0000","last_undegraded":"2024-03-26T04:00:54.401406+0000","last_fullsized":"2024-03-26T04:00:54.401406+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.635002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.635002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:45.327684+0000","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":490,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405472+0000","last_change":"2024-03-26T04:00:18.946606+0000","last_active":"2024-03-26T04:00:54.405472+0000","last_peered":"2024-03-26T04:00:54.405472+0000","last_clean":"2024-03-26T04:00:54.405472+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:00:54.405472+0000","last_undegraded":"2024-03-26T04:00:54.405472+0000","last_fullsized":"2024-03-26T04:00:54.405472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:18.946511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.946511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:27.205054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":448,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407029+0000","last_change":"2024-03-26T04:00:19.384672+0000","last_active":"2024-03-26T04:00:54.407029+0000","last_peered":"2024-03-26T04:00:54.407029+0000","last_clean":"2024-03-26T04:00:54.407029+0000","last_became_active":"2024-03-26T03:55:59.304181+0000","last_became_peered":"2024-03-26T03:55:59.304181+0000","last_unstale":"2024-03-26T04:00:54.407029+0000","last_undegraded":"2024-03-26T04:00:54.407029+0000","last_fullsized":"2024-03-26T04:00:54.407029+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.384487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.384487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:25.404691+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":183,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401721+0000","last_change":"2024-03-26T03:59:14.247318+0000","last_active":"2024-03-26T04:00:54.401721+0000","last_peered":"2024-03-26T04:00:54.401721+0000","last_clean":"2024-03-26T04:00:54.401721+0000","last_became_active":"2024-03-26T03:57:49.373634+0000","last_became_peered":"2024-03-26T03:57:49.373634+0000","last_unstale":"2024-03-26T04:00:54.401721+0000","last_undegraded":"2024-03-26T04:00:54.401721+0000","last_fullsized":"2024-03-26T04:00:54.401721+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:14.247185+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.247185+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":481,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399144+0000","last_change":"2024-03-26T04:00:09.430654+0000","last_active":"2024-03-26T04:00:54.399144+0000","last_peered":"2024-03-26T04:00:54.399144+0000","last_clean":"2024-03-26T04:00:54.399144+0000","last_became_active":"2024-03-26T03:55:59.323909+0000","last_became_peered":"2024-03-26T03:55:59.323909+0000","last_unstale":"2024-03-26T04:00:54.399144+0000","last_undegraded":"2024-03-26T04:00:54.399144+0000","last_fullsized":"2024-03-26T04:00:54.399144+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:09.430539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:09.430539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:27.699407+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401320+0000","last_change":"2024-03-26T04:00:30.491018+0000","last_active":"2024-03-26T04:00:54.401320+0000","last_peered":"2024-03-26T04:00:54.401320+0000","last_clean":"2024-03-26T04:00:54.401320+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T04:00:54.401320+0000","last_undegraded":"2024-03-26T04:00:54.401320+0000","last_fullsized":"2024-03-26T04:00:54.401320+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:30.490887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:30.490887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.974095+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":504,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407285+0000","last_change":"2024-03-26T04:00:14.987182+0000","last_active":"2024-03-26T04:00:54.407285+0000","last_peered":"2024-03-26T04:00:54.407285+0000","last_clean":"2024-03-26T04:00:54.407285+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T04:00:54.407285+0000","last_undegraded":"2024-03-26T04:00:54.407285+0000","last_fullsized":"2024-03-26T04:00:54.407285+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:14.987072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:14.987072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:33.039675+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":531,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.409405+0000","last_change":"2024-03-26T03:59:27.366321+0000","last_active":"2024-03-26T04:00:54.409405+0000","last_peered":"2024-03-26T04:00:54.409405+0000","last_clean":"2024-03-26T04:00:54.409405+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T04:00:54.409405+0000","last_undegraded":"2024-03-26T04:00:54.409405+0000","last_fullsized":"2024-03-26T04:00:54.409405+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:27.366230+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.366230+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398577+0000","last_change":"2024-03-26T03:59:28.149466+0000","last_active":"2024-03-26T04:00:54.398577+0000","last_peered":"2024-03-26T04:00:54.398577+0000","last_clean":"2024-03-26T04:00:54.398577+0000","last_became_active":"2024-03-26T03:57:49.370042+0000","last_became_peered":"2024-03-26T03:57:49.370042+0000","last_unstale":"2024-03-26T04:00:54.398577+0000","last_undegraded":"2024-03-26T04:00:54.398577+0000","last_fullsized":"2024-03-26T04:00:54.398577+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.149370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.149370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":495,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.409215+0000","last_change":"2024-03-26T04:00:19.560501+0000","last_active":"2024-03-26T04:00:54.409215+0000","last_peered":"2024-03-26T04:00:54.409215+0000","last_clean":"2024-03-26T04:00:54.409215+0000","last_became_active":"2024-03-26T03:57:49.379985+0000","last_became_peered":"2024-03-26T03:57:49.379985+0000","last_unstale":"2024-03-26T04:00:54.409215+0000","last_undegraded":"2024-03-26T04:00:54.409215+0000","last_fullsized":"2024-03-26T04:00:54.409215+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.560382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.560382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:43.631659+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":490,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395566+0000","last_change":"2024-03-26T03:59:24.336092+0000","last_active":"2024-03-26T04:00:54.395566+0000","last_peered":"2024-03-26T04:00:54.395566+0000","last_clean":"2024-03-26T04:00:54.395566+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T04:00:54.395566+0000","last_undegraded":"2024-03-26T04:00:54.395566+0000","last_fullsized":"2024-03-26T04:00:54.395566+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:24.336010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:24.336010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009291+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":530,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394936+0000","last_change":"2024-03-26T03:59:18.367293+0000","last_active":"2024-03-26T04:00:54.394936+0000","last_peered":"2024-03-26T04:00:54.394936+0000","last_clean":"2024-03-26T04:00:54.394936+0000","last_became_active":"2024-03-26T03:55:59.299354+0000","last_became_peered":"2024-03-26T03:55:59.299354+0000","last_unstale":"2024-03-26T04:00:54.394936+0000","last_undegraded":"2024-03-26T04:00:54.394936+0000","last_fullsized":"2024-03-26T04:00:54.394936+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:18.367180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.367180+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":476,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397755+0000","last_change":"2024-03-26T03:59:17.820332+0000","last_active":"2024-03-26T04:00:54.397755+0000","last_peered":"2024-03-26T04:00:54.397755+0000","last_clean":"2024-03-26T04:00:54.397755+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T04:00:54.397755+0000","last_undegraded":"2024-03-26T04:00:54.397755+0000","last_fullsized":"2024-03-26T04:00:54.397755+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:17.820216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.820216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.437971+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"181'10","reported_seq":611,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.414920+0000","last_change":"2024-03-26T04:00:23.341544+0000","last_active":"2024-03-26T04:00:54.414920+0000","last_peered":"2024-03-26T04:00:54.414920+0000","last_clean":"2024-03-26T04:00:54.414920+0000","last_became_active":"2024-03-26T03:55:59.340338+0000","last_became_peered":"2024-03-26T03:55:59.340338+0000","last_unstale":"2024-03-26T04:00:54.414920+0000","last_undegraded":"2024-03-26T04:00:54.414920+0000","last_fullsized":"2024-03-26T04:00:54.414920+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:23.341352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.341352+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:33.414175+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":500,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406265+0000","last_change":"2024-03-26T04:00:36.999119+0000","last_active":"2024-03-26T04:00:54.406265+0000","last_peered":"2024-03-26T04:00:54.406265+0000","last_clean":"2024-03-26T04:00:54.406265+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T04:00:54.406265+0000","last_undegraded":"2024-03-26T04:00:54.406265+0000","last_fullsized":"2024-03-26T04:00:54.406265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:36.998999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.998999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:00.860008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":188,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398551+0000","last_change":"2024-03-26T04:00:17.439081+0000","last_active":"2024-03-26T04:00:54.398551+0000","last_peered":"2024-03-26T04:00:54.398551+0000","last_clean":"2024-03-26T04:00:54.398551+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T04:00:54.398551+0000","last_undegraded":"2024-03-26T04:00:54.398551+0000","last_fullsized":"2024-03-26T04:00:54.398551+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-03-26T04:00:17.438826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.438826+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:45.150013+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":480,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395099+0000","last_change":"2024-03-26T04:00:27.428392+0000","last_active":"2024-03-26T04:00:54.395099+0000","last_peered":"2024-03-26T04:00:54.395099+0000","last_clean":"2024-03-26T04:00:54.395099+0000","last_became_active":"2024-03-26T03:55:59.298661+0000","last_became_peered":"2024-03-26T03:55:59.298661+0000","last_unstale":"2024-03-26T04:00:54.395099+0000","last_undegraded":"2024-03-26T04:00:54.395099+0000","last_fullsized":"2024-03-26T04:00:54.395099+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.428232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.428232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:28.981979+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":199,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398338+0000","last_change":"2024-03-26T04:00:20.443100+0000","last_active":"2024-03-26T04:00:54.398338+0000","last_peered":"2024-03-26T04:00:54.398338+0000","last_clean":"2024-03-26T04:00:54.398338+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T04:00:54.398338+0000","last_undegraded":"2024-03-26T04:00:54.398338+0000","last_fullsized":"2024-03-26T04:00:54.398338+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'10","last_scrub_stamp":"2024-03-26T04:00:20.442965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:20.442965+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:33.722836+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":522,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404923+0000","last_change":"2024-03-26T04:00:31.362526+0000","last_active":"2024-03-26T04:00:54.404923+0000","last_peered":"2024-03-26T04:00:54.404923+0000","last_clean":"2024-03-26T04:00:54.404923+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T04:00:54.404923+0000","last_undegraded":"2024-03-26T04:00:54.404923+0000","last_fullsized":"2024-03-26T04:00:54.404923+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:00:31.362386+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:31.362386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:44.043312+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":523,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.415113+0000","last_change":"2024-03-26T04:00:28.238199+0000","last_active":"2024-03-26T04:00:54.415113+0000","last_peered":"2024-03-26T04:00:54.415113+0000","last_clean":"2024-03-26T04:00:54.415113+0000","last_became_active":"2024-03-26T03:57:49.382991+0000","last_became_peered":"2024-03-26T03:57:49.382991+0000","last_unstale":"2024-03-26T04:00:54.415113+0000","last_undegraded":"2024-03-26T04:00:54.415113+0000","last_fullsized":"2024-03-26T04:00:54.415113+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:00:28.238100+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:00:28.238100+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:42.837610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":490,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402957+0000","last_change":"2024-03-26T03:59:17.399536+0000","last_active":"2024-03-26T04:00:54.402957+0000","last_peered":"2024-03-26T04:00:54.402957+0000","last_clean":"2024-03-26T04:00:54.402957+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T04:00:54.402957+0000","last_undegraded":"2024-03-26T04:00:54.402957+0000","last_fullsized":"2024-03-26T04:00:54.402957+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T03:59:17.399444+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T03:59:17.399444+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.168813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402556+0000","last_change":"2024-03-26T04:00:22.523367+0000","last_active":"2024-03-26T04:00:54.402556+0000","last_peered":"2024-03-26T04:00:54.402556+0000","last_clean":"2024-03-26T04:00:54.402556+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T04:00:54.402556+0000","last_undegraded":"2024-03-26T04:00:54.402556+0000","last_fullsized":"2024-03-26T04:00:54.402556+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.523245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.523245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.313797+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":184,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393693+0000","last_change":"2024-03-26T03:58:40.497439+0000","last_active":"2024-03-26T04:00:54.393693+0000","last_peered":"2024-03-26T04:00:54.393693+0000","last_clean":"2024-03-26T04:00:54.393693+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:00:54.393693+0000","last_undegraded":"2024-03-26T04:00:54.393693+0000","last_fullsized":"2024-03-26T04:00:54.393693+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:58:15.134741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:58:15.134741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":569,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398151+0000","last_change":"2024-03-26T04:00:25.344041+0000","last_active":"2024-03-26T04:00:54.398151+0000","last_peered":"2024-03-26T04:00:54.398151+0000","last_clean":"2024-03-26T04:00:54.398151+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T04:00:54.398151+0000","last_undegraded":"2024-03-26T04:00:54.398151+0000","last_fullsized":"2024-03-26T04:00:54.398151+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:25.343896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:25.343896+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:52.529323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":487,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407200+0000","last_change":"2024-03-26T03:59:28.005851+0000","last_active":"2024-03-26T04:00:54.407200+0000","last_peered":"2024-03-26T04:00:54.407200+0000","last_clean":"2024-03-26T04:00:54.407200+0000","last_became_active":"2024-03-26T03:55:59.299873+0000","last_became_peered":"2024-03-26T03:55:59.299873+0000","last_unstale":"2024-03-26T04:00:54.407200+0000","last_undegraded":"2024-03-26T04:00:54.407200+0000","last_fullsized":"2024-03-26T04:00:54.407200+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:28.005705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:28.005705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.978808+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":445,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407200+0000","last_change":"2024-03-26T03:59:15.662289+0000","last_active":"2024-03-26T04:00:54.407200+0000","last_peered":"2024-03-26T04:00:54.407200+0000","last_clean":"2024-03-26T04:00:54.407200+0000","last_became_active":"2024-03-26T03:57:49.380167+0000","last_became_peered":"2024-03-26T03:57:49.380167+0000","last_unstale":"2024-03-26T04:00:54.407200+0000","last_undegraded":"2024-03-26T04:00:54.407200+0000","last_fullsized":"2024-03-26T04:00:54.407200+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:15.662211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.662211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.171742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":435,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399505+0000","last_change":"2024-03-26T03:59:20.739789+0000","last_active":"2024-03-26T04:00:54.399505+0000","last_peered":"2024-03-26T04:00:54.399505+0000","last_clean":"2024-03-26T04:00:54.399505+0000","last_became_active":"2024-03-26T03:57:49.377869+0000","last_became_peered":"2024-03-26T03:57:49.377869+0000","last_unstale":"2024-03-26T04:00:54.399505+0000","last_undegraded":"2024-03-26T04:00:54.399505+0000","last_fullsized":"2024-03-26T04:00:54.399505+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":176,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402153+0000","last_change":"2024-03-26T04:00:21.475401+0000","last_active":"2024-03-26T04:00:54.402153+0000","last_peered":"2024-03-26T04:00:54.402153+0000","last_clean":"2024-03-26T04:00:54.402153+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T04:00:54.402153+0000","last_undegraded":"2024-03-26T04:00:54.402153+0000","last_fullsized":"2024-03-26T04:00:54.402153+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:00:21.475271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.475271+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:44.087028+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"284'7854","reported_seq":7926,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.305305+0000","last_change":"2024-03-26T03:59:23.518905+0000","last_active":"2024-03-26T04:00:55.305305+0000","last_peered":"2024-03-26T04:00:55.305305+0000","last_clean":"2024-03-26T04:00:55.305305+0000","last_became_active":"2024-03-26T03:59:23.518616+0000","last_became_peered":"2024-03-26T03:59:23.518616+0000","last_unstale":"2024-03-26T04:00:55.305305+0000","last_undegraded":"2024-03-26T04:00:55.305305+0000","last_fullsized":"2024-03-26T04:00:55.305305+0000","mapping_epoch":253,"log_start":"260'3058","ondisk_log_start":"260'3058","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4796,"log_dups_size":0,"ondisk_log_size":4796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514719744,"num_objects":7854,"num_object_clones":0,"num_object_copies":23562,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7854,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7854,"num_write_kb":502656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":432,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405887+0000","last_change":"2024-03-26T03:59:31.507063+0000","last_active":"2024-03-26T04:00:54.405887+0000","last_peered":"2024-03-26T04:00:54.405887+0000","last_clean":"2024-03-26T04:00:54.405887+0000","last_became_active":"2024-03-26T03:55:59.312878+0000","last_became_peered":"2024-03-26T03:55:59.312878+0000","last_unstale":"2024-03-26T04:00:54.405887+0000","last_undegraded":"2024-03-26T04:00:54.405887+0000","last_fullsized":"2024-03-26T04:00:54.405887+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:31.506978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:31.506978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.951142+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":504,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.409849+0000","last_change":"2024-03-26T04:00:17.997376+0000","last_active":"2024-03-26T04:00:54.409849+0000","last_peered":"2024-03-26T04:00:54.409849+0000","last_clean":"2024-03-26T04:00:54.409849+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:00:54.409849+0000","last_undegraded":"2024-03-26T04:00:54.409849+0000","last_fullsized":"2024-03-26T04:00:54.409849+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T04:00:17.997292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.997292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:35.423128+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":523,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405004+0000","last_change":"2024-03-26T04:00:11.391330+0000","last_active":"2024-03-26T04:00:54.405004+0000","last_peered":"2024-03-26T04:00:54.405004+0000","last_clean":"2024-03-26T04:00:54.405004+0000","last_became_active":"2024-03-26T03:55:59.292636+0000","last_became_peered":"2024-03-26T03:55:59.292636+0000","last_unstale":"2024-03-26T04:00:54.405004+0000","last_undegraded":"2024-03-26T04:00:54.405004+0000","last_fullsized":"2024-03-26T04:00:54.405004+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:00:11.391188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.391188+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:40.556279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":62,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":658,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406401+0000","last_change":"2024-03-26T04:00:34.970757+0000","last_active":"2024-03-26T04:00:54.406401+0000","last_peered":"2024-03-26T04:00:54.406401+0000","last_clean":"2024-03-26T04:00:54.406401+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:00:54.406401+0000","last_undegraded":"2024-03-26T04:00:54.406401+0000","last_fullsized":"2024-03-26T04:00:54.406401+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:34.970608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.970608+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:56.195694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"284'7696","reported_seq":7768,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.462279+0000","last_change":"2024-03-26T03:59:23.931890+0000","last_active":"2024-03-26T04:00:55.462279+0000","last_peered":"2024-03-26T04:00:55.462279+0000","last_clean":"2024-03-26T04:00:55.462279+0000","last_became_active":"2024-03-26T03:59:23.931611+0000","last_became_peered":"2024-03-26T03:59:23.931611+0000","last_unstale":"2024-03-26T04:00:55.462279+0000","last_undegraded":"2024-03-26T04:00:55.462279+0000","last_fullsized":"2024-03-26T04:00:55.462279+0000","mapping_epoch":253,"log_start":"262'3610","ondisk_log_start":"262'3610","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4086,"log_dups_size":0,"ondisk_log_size":4086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.476368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":504365056,"num_objects":7696,"num_object_clones":0,"num_object_copies":23088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7696,"num_write_kb":492544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401227+0000","last_change":"2024-03-26T03:59:30.159583+0000","last_active":"2024-03-26T04:00:54.401227+0000","last_peered":"2024-03-26T04:00:54.401227+0000","last_clean":"2024-03-26T04:00:54.401227+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T04:00:54.401227+0000","last_undegraded":"2024-03-26T04:00:54.401227+0000","last_fullsized":"2024-03-26T04:00:54.401227+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.159481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.159481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.562230+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407815+0000","last_change":"2024-03-26T04:00:13.964333+0000","last_active":"2024-03-26T04:00:54.407815+0000","last_peered":"2024-03-26T04:00:54.407815+0000","last_clean":"2024-03-26T04:00:54.407815+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:00:54.407815+0000","last_undegraded":"2024-03-26T04:00:54.407815+0000","last_fullsized":"2024-03-26T04:00:54.407815+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:13.964232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:13.964232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:35.010968+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":580,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393798+0000","last_change":"2024-03-26T04:00:36.093635+0000","last_active":"2024-03-26T04:00:54.393798+0000","last_peered":"2024-03-26T04:00:54.393798+0000","last_clean":"2024-03-26T04:00:54.393798+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T04:00:54.393798+0000","last_undegraded":"2024-03-26T04:00:54.393798+0000","last_fullsized":"2024-03-26T04:00:54.393798+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:00:36.093314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.093314+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.980008+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":492,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401289+0000","last_change":"2024-03-26T03:59:22.387740+0000","last_active":"2024-03-26T04:00:54.401289+0000","last_peered":"2024-03-26T04:00:54.401289+0000","last_clean":"2024-03-26T04:00:54.401289+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T04:00:54.401289+0000","last_undegraded":"2024-03-26T04:00:54.401289+0000","last_fullsized":"2024-03-26T04:00:54.401289+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.387664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.387664+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.221510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"284'8169","reported_seq":8241,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:58.358387+0000","last_change":"2024-03-26T03:59:23.521792+0000","last_active":"2024-03-26T04:00:58.358387+0000","last_peered":"2024-03-26T04:00:58.358387+0000","last_clean":"2024-03-26T04:00:58.358387+0000","last_became_active":"2024-03-26T03:59:23.521399+0000","last_became_peered":"2024-03-26T03:59:23.521399+0000","last_unstale":"2024-03-26T04:00:58.358387+0000","last_undegraded":"2024-03-26T04:00:58.358387+0000","last_fullsized":"2024-03-26T04:00:58.358387+0000","mapping_epoch":253,"log_start":"262'3703","ondisk_log_start":"262'3703","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.172252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":535363584,"num_objects":8169,"num_object_clones":0,"num_object_copies":24507,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8169,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8169,"num_write_kb":522816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":443,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401494+0000","last_change":"2024-03-26T03:59:30.465692+0000","last_active":"2024-03-26T04:00:54.401494+0000","last_peered":"2024-03-26T04:00:54.401494+0000","last_clean":"2024-03-26T04:00:54.401494+0000","last_became_active":"2024-03-26T03:58:40.502368+0000","last_became_peered":"2024-03-26T03:58:40.502368+0000","last_unstale":"2024-03-26T04:00:54.401494+0000","last_undegraded":"2024-03-26T04:00:54.401494+0000","last_fullsized":"2024-03-26T04:00:54.401494+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.465607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.465607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.813985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":501,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402523+0000","last_change":"2024-03-26T04:00:26.026709+0000","last_active":"2024-03-26T04:00:54.402523+0000","last_peered":"2024-03-26T04:00:54.402523+0000","last_clean":"2024-03-26T04:00:54.402523+0000","last_became_active":"2024-03-26T03:55:59.304410+0000","last_became_peered":"2024-03-26T03:55:59.304410+0000","last_unstale":"2024-03-26T04:00:54.402523+0000","last_undegraded":"2024-03-26T04:00:54.402523+0000","last_fullsized":"2024-03-26T04:00:54.402523+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.026626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.026626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:30.229285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":469,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.415567+0000","last_change":"2024-03-26T04:00:27.278080+0000","last_active":"2024-03-26T04:00:54.415567+0000","last_peered":"2024-03-26T04:00:54.415567+0000","last_clean":"2024-03-26T04:00:54.415567+0000","last_became_active":"2024-03-26T03:58:40.499811+0000","last_became_peered":"2024-03-26T03:58:40.499811+0000","last_unstale":"2024-03-26T04:00:54.415567+0000","last_undegraded":"2024-03-26T04:00:54.415567+0000","last_fullsized":"2024-03-26T04:00:54.415567+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.277958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.277958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:37.230067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"241'41","reported_seq":538,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.409061+0000","last_change":"2024-03-26T04:00:36.459990+0000","last_active":"2024-03-26T04:00:54.409061+0000","last_peered":"2024-03-26T04:00:54.409061+0000","last_clean":"2024-03-26T04:00:54.409061+0000","last_became_active":"2024-03-26T03:55:59.299155+0000","last_became_peered":"2024-03-26T03:55:59.299155+0000","last_unstale":"2024-03-26T04:00:54.409061+0000","last_undegraded":"2024-03-26T04:00:54.409061+0000","last_fullsized":"2024-03-26T04:00:54.409061+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-26T04:00:36.459640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.459640+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:58.046670+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"284'8122","reported_seq":8194,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:58.339881+0000","last_change":"2024-03-26T03:59:23.932187+0000","last_active":"2024-03-26T04:00:58.339881+0000","last_peered":"2024-03-26T04:00:58.339881+0000","last_clean":"2024-03-26T04:00:58.339881+0000","last_became_active":"2024-03-26T03:59:23.931234+0000","last_became_peered":"2024-03-26T03:59:23.931234+0000","last_unstale":"2024-03-26T04:00:58.339881+0000","last_undegraded":"2024-03-26T04:00:58.339881+0000","last_fullsized":"2024-03-26T04:00:58.339881+0000","mapping_epoch":253,"log_start":"261'3612","ondisk_log_start":"261'3612","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4510,"log_dups_size":0,"ondisk_log_size":4510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.169991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532283392,"num_objects":8122,"num_object_clones":0,"num_object_copies":24366,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8122,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8122,"num_write_kb":519808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402380+0000","last_change":"2024-03-26T04:00:22.410543+0000","last_active":"2024-03-26T04:00:54.402380+0000","last_peered":"2024-03-26T04:00:54.402380+0000","last_clean":"2024-03-26T04:00:54.402380+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T04:00:54.402380+0000","last_undegraded":"2024-03-26T04:00:54.402380+0000","last_fullsized":"2024-03-26T04:00:54.402380+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.410420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.410420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:34.898584+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":504,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402155+0000","last_change":"2024-03-26T03:59:16.377443+0000","last_active":"2024-03-26T04:00:54.402155+0000","last_peered":"2024-03-26T04:00:54.402155+0000","last_clean":"2024-03-26T04:00:54.402155+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T04:00:54.402155+0000","last_undegraded":"2024-03-26T04:00:54.402155+0000","last_fullsized":"2024-03-26T04:00:54.402155+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:16.377329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.377329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.221478+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":467,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405252+0000","last_change":"2024-03-26T04:00:09.385495+0000","last_active":"2024-03-26T04:00:54.405252+0000","last_peered":"2024-03-26T04:00:54.405252+0000","last_clean":"2024-03-26T04:00:54.405252+0000","last_became_active":"2024-03-26T03:57:49.367885+0000","last_became_peered":"2024-03-26T03:57:49.367885+0000","last_unstale":"2024-03-26T04:00:54.405252+0000","last_undegraded":"2024-03-26T04:00:54.405252+0000","last_fullsized":"2024-03-26T04:00:54.405252+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:09.385412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:00:09.385412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:37.715425+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":544,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399838+0000","last_change":"2024-03-26T03:59:22.808667+0000","last_active":"2024-03-26T04:00:54.399838+0000","last_peered":"2024-03-26T04:00:54.399838+0000","last_clean":"2024-03-26T04:00:54.399838+0000","last_became_active":"2024-03-26T03:55:59.307802+0000","last_became_peered":"2024-03-26T03:55:59.307802+0000","last_unstale":"2024-03-26T04:00:54.399838+0000","last_undegraded":"2024-03-26T04:00:54.399838+0000","last_fullsized":"2024-03-26T04:00:54.399838+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T03:59:22.808562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.808562+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.437915+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405155+0000","last_change":"2024-03-26T04:00:34.306630+0000","last_active":"2024-03-26T04:00:54.405155+0000","last_peered":"2024-03-26T04:00:54.405155+0000","last_clean":"2024-03-26T04:00:54.405155+0000","last_became_active":"2024-03-26T04:00:32.429409+0000","last_became_peered":"2024-03-26T04:00:32.429409+0000","last_unstale":"2024-03-26T04:00:54.405155+0000","last_undegraded":"2024-03-26T04:00:54.405155+0000","last_fullsized":"2024-03-26T04:00:54.405155+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.306506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.306506+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:46.443171+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"284'7604","reported_seq":7676,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.254987+0000","last_change":"2024-03-26T03:59:23.520900+0000","last_active":"2024-03-26T04:00:55.254987+0000","last_peered":"2024-03-26T04:00:55.254987+0000","last_clean":"2024-03-26T04:00:55.254987+0000","last_became_active":"2024-03-26T03:59:23.520581+0000","last_became_peered":"2024-03-26T03:59:23.520581+0000","last_unstale":"2024-03-26T04:00:55.254987+0000","last_undegraded":"2024-03-26T04:00:55.254987+0000","last_fullsized":"2024-03-26T04:00:55.254987+0000","mapping_epoch":253,"log_start":"260'2841","ondisk_log_start":"260'2841","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4763,"log_dups_size":0,"ondisk_log_size":4763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498335744,"num_objects":7604,"num_object_clones":0,"num_object_copies":22812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7604,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7604,"num_write_kb":486656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.400095+0000","last_change":"2024-03-26T04:00:19.502445+0000","last_active":"2024-03-26T04:00:54.400095+0000","last_peered":"2024-03-26T04:00:54.400095+0000","last_clean":"2024-03-26T04:00:54.400095+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T04:00:54.400095+0000","last_undegraded":"2024-03-26T04:00:54.400095+0000","last_fullsized":"2024-03-26T04:00:54.400095+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.502323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.502323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:48.282202+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":436,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405775+0000","last_change":"2024-03-26T03:59:27.539611+0000","last_active":"2024-03-26T04:00:54.405775+0000","last_peered":"2024-03-26T04:00:54.405775+0000","last_clean":"2024-03-26T04:00:54.405775+0000","last_became_active":"2024-03-26T03:55:59.308354+0000","last_became_peered":"2024-03-26T03:55:59.308354+0000","last_unstale":"2024-03-26T04:00:54.405775+0000","last_undegraded":"2024-03-26T04:00:54.405775+0000","last_fullsized":"2024-03-26T04:00:54.405775+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.539528+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.171687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":482,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394565+0000","last_change":"2024-03-26T03:55:25.239440+0000","last_active":"2024-03-26T04:00:54.394565+0000","last_peered":"2024-03-26T04:00:54.394565+0000","last_clean":"2024-03-26T04:00:54.394565+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:00:54.394565+0000","last_undegraded":"2024-03-26T04:00:54.394565+0000","last_fullsized":"2024-03-26T04:00:54.394565+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:55:25.239313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T03:55:25.239313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009058+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"241'48","reported_seq":763,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.408426+0000","last_change":"2024-03-26T03:59:18.299385+0000","last_active":"2024-03-26T04:00:54.408426+0000","last_peered":"2024-03-26T04:00:54.408426+0000","last_clean":"2024-03-26T04:00:54.408426+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T04:00:54.408426+0000","last_undegraded":"2024-03-26T04:00:54.408426+0000","last_fullsized":"2024-03-26T04:00:54.408426+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T03:59:18.299233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.299233+0000","objects_scrubbed":11,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":476,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.403840+0000","last_change":"2024-03-26T03:59:29.728348+0000","last_active":"2024-03-26T04:00:54.403840+0000","last_peered":"2024-03-26T04:00:54.403840+0000","last_clean":"2024-03-26T04:00:54.403840+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:00:54.403840+0000","last_undegraded":"2024-03-26T04:00:54.403840+0000","last_fullsized":"2024-03-26T04:00:54.403840+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.728266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.728266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438007+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":326,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404629+0000","last_change":"2024-03-26T04:00:34.281218+0000","last_active":"2024-03-26T04:00:54.404629+0000","last_peered":"2024-03-26T04:00:54.404629+0000","last_clean":"2024-03-26T04:00:54.404629+0000","last_became_active":"2024-03-26T04:00:32.425416+0000","last_became_peered":"2024-03-26T04:00:32.425416+0000","last_unstale":"2024-03-26T04:00:54.404629+0000","last_undegraded":"2024-03-26T04:00:54.404629+0000","last_fullsized":"2024-03-26T04:00:54.404629+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.281069+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:44.378376+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"284'7936","reported_seq":8007,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:58.369978+0000","last_change":"2024-03-26T03:59:23.522695+0000","last_active":"2024-03-26T04:00:58.369978+0000","last_peered":"2024-03-26T04:00:58.369978+0000","last_clean":"2024-03-26T04:00:58.369978+0000","last_became_active":"2024-03-26T03:59:23.522433+0000","last_became_peered":"2024-03-26T03:59:23.522433+0000","last_unstale":"2024-03-26T04:00:58.369978+0000","last_undegraded":"2024-03-26T04:00:58.369978+0000","last_fullsized":"2024-03-26T04:00:58.369978+0000","mapping_epoch":253,"log_start":"261'3472","ondisk_log_start":"261'3472","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4464,"log_dups_size":0,"ondisk_log_size":4464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.172290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":520093696,"num_objects":7936,"num_object_clones":0,"num_object_copies":23808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7936,"num_write_kb":507904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":507,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407594+0000","last_change":"2024-03-26T04:00:15.989927+0000","last_active":"2024-03-26T04:00:54.407594+0000","last_peered":"2024-03-26T04:00:54.407594+0000","last_clean":"2024-03-26T04:00:54.407594+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T04:00:54.407594+0000","last_undegraded":"2024-03-26T04:00:54.407594+0000","last_fullsized":"2024-03-26T04:00:54.407594+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:15.989815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.989815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:33.856634+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":497,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.410410+0000","last_change":"2024-03-26T03:59:30.302063+0000","last_active":"2024-03-26T04:00:54.410410+0000","last_peered":"2024-03-26T04:00:54.410410+0000","last_clean":"2024-03-26T04:00:54.410410+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T04:00:54.410410+0000","last_undegraded":"2024-03-26T04:00:54.410410+0000","last_fullsized":"2024-03-26T04:00:54.410410+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.301949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.301949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.722337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":521,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401390+0000","last_change":"2024-03-26T04:00:05.018260+0000","last_active":"2024-03-26T04:00:54.401390+0000","last_peered":"2024-03-26T04:00:54.401390+0000","last_clean":"2024-03-26T04:00:54.401390+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T04:00:54.401390+0000","last_undegraded":"2024-03-26T04:00:54.401390+0000","last_fullsized":"2024-03-26T04:00:54.401390+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:05.018129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:00:05.018129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:24.981088+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'23","reported_seq":532,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406903+0000","last_change":"2024-03-26T04:00:11.534975+0000","last_active":"2024-03-26T04:00:54.406903+0000","last_peered":"2024-03-26T04:00:54.406903+0000","last_clean":"2024-03-26T04:00:54.406903+0000","last_became_active":"2024-03-26T03:55:59.299436+0000","last_became_peered":"2024-03-26T03:55:59.299436+0000","last_unstale":"2024-03-26T04:00:54.406903+0000","last_undegraded":"2024-03-26T04:00:54.406903+0000","last_fullsized":"2024-03-26T04:00:54.406903+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'23","last_scrub_stamp":"2024-03-26T04:00:11.534660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.534660+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:14.681808+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":493,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.403053+0000","last_change":"2024-03-26T04:00:06.045528+0000","last_active":"2024-03-26T04:00:54.403053+0000","last_peered":"2024-03-26T04:00:54.403053+0000","last_clean":"2024-03-26T04:00:54.403053+0000","last_became_active":"2024-03-26T03:55:59.310048+0000","last_became_peered":"2024-03-26T03:55:59.310048+0000","last_unstale":"2024-03-26T04:00:54.403053+0000","last_undegraded":"2024-03-26T04:00:54.403053+0000","last_fullsized":"2024-03-26T04:00:54.403053+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:06.045409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:06.045409+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:25.286379+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"284'7748","reported_seq":7820,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.043148+0000","last_change":"2024-03-26T03:59:23.518791+0000","last_active":"2024-03-26T04:00:55.043148+0000","last_peered":"2024-03-26T04:00:55.043148+0000","last_clean":"2024-03-26T04:00:55.043148+0000","last_became_active":"2024-03-26T03:59:23.518459+0000","last_became_peered":"2024-03-26T03:59:23.518459+0000","last_unstale":"2024-03-26T04:00:55.043148+0000","last_undegraded":"2024-03-26T04:00:55.043148+0000","last_fullsized":"2024-03-26T04:00:55.043148+0000","mapping_epoch":253,"log_start":"262'3678","ondisk_log_start":"262'3678","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4070,"log_dups_size":0,"ondisk_log_size":4070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.562198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":507772928,"num_objects":7748,"num_object_clones":0,"num_object_copies":23244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7748,"num_write_kb":495872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":449,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407610+0000","last_change":"2024-03-26T04:00:29.313392+0000","last_active":"2024-03-26T04:00:54.407610+0000","last_peered":"2024-03-26T04:00:54.407610+0000","last_clean":"2024-03-26T04:00:54.407610+0000","last_became_active":"2024-03-26T03:57:49.367895+0000","last_became_peered":"2024-03-26T03:57:49.367895+0000","last_unstale":"2024-03-26T04:00:54.407610+0000","last_undegraded":"2024-03-26T04:00:54.407610+0000","last_fullsized":"2024-03-26T04:00:54.407610+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.313268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:29.313268+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.313268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:54.685724+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":774,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405204+0000","last_change":"2024-03-26T04:00:27.961142+0000","last_active":"2024-03-26T04:00:54.405204+0000","last_peered":"2024-03-26T04:00:54.405204+0000","last_clean":"2024-03-26T04:00:54.405204+0000","last_became_active":"2024-03-26T04:00:27.442487+0000","last_became_peered":"2024-03-26T04:00:27.442487+0000","last_unstale":"2024-03-26T04:00:54.405204+0000","last_undegraded":"2024-03-26T04:00:54.405204+0000","last_fullsized":"2024-03-26T04:00:54.405204+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:27.960979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.960979+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:28.537740+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402460+0000","last_change":"2024-03-26T03:59:29.150185+0000","last_active":"2024-03-26T04:00:54.402460+0000","last_peered":"2024-03-26T04:00:54.402460+0000","last_clean":"2024-03-26T04:00:54.402460+0000","last_became_active":"2024-03-26T03:57:49.369463+0000","last_became_peered":"2024-03-26T03:57:49.369463+0000","last_unstale":"2024-03-26T04:00:54.402460+0000","last_undegraded":"2024-03-26T04:00:54.402460+0000","last_fullsized":"2024-03-26T04:00:54.402460+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:29.150101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:29.150101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":607,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395126+0000","last_change":"2024-03-26T03:55:19.240262+0000","last_active":"2024-03-26T04:00:54.395126+0000","last_peered":"2024-03-26T04:00:54.395126+0000","last_clean":"2024-03-26T04:00:54.395126+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:00:54.395126+0000","last_undegraded":"2024-03-26T04:00:54.395126+0000","last_fullsized":"2024-03-26T04:00:54.395126+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'17","last_scrub_stamp":"2024-03-26T03:55:19.238420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:55:19.238420+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009091+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":375,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.403077+0000","last_change":"2024-03-26T03:59:25.418140+0000","last_active":"2024-03-26T04:00:54.403077+0000","last_peered":"2024-03-26T04:00:54.403077+0000","last_clean":"2024-03-26T04:00:54.403077+0000","last_became_active":"2024-03-26T03:58:40.497499+0000","last_became_peered":"2024-03-26T03:58:40.497499+0000","last_unstale":"2024-03-26T04:00:54.403077+0000","last_undegraded":"2024-03-26T04:00:54.403077+0000","last_fullsized":"2024-03-26T04:00:54.403077+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.418009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.418009+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.221568+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":371,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407080+0000","last_change":"2024-03-26T04:00:32.593048+0000","last_active":"2024-03-26T04:00:54.407080+0000","last_peered":"2024-03-26T04:00:54.407080+0000","last_clean":"2024-03-26T04:00:54.407080+0000","last_became_active":"2024-03-26T04:00:32.432038+0000","last_became_peered":"2024-03-26T04:00:32.432038+0000","last_unstale":"2024-03-26T04:00:54.407080+0000","last_undegraded":"2024-03-26T04:00:54.407080+0000","last_fullsized":"2024-03-26T04:00:54.407080+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:32.592884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:32.592884+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:53.373365+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"284'7805","reported_seq":7877,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.249574+0000","last_change":"2024-03-26T03:59:23.931889+0000","last_active":"2024-03-26T04:00:55.249574+0000","last_peered":"2024-03-26T04:00:55.249574+0000","last_clean":"2024-03-26T04:00:55.249574+0000","last_became_active":"2024-03-26T03:59:23.931557+0000","last_became_peered":"2024-03-26T03:59:23.931557+0000","last_unstale":"2024-03-26T04:00:55.249574+0000","last_undegraded":"2024-03-26T04:00:55.249574+0000","last_fullsized":"2024-03-26T04:00:55.249574+0000","mapping_epoch":253,"log_start":"260'3003","ondisk_log_start":"260'3003","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4802,"log_dups_size":0,"ondisk_log_size":4802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511508480,"num_objects":7805,"num_object_clones":0,"num_object_copies":23415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7805,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7805,"num_write_kb":499520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":491,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.408128+0000","last_change":"2024-03-26T04:00:29.552584+0000","last_active":"2024-03-26T04:00:54.408128+0000","last_peered":"2024-03-26T04:00:54.408128+0000","last_clean":"2024-03-26T04:00:54.408128+0000","last_became_active":"2024-03-26T03:57:49.379569+0000","last_became_peered":"2024-03-26T03:57:49.379569+0000","last_unstale":"2024-03-26T04:00:54.408128+0000","last_undegraded":"2024-03-26T04:00:54.408128+0000","last_fullsized":"2024-03-26T04:00:54.408128+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.552445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.552445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:47.963533+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":513,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401589+0000","last_change":"2024-03-26T04:00:23.326811+0000","last_active":"2024-03-26T04:00:54.401589+0000","last_peered":"2024-03-26T04:00:54.401589+0000","last_clean":"2024-03-26T04:00:54.401589+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T04:00:54.401589+0000","last_undegraded":"2024-03-26T04:00:54.401589+0000","last_fullsized":"2024-03-26T04:00:54.401589+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:23.326554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.326554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.735987+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1240,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:57.736147+0000","last_change":"2024-03-26T04:00:28.914240+0000","last_active":"2024-03-26T04:00:57.736147+0000","last_peered":"2024-03-26T04:00:57.736147+0000","last_clean":"2024-03-26T04:00:57.736147+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:00:57.736147+0000","last_undegraded":"2024-03-26T04:00:57.736147+0000","last_fullsized":"2024-03-26T04:00:57.736147+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:00:28.914032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:00:28.914032+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:42.430167+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":727,"num_read_kb":722,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":516,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397811+0000","last_change":"2024-03-26T04:00:18.345453+0000","last_active":"2024-03-26T04:00:54.397811+0000","last_peered":"2024-03-26T04:00:54.397811+0000","last_clean":"2024-03-26T04:00:54.397811+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T04:00:54.397811+0000","last_undegraded":"2024-03-26T04:00:54.397811+0000","last_fullsized":"2024-03-26T04:00:54.397811+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:00:18.345305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.345305+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:37.033184+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":422,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407637+0000","last_change":"2024-03-26T03:59:32.538750+0000","last_active":"2024-03-26T04:00:54.407637+0000","last_peered":"2024-03-26T04:00:54.407637+0000","last_clean":"2024-03-26T04:00:54.407637+0000","last_became_active":"2024-03-26T03:55:59.305885+0000","last_became_peered":"2024-03-26T03:55:59.305885+0000","last_unstale":"2024-03-26T04:00:54.407637+0000","last_undegraded":"2024-03-26T04:00:54.407637+0000","last_fullsized":"2024-03-26T04:00:54.407637+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:32.538628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:32.538628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.172318+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":326,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397556+0000","last_change":"2024-03-26T04:00:33.288471+0000","last_active":"2024-03-26T04:00:54.397556+0000","last_peered":"2024-03-26T04:00:54.397556+0000","last_clean":"2024-03-26T04:00:54.397556+0000","last_became_active":"2024-03-26T04:00:32.425047+0000","last_became_peered":"2024-03-26T04:00:32.425047+0000","last_unstale":"2024-03-26T04:00:54.397556+0000","last_undegraded":"2024-03-26T04:00:54.397556+0000","last_fullsized":"2024-03-26T04:00:54.397556+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.288346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.288346+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:54.867094+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"284'7924","reported_seq":7996,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:59.281085+0000","last_change":"2024-03-26T03:59:23.930380+0000","last_active":"2024-03-26T04:00:59.281085+0000","last_peered":"2024-03-26T04:00:59.281085+0000","last_clean":"2024-03-26T04:00:59.281085+0000","last_became_active":"2024-03-26T03:59:23.930042+0000","last_became_peered":"2024-03-26T03:59:23.930042+0000","last_unstale":"2024-03-26T04:00:59.281085+0000","last_undegraded":"2024-03-26T04:00:59.281085+0000","last_fullsized":"2024-03-26T04:00:59.281085+0000","mapping_epoch":253,"log_start":"262'3421","ondisk_log_start":"262'3421","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":519307264,"num_objects":7924,"num_object_clones":0,"num_object_copies":23772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7924,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7924,"num_write_kb":507136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":484,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406503+0000","last_change":"2024-03-26T04:00:35.502278+0000","last_active":"2024-03-26T04:00:54.406503+0000","last_peered":"2024-03-26T04:00:54.406503+0000","last_clean":"2024-03-26T04:00:54.406503+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T04:00:54.406503+0000","last_undegraded":"2024-03-26T04:00:54.406503+0000","last_fullsized":"2024-03-26T04:00:54.406503+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.502151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:41.588880+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":499,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.408336+0000","last_change":"2024-03-26T04:00:34.550294+0000","last_active":"2024-03-26T04:00:54.408336+0000","last_peered":"2024-03-26T04:00:54.408336+0000","last_clean":"2024-03-26T04:00:54.408336+0000","last_became_active":"2024-03-26T03:55:59.300059+0000","last_became_peered":"2024-03-26T03:55:59.300059+0000","last_unstale":"2024-03-26T04:00:54.408336+0000","last_undegraded":"2024-03-26T04:00:54.408336+0000","last_fullsized":"2024-03-26T04:00:54.408336+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:34.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.550087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:03.372417+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"241'35","reported_seq":560,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399490+0000","last_change":"2024-03-26T04:00:38.315918+0000","last_active":"2024-03-26T04:00:54.399490+0000","last_peered":"2024-03-26T04:00:54.399490+0000","last_clean":"2024-03-26T04:00:54.399490+0000","last_became_active":"2024-03-26T03:55:59.308347+0000","last_became_peered":"2024-03-26T03:55:59.308347+0000","last_unstale":"2024-03-26T04:00:54.399490+0000","last_undegraded":"2024-03-26T04:00:54.399490+0000","last_fullsized":"2024-03-26T04:00:54.399490+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'35","last_scrub_stamp":"2024-03-26T04:00:38.314372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.314372+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:56.337027+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":423,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404832+0000","last_change":"2024-03-26T03:59:27.734291+0000","last_active":"2024-03-26T04:00:54.404832+0000","last_peered":"2024-03-26T04:00:54.404832+0000","last_clean":"2024-03-26T04:00:54.404832+0000","last_became_active":"2024-03-26T03:55:59.307679+0000","last_became_peered":"2024-03-26T03:55:59.307679+0000","last_unstale":"2024-03-26T04:00:54.404832+0000","last_undegraded":"2024-03-26T04:00:54.404832+0000","last_fullsized":"2024-03-26T04:00:54.404832+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398536+0000","last_change":"2024-03-26T04:00:33.391324+0000","last_active":"2024-03-26T04:00:54.398536+0000","last_peered":"2024-03-26T04:00:54.398536+0000","last_clean":"2024-03-26T04:00:54.398536+0000","last_became_active":"2024-03-26T04:00:32.446284+0000","last_became_peered":"2024-03-26T04:00:32.446284+0000","last_unstale":"2024-03-26T04:00:54.398536+0000","last_undegraded":"2024-03-26T04:00:54.398536+0000","last_fullsized":"2024-03-26T04:00:54.398536+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.390854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.390854+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:50.683670+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"284'8124","reported_seq":8196,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:58.352716+0000","last_change":"2024-03-26T03:59:23.523599+0000","last_active":"2024-03-26T04:00:58.352716+0000","last_peered":"2024-03-26T04:00:58.352716+0000","last_clean":"2024-03-26T04:00:58.352716+0000","last_became_active":"2024-03-26T03:59:23.523293+0000","last_became_peered":"2024-03-26T03:59:23.523293+0000","last_unstale":"2024-03-26T04:00:58.352716+0000","last_undegraded":"2024-03-26T04:00:58.352716+0000","last_fullsized":"2024-03-26T04:00:58.352716+0000","mapping_epoch":253,"log_start":"262'3708","ondisk_log_start":"262'3708","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4416,"log_dups_size":0,"ondisk_log_size":4416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.171645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":532414464,"num_objects":8124,"num_object_clones":0,"num_object_copies":24372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8124,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8124,"num_write_kb":519936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":486,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.399286+0000","last_change":"2024-03-26T03:59:13.839565+0000","last_active":"2024-03-26T04:00:54.399286+0000","last_peered":"2024-03-26T04:00:54.399286+0000","last_clean":"2024-03-26T04:00:54.399286+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T04:00:54.399286+0000","last_undegraded":"2024-03-26T04:00:54.399286+0000","last_fullsized":"2024-03-26T04:00:54.399286+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:13.839415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.839415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.437944+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":453,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.415410+0000","last_change":"2024-03-26T04:00:14.456638+0000","last_active":"2024-03-26T04:00:54.415410+0000","last_peered":"2024-03-26T04:00:54.415410+0000","last_clean":"2024-03-26T04:00:54.415410+0000","last_became_active":"2024-03-26T03:57:49.367990+0000","last_became_peered":"2024-03-26T03:57:49.367990+0000","last_unstale":"2024-03-26T04:00:54.415410+0000","last_undegraded":"2024-03-26T04:00:54.415410+0000","last_fullsized":"2024-03-26T04:00:54.415410+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:14.456520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:14.456520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:38.703690+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":529,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.409306+0000","last_change":"2024-03-26T03:59:15.299800+0000","last_active":"2024-03-26T04:00:54.409306+0000","last_peered":"2024-03-26T04:00:54.409306+0000","last_clean":"2024-03-26T04:00:54.409306+0000","last_became_active":"2024-03-26T03:55:59.298912+0000","last_became_peered":"2024-03-26T03:55:59.298912+0000","last_unstale":"2024-03-26T04:00:54.409306+0000","last_undegraded":"2024-03-26T04:00:54.409306+0000","last_fullsized":"2024-03-26T04:00:54.409306+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T03:59:15.299680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:15.299680+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"215'13","reported_seq":664,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398002+0000","last_change":"2024-03-26T03:59:16.761537+0000","last_active":"2024-03-26T04:00:54.398002+0000","last_peered":"2024-03-26T04:00:54.398002+0000","last_clean":"2024-03-26T04:00:54.398002+0000","last_became_active":"2024-03-26T03:57:49.361026+0000","last_became_peered":"2024-03-26T03:57:49.361026+0000","last_unstale":"2024-03-26T04:00:54.398002+0000","last_undegraded":"2024-03-26T04:00:54.398002+0000","last_fullsized":"2024-03-26T04:00:54.398002+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'13","last_scrub_stamp":"2024-03-26T03:59:16.761422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.761422+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397776+0000","last_change":"2024-03-26T04:00:34.424149+0000","last_active":"2024-03-26T04:00:54.397776+0000","last_peered":"2024-03-26T04:00:54.397776+0000","last_clean":"2024-03-26T04:00:54.397776+0000","last_became_active":"2024-03-26T04:00:32.449083+0000","last_became_peered":"2024-03-26T04:00:32.449083+0000","last_unstale":"2024-03-26T04:00:54.397776+0000","last_undegraded":"2024-03-26T04:00:54.397776+0000","last_fullsized":"2024-03-26T04:00:54.397776+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.423937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.423937+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:45.148661+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"284'7808","reported_seq":7880,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.050362+0000","last_change":"2024-03-26T03:59:23.930568+0000","last_active":"2024-03-26T04:00:55.050362+0000","last_peered":"2024-03-26T04:00:55.050362+0000","last_clean":"2024-03-26T04:00:55.050362+0000","last_became_active":"2024-03-26T03:59:23.930275+0000","last_became_peered":"2024-03-26T03:59:23.930275+0000","last_unstale":"2024-03-26T04:00:55.050362+0000","last_undegraded":"2024-03-26T04:00:55.050362+0000","last_fullsized":"2024-03-26T04:00:55.050362+0000","mapping_epoch":253,"log_start":"262'3726","ondisk_log_start":"262'3726","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4082,"log_dups_size":0,"ondisk_log_size":4082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.561874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":511705088,"num_objects":7808,"num_object_clones":0,"num_object_copies":23424,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7808,"num_write_kb":499712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":486,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.406636+0000","last_change":"2024-03-26T04:00:07.116295+0000","last_active":"2024-03-26T04:00:54.406636+0000","last_peered":"2024-03-26T04:00:54.406636+0000","last_clean":"2024-03-26T04:00:54.406636+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T04:00:54.406636+0000","last_undegraded":"2024-03-26T04:00:54.406636+0000","last_fullsized":"2024-03-26T04:00:54.406636+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:07.116198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:00:07.116198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:23.405005+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":485,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397911+0000","last_change":"2024-03-26T03:59:14.764278+0000","last_active":"2024-03-26T04:00:54.397911+0000","last_peered":"2024-03-26T04:00:54.397911+0000","last_clean":"2024-03-26T04:00:54.397911+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:00:54.397911+0000","last_undegraded":"2024-03-26T04:00:54.397911+0000","last_fullsized":"2024-03-26T04:00:54.397911+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-03-26T03:59:14.764155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.764155+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.438448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":567,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393791+0000","last_change":"2024-03-26T04:00:29.240030+0000","last_active":"2024-03-26T04:00:54.393791+0000","last_peered":"2024-03-26T04:00:54.393791+0000","last_clean":"2024-03-26T04:00:54.393791+0000","last_became_active":"2024-03-26T03:55:59.301950+0000","last_became_peered":"2024-03-26T03:55:59.301950+0000","last_unstale":"2024-03-26T04:00:54.393791+0000","last_undegraded":"2024-03-26T04:00:54.393791+0000","last_fullsized":"2024-03-26T04:00:54.393791+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:00:29.239892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.239892+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:50.493451+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":199,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397417+0000","last_change":"2024-03-26T04:00:15.479476+0000","last_active":"2024-03-26T04:00:54.397417+0000","last_peered":"2024-03-26T04:00:54.397417+0000","last_clean":"2024-03-26T04:00:54.397417+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T04:00:54.397417+0000","last_undegraded":"2024-03-26T04:00:54.397417+0000","last_fullsized":"2024-03-26T04:00:54.397417+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'11","last_scrub_stamp":"2024-03-26T04:00:15.479350+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.479350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:20.544119+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394776+0000","last_change":"2024-03-26T04:00:33.175124+0000","last_active":"2024-03-26T04:00:54.394776+0000","last_peered":"2024-03-26T04:00:54.394776+0000","last_clean":"2024-03-26T04:00:54.394776+0000","last_became_active":"2024-03-26T04:00:32.434318+0000","last_became_peered":"2024-03-26T04:00:32.434318+0000","last_unstale":"2024-03-26T04:00:54.394776+0000","last_undegraded":"2024-03-26T04:00:54.394776+0000","last_fullsized":"2024-03-26T04:00:54.394776+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.174993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.174993+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:51.395656+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"284'8066","reported_seq":8138,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:59.288510+0000","last_change":"2024-03-26T03:59:23.525085+0000","last_active":"2024-03-26T04:00:59.288510+0000","last_peered":"2024-03-26T04:00:59.288510+0000","last_clean":"2024-03-26T04:00:59.288510+0000","last_became_active":"2024-03-26T03:59:23.524729+0000","last_became_peered":"2024-03-26T03:59:23.524729+0000","last_unstale":"2024-03-26T04:00:59.288510+0000","last_undegraded":"2024-03-26T04:00:59.288510+0000","last_fullsized":"2024-03-26T04:00:59.288510+0000","mapping_epoch":253,"log_start":"262'3632","ondisk_log_start":"262'3632","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":528613376,"num_objects":8066,"num_object_clones":0,"num_object_copies":24198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8066,"num_write_kb":516224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":476,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394468+0000","last_change":"2024-03-26T04:00:10.189235+0000","last_active":"2024-03-26T04:00:54.394468+0000","last_peered":"2024-03-26T04:00:54.394468+0000","last_clean":"2024-03-26T04:00:54.394468+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T04:00:54.394468+0000","last_undegraded":"2024-03-26T04:00:54.394468+0000","last_fullsized":"2024-03-26T04:00:54.394468+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.189077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.189077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:36.465064+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402660+0000","last_change":"2024-03-26T03:59:12.184503+0000","last_active":"2024-03-26T04:00:54.402660+0000","last_peered":"2024-03-26T04:00:54.402660+0000","last_clean":"2024-03-26T04:00:54.402660+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T04:00:54.402660+0000","last_undegraded":"2024-03-26T04:00:54.402660+0000","last_fullsized":"2024-03-26T04:00:54.402660+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.184357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.184357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":61,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401061+0000","last_change":"2024-03-26T04:00:25.473605+0000","last_active":"2024-03-26T04:00:54.401061+0000","last_peered":"2024-03-26T04:00:54.401061+0000","last_clean":"2024-03-26T04:00:54.401061+0000","last_became_active":"2024-03-26T04:00:25.423339+0000","last_became_peered":"2024-03-26T04:00:25.423339+0000","last_unstale":"2024-03-26T04:00:54.401061+0000","last_undegraded":"2024-03-26T04:00:54.401061+0000","last_fullsized":"2024-03-26T04:00:54.401061+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T03:59:21.245251+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.245251+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:58.221542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":488,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393555+0000","last_change":"2024-03-26T04:00:34.128491+0000","last_active":"2024-03-26T04:00:54.393555+0000","last_peered":"2024-03-26T04:00:54.393555+0000","last_clean":"2024-03-26T04:00:54.393555+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T04:00:54.393555+0000","last_undegraded":"2024-03-26T04:00:54.393555+0000","last_fullsized":"2024-03-26T04:00:54.393555+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:34.128366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.128366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:57.367315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.405572+0000","last_change":"2024-03-26T04:00:35.340211+0000","last_active":"2024-03-26T04:00:54.405572+0000","last_peered":"2024-03-26T04:00:54.405572+0000","last_clean":"2024-03-26T04:00:54.405572+0000","last_became_active":"2024-03-26T04:00:32.433957+0000","last_became_peered":"2024-03-26T04:00:32.433957+0000","last_unstale":"2024-03-26T04:00:54.405572+0000","last_undegraded":"2024-03-26T04:00:54.405572+0000","last_fullsized":"2024-03-26T04:00:54.405572+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.340085+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:45.806762+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"284'7714","reported_seq":7785,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.068523+0000","last_change":"2024-03-26T03:59:23.932941+0000","last_active":"2024-03-26T04:00:55.068523+0000","last_peered":"2024-03-26T04:00:55.068523+0000","last_clean":"2024-03-26T04:00:55.068523+0000","last_became_active":"2024-03-26T03:59:23.932638+0000","last_became_peered":"2024-03-26T03:59:23.932638+0000","last_unstale":"2024-03-26T04:00:55.068523+0000","last_undegraded":"2024-03-26T04:00:55.068523+0000","last_fullsized":"2024-03-26T04:00:55.068523+0000","mapping_epoch":253,"log_start":"262'3616","ondisk_log_start":"262'3616","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4098,"log_dups_size":0,"ondisk_log_size":4098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.561840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505544704,"num_objects":7714,"num_object_clones":0,"num_object_copies":23142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7714,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7714,"num_write_kb":493696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":400,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.415285+0000","last_change":"2024-03-26T03:59:25.561119+0000","last_active":"2024-03-26T04:00:54.415285+0000","last_peered":"2024-03-26T04:00:54.415285+0000","last_clean":"2024-03-26T04:00:54.415285+0000","last_became_active":"2024-03-26T03:55:59.340297+0000","last_became_peered":"2024-03-26T03:55:59.340297+0000","last_unstale":"2024-03-26T04:00:54.415285+0000","last_undegraded":"2024-03-26T04:00:54.415285+0000","last_fullsized":"2024-03-26T04:00:54.415285+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:25.561017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T03:59:25.561017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.171716+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"181'10","reported_seq":642,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395149+0000","last_change":"2024-03-26T04:00:22.364328+0000","last_active":"2024-03-26T04:00:54.395149+0000","last_peered":"2024-03-26T04:00:54.395149+0000","last_clean":"2024-03-26T04:00:54.395149+0000","last_became_active":"2024-03-26T03:55:59.295551+0000","last_became_peered":"2024-03-26T03:55:59.295551+0000","last_unstale":"2024-03-26T04:00:54.395149+0000","last_undegraded":"2024-03-26T04:00:54.395149+0000","last_fullsized":"2024-03-26T04:00:54.395149+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:22.364212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.364212+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:34.784802+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.397973+0000","last_change":"2024-03-26T03:59:27.171191+0000","last_active":"2024-03-26T04:00:54.397973+0000","last_peered":"2024-03-26T04:00:54.397973+0000","last_clean":"2024-03-26T04:00:54.397973+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T04:00:54.397973+0000","last_undegraded":"2024-03-26T04:00:54.397973+0000","last_fullsized":"2024-03-26T04:00:54.397973+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.171113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.171113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:56.562261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":573,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407018+0000","last_change":"2024-03-26T04:00:15.456301+0000","last_active":"2024-03-26T04:00:54.407018+0000","last_peered":"2024-03-26T04:00:54.407018+0000","last_clean":"2024-03-26T04:00:54.407018+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T04:00:54.407018+0000","last_undegraded":"2024-03-26T04:00:54.407018+0000","last_fullsized":"2024-03-26T04:00:54.407018+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:15.456156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.456156+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:39.174797+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"284'7867","reported_seq":7939,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:59.264233+0000","last_change":"2024-03-26T03:59:23.523826+0000","last_active":"2024-03-26T04:00:59.264233+0000","last_peered":"2024-03-26T04:00:59.264233+0000","last_clean":"2024-03-26T04:00:59.264233+0000","last_became_active":"2024-03-26T03:59:23.523509+0000","last_became_peered":"2024-03-26T03:59:23.523509+0000","last_unstale":"2024-03-26T04:00:59.264233+0000","last_undegraded":"2024-03-26T04:00:59.264233+0000","last_fullsized":"2024-03-26T04:00:59.264233+0000","mapping_epoch":253,"log_start":"261'3388","ondisk_log_start":"261'3388","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4479,"log_dups_size":0,"ondisk_log_size":4479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":515571712,"num_objects":7867,"num_object_clones":0,"num_object_copies":23601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7867,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7867,"num_write_kb":503488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.398440+0000","last_change":"2024-03-26T04:00:10.493315+0000","last_active":"2024-03-26T04:00:54.398440+0000","last_peered":"2024-03-26T04:00:54.398440+0000","last_clean":"2024-03-26T04:00:54.398440+0000","last_became_active":"2024-03-26T03:57:49.372744+0000","last_became_peered":"2024-03-26T03:57:49.372744+0000","last_unstale":"2024-03-26T04:00:54.398440+0000","last_undegraded":"2024-03-26T04:00:54.398440+0000","last_fullsized":"2024-03-26T04:00:54.398440+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.493215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:35.545235+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":442,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.407533+0000","last_change":"2024-03-26T04:00:10.363895+0000","last_active":"2024-03-26T04:00:54.407533+0000","last_peered":"2024-03-26T04:00:54.407533+0000","last_clean":"2024-03-26T04:00:54.407533+0000","last_became_active":"2024-03-26T03:55:59.300192+0000","last_became_peered":"2024-03-26T03:55:59.300192+0000","last_unstale":"2024-03-26T04:00:54.407533+0000","last_undegraded":"2024-03-26T04:00:54.407533+0000","last_fullsized":"2024-03-26T04:00:54.407533+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.363768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.363768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:26.400425+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":484,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.393635+0000","last_change":"2024-03-26T04:00:17.365750+0000","last_active":"2024-03-26T04:00:54.393635+0000","last_peered":"2024-03-26T04:00:54.393635+0000","last_clean":"2024-03-26T04:00:54.393635+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T04:00:54.393635+0000","last_undegraded":"2024-03-26T04:00:54.393635+0000","last_fullsized":"2024-03-26T04:00:54.393635+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:25.118197+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":592,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394858+0000","last_change":"2024-03-26T03:59:14.317073+0000","last_active":"2024-03-26T04:00:54.394858+0000","last_peered":"2024-03-26T04:00:54.394858+0000","last_clean":"2024-03-26T04:00:54.394858+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T04:00:54.394858+0000","last_undegraded":"2024-03-26T04:00:54.394858+0000","last_fullsized":"2024-03-26T04:00:54.394858+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T03:59:14.316934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.316934+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:57.009353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":321,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404791+0000","last_change":"2024-03-26T04:00:33.276533+0000","last_active":"2024-03-26T04:00:54.404791+0000","last_peered":"2024-03-26T04:00:54.404791+0000","last_clean":"2024-03-26T04:00:54.404791+0000","last_became_active":"2024-03-26T04:00:32.428922+0000","last_became_peered":"2024-03-26T04:00:32.428922+0000","last_unstale":"2024-03-26T04:00:54.404791+0000","last_undegraded":"2024-03-26T04:00:54.404791+0000","last_fullsized":"2024-03-26T04:00:54.404791+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.276421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.276421+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:00.079299+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"284'7788","reported_seq":7860,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.450004+0000","last_change":"2024-03-26T03:59:23.931746+0000","last_active":"2024-03-26T04:00:55.450004+0000","last_peered":"2024-03-26T04:00:55.450004+0000","last_clean":"2024-03-26T04:00:55.450004+0000","last_became_active":"2024-03-26T03:59:23.931459+0000","last_became_peered":"2024-03-26T03:59:23.931459+0000","last_unstale":"2024-03-26T04:00:55.450004+0000","last_undegraded":"2024-03-26T04:00:55.450004+0000","last_fullsized":"2024-03-26T04:00:55.450004+0000","mapping_epoch":253,"log_start":"262'3658","ondisk_log_start":"262'3658","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:59.480490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":510394368,"num_objects":7788,"num_object_clones":0,"num_object_copies":23364,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7788,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7788,"num_write_kb":498432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":508,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.403322+0000","last_change":"2024-03-26T04:00:27.251209+0000","last_active":"2024-03-26T04:00:54.403322+0000","last_peered":"2024-03-26T04:00:54.403322+0000","last_clean":"2024-03-26T04:00:54.403322+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T04:00:54.403322+0000","last_undegraded":"2024-03-26T04:00:54.403322+0000","last_fullsized":"2024-03-26T04:00:54.403322+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.251082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.251082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:56.805532+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":474,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.408586+0000","last_change":"2024-03-26T03:59:19.316596+0000","last_active":"2024-03-26T04:00:54.408586+0000","last_peered":"2024-03-26T04:00:54.408586+0000","last_clean":"2024-03-26T04:00:54.408586+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:00:54.408586+0000","last_undegraded":"2024-03-26T04:00:54.408586+0000","last_fullsized":"2024-03-26T04:00:54.408586+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:19.316523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:19.316523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:00:55.465273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":435,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.394570+0000","last_change":"2024-03-26T03:59:27.313345+0000","last_active":"2024-03-26T04:00:54.394570+0000","last_peered":"2024-03-26T04:00:54.394570+0000","last_clean":"2024-03-26T04:00:54.394570+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T04:00:54.394570+0000","last_undegraded":"2024-03-26T04:00:54.394570+0000","last_fullsized":"2024-03-26T04:00:54.394570+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.313255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.313255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":529,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395256+0000","last_change":"2024-03-26T03:59:26.340091+0000","last_active":"2024-03-26T04:00:54.395256+0000","last_peered":"2024-03-26T04:00:54.395256+0000","last_clean":"2024-03-26T04:00:54.395256+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T04:00:54.395256+0000","last_undegraded":"2024-03-26T04:00:54.395256+0000","last_fullsized":"2024-03-26T04:00:54.395256+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:26.339621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.339621+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":781,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.404976+0000","last_change":"2024-03-26T04:00:33.001695+0000","last_active":"2024-03-26T04:00:54.404976+0000","last_peered":"2024-03-26T04:00:54.404976+0000","last_clean":"2024-03-26T04:00:54.404976+0000","last_became_active":"2024-03-26T04:00:32.428836+0000","last_became_peered":"2024-03-26T04:00:32.428836+0000","last_unstale":"2024-03-26T04:00:54.404976+0000","last_undegraded":"2024-03-26T04:00:54.404976+0000","last_fullsized":"2024-03-26T04:00:54.404976+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.001580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.001580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:40.740565+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"284'7833","reported_seq":7905,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:55.050199+0000","last_change":"2024-03-26T03:59:23.930540+0000","last_active":"2024-03-26T04:00:55.050199+0000","last_peered":"2024-03-26T04:00:55.050199+0000","last_clean":"2024-03-26T04:00:55.050199+0000","last_became_active":"2024-03-26T03:59:23.930238+0000","last_became_peered":"2024-03-26T03:59:23.930238+0000","last_unstale":"2024-03-26T04:00:55.050199+0000","last_undegraded":"2024-03-26T04:00:55.050199+0000","last_fullsized":"2024-03-26T04:00:55.050199+0000","mapping_epoch":253,"log_start":"262'3775","ondisk_log_start":"262'3775","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":4058,"log_dups_size":0,"ondisk_log_size":4058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":513343488,"num_objects":7833,"num_object_clones":0,"num_object_copies":23499,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7833,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7833,"num_write_kb":501312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":472,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395410+0000","last_change":"2024-03-26T04:00:13.290591+0000","last_active":"2024-03-26T04:00:54.395410+0000","last_peered":"2024-03-26T04:00:54.395410+0000","last_clean":"2024-03-26T04:00:54.395410+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T04:00:54.395410+0000","last_undegraded":"2024-03-26T04:00:54.395410+0000","last_fullsized":"2024-03-26T04:00:54.395410+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:13.290512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:00:13.290512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:35.584608+0000","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"181'10","reported_seq":658,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.402816+0000","last_change":"2024-03-26T04:00:23.973703+0000","last_active":"2024-03-26T04:00:54.402816+0000","last_peered":"2024-03-26T04:00:54.402816+0000","last_clean":"2024-03-26T04:00:54.402816+0000","last_became_active":"2024-03-26T03:55:59.351292+0000","last_became_peered":"2024-03-26T03:55:59.351292+0000","last_unstale":"2024-03-26T04:00:54.402816+0000","last_undegraded":"2024-03-26T04:00:54.402816+0000","last_fullsized":"2024-03-26T04:00:54.402816+0000","mapping_epoch":180,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":181,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:23.973576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.973576+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:27.272857+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":157,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.401135+0000","last_change":"2024-03-26T04:00:18.449990+0000","last_active":"2024-03-26T04:00:54.401135+0000","last_peered":"2024-03-26T04:00:54.401135+0000","last_clean":"2024-03-26T04:00:54.401135+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T04:00:54.401135+0000","last_undegraded":"2024-03-26T04:00:54.401135+0000","last_fullsized":"2024-03-26T04:00:54.401135+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:18.449869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.449869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:26.873893+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":564,"reported_epoch":284,"state":"active+clean","last_fresh":"2024-03-26T04:00:54.395663+0000","last_change":"2024-03-26T04:00:37.112949+0000","last_active":"2024-03-26T04:00:54.395663+0000","last_peered":"2024-03-26T04:00:54.395663+0000","last_clean":"2024-03-26T04:00:54.395663+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T04:00:54.395663+0000","last_undegraded":"2024-03-26T04:00:54.395663+0000","last_fullsized":"2024-03-26T04:00:54.395663+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:00:37.112810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:37.112810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:59.410286+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":8261337088,"num_objects":126058,"num_object_clones":0,"num_object_copies":378174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":126058,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":126058,"num_write_kb":8067712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24717361152,"data_stored":24717361152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70157,"ondisk_log_size":70157,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":727,"num_read_kb":722,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1407,"num_read_kb":1132,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":4720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":454656,"data_stored":12140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070615,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4986896,"kb_used_data":4622536,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":88758256,"statfs":{"total":95995035648,"available":90888454144,"internally_reserved":0,"allocated":4733476864,"data_stored":4732950840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":373094858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364957,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4901484,"kb_used_data":4410784,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":88843668,"statfs":{"total":95995035648,"available":90975916032,"internally_reserved":0,"allocated":4516642816,"data_stored":4516091369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2866404,"kb_used_data":2461784,"kb_used_omap":1,"kb_used_meta":404606,"kb_avail":90878748,"statfs":{"total":95995035648,"available":93059837952,"internally_reserved":0,"allocated":2520866816,"data_stored":2520324530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":414316997},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,4,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411504,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2462068,"kb_used_data":1971372,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91283084,"statfs":{"total":95995035648,"available":93473878016,"internally_reserved":0,"allocated":2018684928,"data_stored":2018179219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182570,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4343972,"kb_used_data":3951832,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":89401180,"statfs":{"total":95995035648,"available":91546808320,"internally_reserved":0,"allocated":4046675968,"data_stored":4046133930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":401537479},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953637,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2349384,"kb_used_data":1957248,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":91395768,"statfs":{"total":95995035648,"available":93589266432,"internally_reserved":0,"allocated":2004221952,"data_stored":2003695489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113341,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5056328,"kb_used_data":4625728,"kb_used_omap":4,"kb_used_meta":430587,"kb_avail":88688824,"statfs":{"total":95995035648,"available":90817355776,"internally_reserved":0,"allocated":4736745472,"data_stored":4736226538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":440921760},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2543779840,"data_stored":2543779840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4563795968,"data_stored":4563795968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2034040832,"data_stored":2034040832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4066705408,"data_stored":4066705408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2011561984,"data_stored":2011561984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4744347648,"data_stored":4744347648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4753129472,"data_stored":4753129472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:01:03.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-26T04:01:03.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:03.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:03.604 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:03.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T04:01:03.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:03.711 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:03.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:03.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:03.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:03.990 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:03.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:04.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:04.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:04.222 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:04.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:04.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:04.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:04.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:04.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:04.457 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:04.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:04.552 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:04.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:04.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:04.796 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:04.832 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:04 smithi006 ceph-mon[37501]: pgmap v559: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.24k op/s 2024-03-26T04:01:04.832 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:04 smithi006 ceph-mon[29446]: pgmap v559: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.24k op/s 2024-03-26T04:01:04.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:04.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:04 smithi086 ceph-mon[32965]: pgmap v559: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 75 MiB/s wr, 1.24k op/s 2024-03-26T04:01:04.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:04.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:05.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:05.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:05.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:05.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:05.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:05.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:05.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.377 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:05.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:05.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:05.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:05.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:05.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:05.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:05.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:05.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:05.833 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:05.835 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:05.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:05.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:05.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:06.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:06.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:06.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:06.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:06.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:06.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:06.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:06.387 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:06.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:06.535 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:06.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:06.710 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:06 smithi006 ceph-mon[37501]: pgmap v560: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1020 B/s rd, 70 MiB/s wr, 1.39k op/s 2024-03-26T04:01:06.711 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:06 smithi006 ceph-mon[29446]: pgmap v560: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1020 B/s rd, 70 MiB/s wr, 1.39k op/s 2024-03-26T04:01:06.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:06 smithi086 ceph-mon[32965]: pgmap v560: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1020 B/s rd, 70 MiB/s wr, 1.39k op/s 2024-03-26T04:01:06.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:06.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:06.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:07.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:07.077 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:07.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:07.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:07.220 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:07.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:07.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:07.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:07.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:07.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:07.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:07.568 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:01:07.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:07.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:07.693 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:07 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4159604646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.693 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:07 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.694 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:07 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4159604646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.694 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:07 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.774 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:07.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:07.788 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:01:07.829 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:01:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:07 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4159604646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:07 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:01:07.874 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:01:07.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:07.919 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:07.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:07.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:07.927 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:01:07.967 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:01:08.015 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:01:08.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:08.056 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:01:08.059 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:08.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:08.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:01:08.100 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:01:08.101 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:01:08.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:08.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:08.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:08.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:08.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:08.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:08.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:08.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:08.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:08.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:08.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:08.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:08.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:08 smithi086 ceph-mon[32965]: pgmap v561: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 938 B/s rd, 52 MiB/s wr, 1.12k op/s 2024-03-26T04:01:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:08 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:08 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:01:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:08 smithi086 ceph-mon[32965]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T04:01:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:08 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1978754144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:08.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:08.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:08.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:08.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[37501]: pgmap v561: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 938 B/s rd, 52 MiB/s wr, 1.12k op/s 2024-03-26T04:01:08.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:08.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[37501]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1978754144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[29446]: pgmap v561: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 938 B/s rd, 52 MiB/s wr, 1.12k op/s 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[29446]: osdmap e285: 8 total, 8 up, 7 in 2024-03-26T04:01:08.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:08 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1978754144' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:08.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:09.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:09.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:09.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:09.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:09.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:09.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:09.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:09.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:09.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:09.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:09.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:09.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:09.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:09.701 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:09.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:09.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 4.1b scrub starts 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 4.1b scrub ok 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: 9.4 scrub starts 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:09.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:09 smithi086 ceph-mon[32965]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T04:01:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 4.1b scrub starts 2024-03-26T04:01:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 4.1b scrub ok 2024-03-26T04:01:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: 9.4 scrub starts 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:09.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[37501]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T04:01:09.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 4.1b scrub starts 2024-03-26T04:01:09.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 4.1b scrub ok 2024-03-26T04:01:09.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T04:01:09.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T04:01:09.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T04:01:09.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T04:01:09.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: 9.4 scrub starts 2024-03-26T04:01:09.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:09.893 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:09 smithi006 ceph-mon[29446]: osdmap e286: 8 total, 8 up, 7 in 2024-03-26T04:01:09.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:09.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:09.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:10.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:10.110 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:10.145 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:10.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:10.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:10.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.382 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:10.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:10.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:10.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:10.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:10.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:10.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:10.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:10.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:10.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:10 smithi006 ceph-mon[37501]: pgmap v564: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.16k op/s 2024-03-26T04:01:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:10 smithi006 ceph-mon[29446]: pgmap v564: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.16k op/s 2024-03-26T04:01:10.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:10.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:10.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:10.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:11.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:11.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:10 smithi086 ceph-mon[32965]: pgmap v564: 158 pgs: 158 active+clean; 7.7 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.4 KiB/s rd, 23 MiB/s wr, 1.16k op/s 2024-03-26T04:01:11.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:11.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:11.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:11.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:11.352 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:11.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:11.492 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:11.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:11.634 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:11.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:11.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:11.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[37501]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:11.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:11 smithi006 ceph-mon[29446]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T04:01:11.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:11.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:11.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:12.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:12.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:12.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:11 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T04:01:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:11 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T04:01:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:11 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:11 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:11 smithi086 ceph-mon[32965]: osdmap e287: 8 total, 8 up, 7 in 2024-03-26T04:01:12.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:12.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:12.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:12.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.312 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:12.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:12.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:12.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:12.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:12.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:12.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:12.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:01:12.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:12.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:12.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:12.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: 9.d scrub starts 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: 9.4 scrub ok 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: pgmap v566: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 2.8 MiB/s wr, 1.23k op/s 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[37501]: 9.0 scrub starts 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: 9.d scrub starts 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: 9.4 scrub ok 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T04:01:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T04:01:12.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: pgmap v566: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 2.8 MiB/s wr, 1.23k op/s 2024-03-26T04:01:12.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:12 smithi006 ceph-mon[29446]: 9.0 scrub starts 2024-03-26T04:01:12.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:12.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:12.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:12.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:13.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: 9.d scrub starts 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: 9.4 scrub ok 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: pgmap v566: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.5 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 853 B/s rd, 2.8 MiB/s wr, 1.23k op/s 2024-03-26T04:01:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:12 smithi086 ceph-mon[32965]: 9.0 scrub starts 2024-03-26T04:01:13.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:13.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:01:13.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:13.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.379 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:13.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:13.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:13.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:13.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:13.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:13.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:13.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:13.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T04:01:13.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T04:01:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:13 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T04:01:13.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:13.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:13.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:14.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:14.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:14.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:13 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T04:01:14.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:13 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T04:01:14.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:13 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T04:01:14.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:13 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T04:01:14.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:14.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:14.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:14.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:14.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:14.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:14.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:14.596 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:14.638 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: 9.d scrub ok 2024-03-26T04:01:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:14.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:14.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:14.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:14.880 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:14.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:14.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: 9.6 scrub starts 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: pgmap v567: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[29446]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: 9.d scrub ok 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: 9.6 scrub starts 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: pgmap v567: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T04:01:14.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:14 smithi006 ceph-mon[37501]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T04:01:14.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: 9.d scrub ok 2024-03-26T04:01:14.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: 9.6 scrub starts 2024-03-26T04:01:14.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T04:01:14.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T04:01:14.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: pgmap v567: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.4 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.34k op/s 2024-03-26T04:01:14.913 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-mon[32965]: osdmap e288: 8 total, 8 up, 7 in 2024-03-26T04:01:14.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:15.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:15.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:15.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:15.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:15.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:15.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:15.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:15.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:15.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:15.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:15.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:15.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:15.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:15.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:15.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:15.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:15.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:15.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:15.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:15.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:16.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[37501]: 9.e scrub starts 2024-03-26T04:01:16.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[37501]: 9.0 scrub ok 2024-03-26T04:01:16.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:16.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[29446]: 9.e scrub starts 2024-03-26T04:01:16.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[29446]: 9.0 scrub ok 2024-03-26T04:01:16.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:16.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:15 smithi086 ceph-mon[32965]: 9.e scrub starts 2024-03-26T04:01:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:15 smithi086 ceph-mon[32965]: 9.0 scrub ok 2024-03-26T04:01:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:16.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:16.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:16.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:16.181 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:01:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:01:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:01:16.181 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:01:16.181 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:01:16.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:16.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:16.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:16.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:16.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:16.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:16.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:16.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:16.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:16.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:16.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:16.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:16.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:17.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:16 smithi086 ceph-mon[32965]: pgmap v569: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-03-26T04:01:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:16 smithi086 ceph-mon[32965]: 9.6 scrub ok 2024-03-26T04:01:17.105 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:17.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:16 smithi006 ceph-mon[37501]: pgmap v569: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-03-26T04:01:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:16 smithi006 ceph-mon[37501]: 9.6 scrub ok 2024-03-26T04:01:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:16 smithi006 ceph-mon[29446]: pgmap v569: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 7.2 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.67k op/s 2024-03-26T04:01:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:16 smithi006 ceph-mon[29446]: 9.6 scrub ok 2024-03-26T04:01:17.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:17.237 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:17.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:17.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:17.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:17.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:17.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:17.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:17.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:17.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:17.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:17.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:17.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:17.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 9.3 scrub starts 2024-03-26T04:01:17.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 9.e scrub ok 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 9.3 scrub starts 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 9.e scrub ok 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T04:01:17.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:17 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T04:01:17.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:17.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:18.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:18.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 9.3 scrub starts 2024-03-26T04:01:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 9.e scrub ok 2024-03-26T04:01:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T04:01:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T04:01:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T04:01:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:17 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T04:01:18.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:18.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T04:01:18.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.207 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:18.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:18.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:18.340 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:18.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:18.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:18.499 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:18.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:18.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:18.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:18.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:18.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:18.748 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:18.780 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:18.814 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:18.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:18.823 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T04:01:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T04:01:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[37501]: pgmap v570: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T04:01:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[37501]: 9.3 scrub ok 2024-03-26T04:01:18.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[37501]: 9.1 scrub starts 2024-03-26T04:01:18.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T04:01:18.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T04:01:18.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[29446]: pgmap v570: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T04:01:18.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[29446]: 9.3 scrub ok 2024-03-26T04:01:18.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:18 smithi006 ceph-mon[29446]: 9.1 scrub starts 2024-03-26T04:01:18.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:18.966 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:18.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:19.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:18 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T04:01:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:18 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T04:01:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:18 smithi086 ceph-mon[32965]: pgmap v570: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 7.1 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.20k op/s 2024-03-26T04:01:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:18 smithi086 ceph-mon[32965]: 9.3 scrub ok 2024-03-26T04:01:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:18 smithi086 ceph-mon[32965]: 9.1 scrub starts 2024-03-26T04:01:19.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:19.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:19.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:19.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:19.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:19.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:19.518 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:19.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:19.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:19.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:19.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:19.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:19.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:19.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:19.966 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T04:01:19.967 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T04:01:19.968 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:19 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T04:01:20.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:20.051 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:20.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T04:01:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T04:01:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T04:01:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T04:01:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T04:01:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:19 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T04:01:20.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:20.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:20.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:20.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.397 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:20.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:20.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:20.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:20.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:20.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:20.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:20.696 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 1 pgp_num to 11 2024-03-26T04:01:20.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:20.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:20.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: pgmap v571: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T04:01:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: pgmap v571: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T04:01:20.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T04:01:20.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:20.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:20.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:20.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:21.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:21.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T04:01:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: pgmap v571: 158 pgs: 158 active+clean; 7.0 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T04:01:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-26T04:01:21.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:21.212 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:21.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:21.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:21.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:21.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:21.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:21.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:21.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.610 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:21.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:21.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:21.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:21.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:21.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:21.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:21.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:21.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:22.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:22.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 4.18 scrub starts 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 4.18 scrub ok 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 9.f scrub starts 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 9.b scrub starts 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T04:01:22.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[29446]: 9.1 scrub ok 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 4.18 scrub starts 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 4.18 scrub ok 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 9.f scrub starts 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 9.b scrub starts 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T04:01:22.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:21 smithi006 ceph-mon[37501]: 9.1 scrub ok 2024-03-26T04:01:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 4.18 scrub starts 2024-03-26T04:01:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 4.18 scrub ok 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 9.f scrub starts 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 9.b scrub starts 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/145591521' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: osdmap e289: 8 total, 8 up, 7 in 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T04:01:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:21 smithi086 ceph-mon[32965]: 9.1 scrub ok 2024-03-26T04:01:22.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:22.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:22.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:22.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:22.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:22.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:22.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:22.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:22.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:22.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:22.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:22.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:22.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:22.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:22.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.850 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:22.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[37501]: pgmap v573: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[29446]: pgmap v573: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T04:01:22.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:22 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T04:01:22.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:22.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:22.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:23.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:22 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T04:01:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:22 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T04:01:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:22 smithi086 ceph-mon[32965]: pgmap v573: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:22 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T04:01:23.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:22 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T04:01:23.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:23.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:23.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:23.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:23.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:23.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:23.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:23.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:23.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:23.697 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:23.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:23.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:23.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:23.931 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:23.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:23.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:24.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:23 smithi086 ceph-mon[32965]: 9.f scrub ok 2024-03-26T04:01:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:23 smithi086 ceph-mon[32965]: 9.9 scrub starts 2024-03-26T04:01:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:23 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T04:01:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:23 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T04:01:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:23 smithi086 ceph-mon[32965]: 9.b scrub ok 2024-03-26T04:01:24.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[37501]: 9.f scrub ok 2024-03-26T04:01:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[37501]: 9.9 scrub starts 2024-03-26T04:01:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T04:01:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T04:01:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[37501]: 9.b scrub ok 2024-03-26T04:01:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[29446]: 9.f scrub ok 2024-03-26T04:01:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[29446]: 9.9 scrub starts 2024-03-26T04:01:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T04:01:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T04:01:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:23 smithi006 ceph-mon[29446]: 9.b scrub ok 2024-03-26T04:01:24.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:24.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:24.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:24.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.402 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:24.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:24.499 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:24.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:24.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:24.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:24.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:24.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:24.759 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: pgmap v574: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[29446]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T04:01:24.760 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: pgmap v574: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:24.761 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:24 smithi006 ceph-mon[37501]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T04:01:24.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:24.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:24.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:24.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:24.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:25.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:25.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: pgmap v574: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.7 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:24 smithi086 ceph-mon[32965]: osdmap e290: 8 total, 8 up, 7 in 2024-03-26T04:01:25.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:25.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:25.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:25.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:25.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:25.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:25.382 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:25.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:25.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:25.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:25.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:25.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:25.816 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179824 8.d 8:ba5e2828:::benchmark_data_smithi006_103169_object49893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.757884+0000", 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.088418176, 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18686103100000001, 2024-03-26T04:01:25.849 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179824 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757884+0000", 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757884+0000", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757885+0000", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3969999999999999e-06 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757891+0000", 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-03-26T04:01:25.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757903+0000", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1813e-05 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.757910+0000", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7330000000000003e-06 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764628+0000", 2024-03-26T04:01:25.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067178130000000004 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764765+0000", 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013682399999999999 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.764906+0000", 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000140879 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773926+0000", 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0090205979999999995 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773974+0000", 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7148999999999999e-05 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.777029+0000", 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0030555439999999999 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.777123+0000", 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3794000000000004e-05 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944745+0000", 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16762194899999999 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179841 8.d 8:b003b8a9:::benchmark_data_smithi006_103169_object49910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.772355+0000", 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.07394659900001, 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17670936200000001, 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.856 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179841 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772355+0000", 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772355+0000", 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772357+0000", 2024-03-26T04:01:25.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.133e-06 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772361+0000", 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8389999999999997e-06 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772371+0000", 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3880000000000006e-06 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772378+0000", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3629999999999996e-06 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774017+0000", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016384349999999999 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774142+0000", 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012499599999999999 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.774277+0000", 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013563299999999999 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948148+0000", 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17387116 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948164+0000", 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5752000000000001e-05 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948200+0000", 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5728000000000001e-05 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.948305+0000", 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105346 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.949065+0000", 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00075958899999999997 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:179843 8.d 8:b91817d9:::benchmark_data_smithi006_103169_object49912:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.772830+0000", 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.07347175000001, 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18273748100000001, 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.863 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.864 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.864 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.864 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 179843 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772830+0000", 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772830+0000", 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772831+0000", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0500000000000002e-07 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772836+0000", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5460000000000002e-06 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772845+0000", 2024-03-26T04:01:25.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7270000000000002e-06 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.772852+0000", 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1219999999999997e-06 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944785+0000", 2024-03-26T04:01:25.867 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.171933004 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944910+0000", 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012439600000000001 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.945040+0000", 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012977600000000001 2024-03-26T04:01:25.868 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953269+0000", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0082293650000000006 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953284+0000", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4982e-05 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953319+0000", 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5049999999999998e-05 2024-03-26T04:01:25.869 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.953626+0000", 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00030677900000000003 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.870 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.955568+0000", 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001941929 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.871 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179839 8.6 e204/155 8:6fb0a9c5:::benchmark_data_smithi006_103169_object49908:head v 204'11031, mlcod=204'11028)", 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.773759+0000", 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.072542924, 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.173202892, 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.872 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773759+0000", 2024-03-26T04:01:25.873 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.03e-07 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773764+0000", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2799999999999997e-06 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773770+0000", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7120000000000003e-06 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.773777+0000", 2024-03-26T04:01:25.874 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5910000000000001e-06 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776407+0000", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026304940000000001 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776428+0000", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0061e-05 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.946907+0000", 2024-03-26T04:01:25.875 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17047985500000001 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.946962+0000", 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4595999999999997e-05 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.876 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179845 8.e e204/155 8:759074b4:::benchmark_data_smithi006_103169_object49914:head v 204'11148, mlcod=204'11146)", 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.776039+0000", 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.07026312100001, 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18142430700000001, 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.877 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776039+0000", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776039+0000", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776040+0000", 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7199999999999997e-07 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.878 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776046+0000", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3069999999999999e-06 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776055+0000", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3300000000000005e-06 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776062+0000", 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9879999999999998e-06 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.879 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776187+0000", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012491299999999999 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776200+0000", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2371000000000001e-05 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957379+0000", 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18117967400000001 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.880 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957463+0000", 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3851999999999996e-05 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179840 8.6 e204/155 8:695aac83:::benchmark_data_smithi006_103169_object49909:head v 204'11032, mlcod=204'11028)", 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.776299+0000", 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.07000315400001, 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18122993900000001, 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.881 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776299+0000", 2024-03-26T04:01:25.882 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9200000000000001e-07 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776305+0000", 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3569999999999997e-06 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776313+0000", 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2709999999999992e-06 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.883 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.776321+0000", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4109999999999999e-06 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943968+0000", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.167647087 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.944005+0000", 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6952000000000001e-05 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.884 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957469+0000", 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013464511 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957529+0000", 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9757999999999999e-05 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:179849 8.e e204/155 8:76272e1c:::benchmark_data_smithi006_103169_object49918:head v 204'11149, mlcod=204'11146)", 2024-03-26T04:01:25.885 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:12.779800+0000", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 253.066502316, 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.17778460700000001, 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T04:01:25.886 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779800+0000", 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9299999999999997e-07 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.887 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.779806+0000", 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5380000000000002e-06 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943759+0000", 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.16395269500000001 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.888 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.943772+0000", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3074000000000001e-05 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.947163+0000", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033908800000000002 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.947181+0000", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8664999999999998e-05 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.889 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957531+0000", 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010350114000000001 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:12.957584+0000", 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2948000000000002e-05 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180185 8.f 8:fadf5a1b:::benchmark_data_smithi006_103169_object50254:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.224359+0000", 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.62194279799999, 2024-03-26T04:01:25.890 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19796323499999999, 2024-03-26T04:01:25.892 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.892 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.892 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.892 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.892 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180185 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224359+0000", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224359+0000", 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.893 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224361+0000", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.813e-06 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224368+0000", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2470000000000002e-06 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224383+0000", 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4498000000000001e-05 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.894 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224392+0000", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3219999999999999e-06 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224458+0000", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5902999999999996e-05 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224792+0000", 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033372499999999999 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.895 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224918+0000", 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012586699999999999 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.896 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228128+0000", 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032105810000000001 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229340+0000", 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001211953 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.897 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.420194+0000", 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19085339700000001 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422132+0000", 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019380059999999999 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422322+0000", 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019092299999999999 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.898 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180178 8.b e204/155 8:de4dc8ce:::benchmark_data_smithi006_103169_object50247:head v 204'11490, mlcod=204'11487)", 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.224599+0000", 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.62170275899999, 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19294374, 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224599+0000", 2024-03-26T04:01:25.899 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224599+0000", 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224600+0000", 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5799999999999998e-07 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.900 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224608+0000", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2930000000000001e-06 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224617+0000", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0399999999999998e-06 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224625+0000", 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3769999999999998e-06 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.901 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224750+0000", 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012485799999999999 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.224769+0000", 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.897e-05 2024-03-26T04:01:25.902 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417449+0000", 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19268063999999999 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417543+0000", 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3703999999999996e-05 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.903 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180180 8.b e204/155 8:d7b21ae2:::benchmark_data_smithi006_103169_object50249:head v 204'11491, mlcod=204'11487)", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.225396+0000", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.62090563500001, 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19221780899999999, 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225396+0000", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225396+0000", 2024-03-26T04:01:25.904 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225397+0000", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1699999999999998e-07 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225402+0000", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3460000000000001e-06 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225410+0000", 2024-03-26T04:01:25.905 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5270000000000001e-06 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.225416+0000", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8919999999999997e-06 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.226972+0000", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001556125 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.226991+0000", 2024-03-26T04:01:25.906 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8947e-05 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417551+0000", 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19056042300000001 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417614+0000", 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3032000000000002e-05 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.907 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "MOSDPGPushReply(8.d 204/181 [PushReplyOp(8:b7097072:::benchmark_data_smithi006_103169_object91940:head)])", 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227013+0000", 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61928925699999, 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19041166800000001, 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.908 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227013+0000", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.58e-07 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227017+0000", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.087e-06 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.909 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227025+0000", 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9410000000000007e-06 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227033+0000", 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4279999999999999e-06 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231756+0000", 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047229999999999998 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.910 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: 9.9 scrub ok 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:25.911 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T04:01:25.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T04:01:25.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:25.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:25.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: 9.9 scrub ok 2024-03-26T04:01:25.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417424+0000", 2024-03-26T04:01:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18566885399999999 2024-03-26T04:01:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.913 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180187 8.6 e204/155 8:624ff609:::benchmark_data_smithi006_103169_object50256:head v 204'11050, mlcod=204'11047)", 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227200+0000", 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61910210100001, 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.197095462, 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.914 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227200+0000", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227200+0000", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227201+0000", 2024-03-26T04:01:25.915 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5099999999999999e-07 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227207+0000", 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3110000000000002e-06 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.916 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227214+0000", 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4479999999999997e-06 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227221+0000", 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6159999999999998e-06 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229327+0000", 2024-03-26T04:01:25.917 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002106101 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229346+0000", 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8715999999999998e-05 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.424258+0000", 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19491228199999999 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.918 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.424295+0000", 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7336999999999998e-05 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180182 8.b e204/155 8:dbfc3080:::benchmark_data_smithi006_103169_object50251:head v 204'11492, mlcod=204'11487)", 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227311+0000", 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61899080399999, 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19036519299999999, 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.919 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227311+0000", 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227311+0000", 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.920 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227312+0000", 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5400000000000001e-07 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227317+0000", 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.434e-06 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.921 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227325+0000", 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0299999999999994e-06 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227331+0000", 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1940000000000003e-06 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.922 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229191+0000", 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018592000000000001 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229211+0000", 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0041e-05 2024-03-26T04:01:25.923 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417618+0000", 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18840707200000001 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417676+0000", 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8668000000000001e-05 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.924 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180189 8.d 8:ba34b65e:::benchmark_data_smithi006_103169_object50258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.227480+0000", 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.618821833, 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19577961699999999, 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180189 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.925 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227480+0000", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227480+0000", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227482+0000", 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5290000000000001e-06 2024-03-26T04:01:25.926 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227488+0000", 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9059999999999999e-06 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227497+0000", 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.132e-06 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.927 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.227504+0000", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5789999999999998e-06 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.228870+0000", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00136516 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229009+0000", 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013946799999999999 2024-03-26T04:01:25.928 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229150+0000", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014095899999999999 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422212+0000", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19306240099999999 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422258+0000", 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5677000000000003e-05 2024-03-26T04:01:25.929 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422312+0000", 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4200999999999998e-05 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422404+0000", 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1336999999999996e-05 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.930 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.423260+0000", 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00085626799999999998 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180192 8.b e204/155 8:d0d6cac1:::benchmark_data_smithi006_103169_object50261:head v 204'11493, mlcod=204'11488)", 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229244+0000", 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61705784700001, 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18849374599999999, 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.931 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229244+0000", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229244+0000", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229245+0000", 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.68e-07 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.932 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229250+0000", 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4960000000000004e-06 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.933 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229258+0000", 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4420000000000001e-06 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229264+0000", 2024-03-26T04:01:25.934 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6510000000000004e-06 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231808+0000", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025432200000000001 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231827+0000", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9038e-05 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417679+0000", 2024-03-26T04:01:25.935 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.185852506 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417738+0000", 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8825e-05 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180188 8.e e204/155 8:76527e2e:::benchmark_data_smithi006_103169_object50257:head v 204'11169, mlcod=204'11167)", 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229725+0000", 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61657708499999, 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20416622300000001, 2024-03-26T04:01:25.936 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.937 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229725+0000", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.44e-07 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229731+0000", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3970000000000002e-06 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229739+0000", 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9780000000000005e-06 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.938 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229745+0000", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3080000000000004e-06 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.230445+0000", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069970999999999998 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.230464+0000", 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8643000000000001e-05 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.939 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433736+0000", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.20327236200000001 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433891+0000", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015528100000000001 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15039.0:180193 8.6 e204/155 8:64b9ce04:::benchmark_data_smithi006_103169_object50262:head v 204'11051, mlcod=204'11047)", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.229761+0000", 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61654111499999, 2024-03-26T04:01:25.940 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.204196553, 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.941 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229761+0000", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7000000000000001e-07 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229765+0000", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.472e-06 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229770+0000", 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9180000000000002e-06 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.942 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.229774+0000", 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5499999999999996e-06 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.943 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.944 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232272+0000", 2024-03-26T04:01:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024981080000000002 2024-03-26T04:01:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.945 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232291+0000", 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8461000000000002e-05 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433897+0000", 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.201606272 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.946 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.433957+0000", 2024-03-26T04:01:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0501999999999998e-05 2024-03-26T04:01:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.947 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:180194 8.f 8:fd5e33ac:::benchmark_data_smithi006_103169_object50263:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.231221+0000", 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61508093099999, 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.19125515100000001, 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:01:25.950 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 180194 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231221+0000", 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231221+0000", 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231222+0000", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3960000000000001e-06 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231229+0000", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2829999999999998e-06 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231240+0000", 2024-03-26T04:01:25.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1445e-05 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231248+0000", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0299999999999994e-06 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231306+0000", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7861e-05 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231423+0000", 2024-03-26T04:01:25.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116883 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.231541+0000", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118492 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417331+0000", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18578977699999999 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417374+0000", 2024-03-26T04:01:25.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2775999999999998e-05 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422378+0000", 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050036409999999996 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:01:25.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422433+0000", 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5676e-05 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422476+0000", 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2891e-05 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:180189 8.d e204/181 ondisk, result = 0)", 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.232503+0000", 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.61379907099999, 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18976958399999999, 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232503+0000", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232503+0000", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232504+0000", 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0100000000000004e-07 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.958 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.232508+0000", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0980000000000004e-06 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417440+0000", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18493225599999999 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417450+0000", 2024-03-26T04:01:25.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0380999999999999e-05 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422242+0000", 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047916160000000003 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422251+0000", 2024-03-26T04:01:25.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8219999999999996e-06 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.422273+0000", 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1710000000000001e-05 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15039.0:180194 8.f e204/155 ondisk, result = 0)", 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:13.234408+0000", 2024-03-26T04:01:25.961 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 252.611893788, 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18297554299999999, 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234408+0000", 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234408+0000", 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:01:25.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234409+0000", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0500000000000003e-07 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.234413+0000", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8739999999999997e-06 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.416658+0000", 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18224552899999999 2024-03-26T04:01:25.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.416672+0000", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3723000000000001e-05 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417358+0000", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00068599199999999996 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417367+0000", 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4159999999999993e-06 2024-03-26T04:01:25.964 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:13.417384+0000", 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6303999999999998e-05 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:01:25.965 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:01:25.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:25.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:25.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T04:01:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T04:01:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: 9.9 scrub ok 2024-03-26T04:01:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:26.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:26.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:26.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:26.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:26.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:26.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:26.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:26.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.425 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:26.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:26.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:26.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:26.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:26.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:26.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:26.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:26.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:26.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:26.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:26.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:26.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:26.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: pgmap v576: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-26T04:01:27.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[37501]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: pgmap v576: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-26T04:01:27.032 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:26 smithi006 ceph-mon[29446]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T04:01:27.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T04:01:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T04:01:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T04:01:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T04:01:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T04:01:27.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T04:01:27.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: pgmap v576: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-26T04:01:27.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:26 smithi086 ceph-mon[32965]: osdmap e291: 8 total, 8 up, 7 in 2024-03-26T04:01:27.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:27.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:27.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:27.252 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:27.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:27.396 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:27.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:27.501 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:27.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:27.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:27.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:27.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:27.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:27.751 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:01:27.768 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T04:01:27.768 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T04:01:27.768 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:01:27.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T04:01:27.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T04:01:27.769 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:27 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:01:27.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:27.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:27.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T04:01:27.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:27.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:27.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:27.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:28.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:27 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T04:01:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:27 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T04:01:28.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:27 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:01:28.103 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 7, 3] out_osds: [6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:01:28.103 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:01:28.103 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 4, 5, 7, 3] 2024-03-26T04:01:28.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd out 5 2024-03-26T04:01:28.115 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:28.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:28.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:28.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:28.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:28.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:28.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:28.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:28.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:28.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:28.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:28.686 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:01:28.732 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:01:28.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:28.781 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:01:28.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:28.829 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:01:28.867 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:01:28.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:28.910 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:01:28.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:28.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:28.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:28.952 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:01:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: pgmap v578: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 757 B/s rd, 0 B/s wr, 859 op/s 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:01:28.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/636736583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:29.000 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:01:29.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T04:01:29.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T04:01:29.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: pgmap v578: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 757 B/s rd, 0 B/s wr, 859 op/s 2024-03-26T04:01:29.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:01:29.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/636736583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: pgmap v578: 158 pgs: 158 active+clean; 6.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 757 B/s rd, 0 B/s wr, 859 op/s 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3318999611' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: osdmap e292: 8 total, 8 up, 7 in 2024-03-26T04:01:29.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:01:29.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:01:29.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/636736583' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:01:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:01:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:01:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:01:29.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:29.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.102 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:29.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:29.240 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:29.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:29.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:29.436 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:29.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:29.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:29.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:29.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.671 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:29.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:29.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:29 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T04:01:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:29 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T04:01:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:29 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T04:01:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:29 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T04:01:29.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:29.953 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:29.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:29.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:30.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:29 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T04:01:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:29 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T04:01:30.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:30.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:30.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:30.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:30.365 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:30.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:30.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:30.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:30.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:30.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:30.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[37501]: 5.c deep-scrub starts 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[37501]: 5.c deep-scrub ok 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[37501]: pgmap v580: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[29446]: 5.c deep-scrub starts 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[29446]: 5.c deep-scrub ok 2024-03-26T04:01:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:30 smithi006 ceph-mon[29446]: pgmap v580: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T04:01:30.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:30.914 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:30.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:31.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:01:31.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:31.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:30 smithi086 ceph-mon[32965]: 5.c deep-scrub starts 2024-03-26T04:01:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:30 smithi086 ceph-mon[32965]: 5.c deep-scrub ok 2024-03-26T04:01:31.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:30 smithi086 ceph-mon[32965]: pgmap v580: 158 pgs: 158 active+clean; 6.2 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-26T04:01:31.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:31.205 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:31.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:31.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:31.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:31.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:31.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:31.487 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.488 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:31.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:31.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:31.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:31.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:31.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:31.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:31.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:01:31.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:32.018 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:31 smithi006 ceph-mon[29446]: 9.7 scrub starts 2024-03-26T04:01:32.019 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:31 smithi006 ceph-mon[37501]: 9.7 scrub starts 2024-03-26T04:01:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:31 smithi086 ceph-mon[32965]: 9.7 scrub starts 2024-03-26T04:01:32.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:32.214 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:32.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:32.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:32.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:32.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:32.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:32.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:32.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:32.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:32.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:32.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:32.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:32.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:32.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:32.934 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:32.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:33.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:33.080 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:33.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:32 smithi086 ceph-mon[32965]: pgmap v581: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-26T04:01:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:32 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T04:01:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:32 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T04:01:33.128 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[29446]: pgmap v581: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-26T04:01:33.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T04:01:33.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T04:01:33.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[37501]: pgmap v581: 158 pgs: 158 active+clean; 6.0 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.54k op/s 2024-03-26T04:01:33.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T04:01:33.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:32 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T04:01:33.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:33.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:33.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:33.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:33.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:33.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:33.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:33.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:33.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:33.623 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:33.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:33.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:33.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:33.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:33.908 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:33.930 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:34.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:34.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:34.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[29446]: 9.7 scrub ok 2024-03-26T04:01:34.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T04:01:34.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T04:01:34.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[29446]: 6.12 deep-scrub starts 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[29446]: 6.12 deep-scrub ok 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[37501]: 9.7 scrub ok 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[37501]: 6.12 deep-scrub starts 2024-03-26T04:01:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:33 smithi006 ceph-mon[37501]: 6.12 deep-scrub ok 2024-03-26T04:01:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:33 smithi086 ceph-mon[32965]: 9.7 scrub ok 2024-03-26T04:01:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:33 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T04:01:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:33 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T04:01:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:33 smithi086 ceph-mon[32965]: 6.12 deep-scrub starts 2024-03-26T04:01:34.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:33 smithi086 ceph-mon[32965]: 6.12 deep-scrub ok 2024-03-26T04:01:34.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:34.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:34.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:34.278 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:01:34.278 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:01:34.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.310 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:34.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:34.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:34.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.443 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:34.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:34.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:34.573 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:34.574 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:01:34.462+0000 7fd465f8b700 -1 mon.a@0(leader).osd e292 definitely_dead 0 2024-03-26T04:01:34.593 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:34.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:34.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:34.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:34.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:34.786 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.5. 2024-03-26T04:01:34.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:34.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:34.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: pgmap v582: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2211063834' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[37501]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T04:01:34.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: pgmap v582: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T04:01:34.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:01:34.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:01:34.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2211063834' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:34.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:34.884 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:34 smithi006 ceph-mon[29446]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T04:01:34.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:34.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:34.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:34.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:35.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:35.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: pgmap v582: 158 pgs: 158 active+clean; 5.9 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2211063834' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-26T04:01:35.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:34 smithi086 ceph-mon[32965]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-26T04:01:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:35.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:35.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.237 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:35.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:35.339 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:35.363 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:35.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:35.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:35.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:35.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:35.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:35.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:35.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:35.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:35.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:35.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:35.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:35.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T04:01:35.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.6 deep-scrub starts 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.6 deep-scrub ok 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: osdmap e293: 8 total, 8 up, 6 in 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.14 deep-scrub starts 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: 3.14 deep-scrub ok 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[37501]: osdmap e294: 8 total, 8 up, 6 in 2024-03-26T04:01:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.6 deep-scrub starts 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.6 deep-scrub ok 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: osdmap e293: 8 total, 8 up, 6 in 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.14 deep-scrub starts 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: 3.14 deep-scrub ok 2024-03-26T04:01:35.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:35 smithi006 ceph-mon[29446]: osdmap e294: 8 total, 8 up, 6 in 2024-03-26T04:01:35.980 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:36.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T04:01:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T04:01:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.6 deep-scrub starts 2024-03-26T04:01:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.6 deep-scrub ok 2024-03-26T04:01:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:01:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: osdmap e293: 8 total, 8 up, 6 in 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.14 deep-scrub starts 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: 3.14 deep-scrub ok 2024-03-26T04:01:36.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:35 smithi086 ceph-mon[32965]: osdmap e294: 8 total, 8 up, 6 in 2024-03-26T04:01:36.107 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:36.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:36.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:36.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:36.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:36.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:36.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:36.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:36.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:36.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:36.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:36.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:36.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:36.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.766 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:36.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:36.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:36.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:36.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:36.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:36.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:37.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:37.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:36 smithi086 ceph-mon[32965]: pgmap v584: 158 pgs: 5 unknown, 30 peering, 123 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 1.50k op/s; 39 KiB/s, 0 objects/s recovering 2024-03-26T04:01:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:36 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T04:01:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:36 smithi086 ceph-mon[32965]: osdmap e295: 8 total, 8 up, 6 in 2024-03-26T04:01:37.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:37.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[37501]: pgmap v584: 158 pgs: 5 unknown, 30 peering, 123 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 1.50k op/s; 39 KiB/s, 0 objects/s recovering 2024-03-26T04:01:37.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T04:01:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[37501]: osdmap e295: 8 total, 8 up, 6 in 2024-03-26T04:01:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[29446]: pgmap v584: 158 pgs: 5 unknown, 30 peering, 123 active+clean; 5.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 1.50k op/s; 39 KiB/s, 0 objects/s recovering 2024-03-26T04:01:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 9 pgs inactive, 9 pgs peering (PG_AVAILABILITY) 2024-03-26T04:01:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:36 smithi006 ceph-mon[29446]: osdmap e295: 8 total, 8 up, 6 in 2024-03-26T04:01:37.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:37.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:37.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:37.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:37.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:37.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:37.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:37.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:37.573 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:37.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:37.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:37.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:37.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:37.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:37.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:38.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T04:01:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T04:01:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.6 starting backfill to osd.0 from (0'0,0'0] MAX to 292'10927 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 292'10704 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 292'10923 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 292'11059 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 292'10886 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 292'10958 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 292'10868 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 292'10784 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 292'10720 2024-03-26T04:01:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:37 smithi086 ceph-mon[32965]: osdmap e296: 8 total, 8 up, 6 in 2024-03-26T04:01:38.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.6 starting backfill to osd.0 from (0'0,0'0] MAX to 292'10927 2024-03-26T04:01:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 292'10704 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 292'10923 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 292'11059 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 292'10886 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 292'10958 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 292'10868 2024-03-26T04:01:38.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 292'10784 2024-03-26T04:01:38.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 292'10720 2024-03-26T04:01:38.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[37501]: osdmap e296: 8 total, 8 up, 6 in 2024-03-26T04:01:38.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T04:01:38.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T04:01:38.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T04:01:38.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T04:01:38.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.6 starting backfill to osd.0 from (0'0,0'0] MAX to 292'10927 2024-03-26T04:01:38.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.a starting backfill to osd.3 from (0'0,0'0] MAX to 292'10704 2024-03-26T04:01:38.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.f starting backfill to osd.2 from (0'0,0'0] MAX to 292'10923 2024-03-26T04:01:38.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.7 starting backfill to osd.2 from (0'0,0'0] MAX to 292'11059 2024-03-26T04:01:38.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.b starting backfill to osd.1 from (0'0,0'0] MAX to 292'10886 2024-03-26T04:01:38.144 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.5 starting backfill to osd.1 from (0'0,0'0] MAX to 292'10958 2024-03-26T04:01:38.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.d starting backfill to osd.4 from (0'0,0'0] MAX to 292'10868 2024-03-26T04:01:38.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.8 starting backfill to osd.7 from (0'0,0'0] MAX to 292'10784 2024-03-26T04:01:38.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: 9.9 starting backfill to osd.3 from (0'0,0'0] MAX to 292'10720 2024-03-26T04:01:38.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:37 smithi006 ceph-mon[29446]: osdmap e296: 8 total, 8 up, 6 in 2024-03-26T04:01:38.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:38.266 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:38.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:38.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.400 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:38.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:38.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:38.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:38.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:38.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:38.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:38.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:38 smithi006 ceph-mon[37501]: pgmap v587: 158 pgs: 7 unknown, 45 peering, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 1.07k op/s; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:01:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:38 smithi006 ceph-mon[37501]: osdmap e297: 8 total, 8 up, 6 in 2024-03-26T04:01:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:38 smithi006 ceph-mon[29446]: pgmap v587: 158 pgs: 7 unknown, 45 peering, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 1.07k op/s; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:01:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:38 smithi006 ceph-mon[29446]: osdmap e297: 8 total, 8 up, 6 in 2024-03-26T04:01:38.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:38.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:38.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:39.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:39.045 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:38 smithi086 ceph-mon[32965]: pgmap v587: 158 pgs: 7 unknown, 45 peering, 106 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 1.07k op/s; 52 KiB/s, 0 objects/s recovering 2024-03-26T04:01:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:38 smithi086 ceph-mon[32965]: osdmap e297: 8 total, 8 up, 6 in 2024-03-26T04:01:39.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:39.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:39.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:39.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:39.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.321 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:39.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:39.422 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:39.463 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:39.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:39.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:39.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:39.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:39.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:39.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:39.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:39.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:39.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:39.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:40.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:39 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:39 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:39 smithi086 ceph-mon[32965]: osdmap e298: 8 total, 8 up, 6 in 2024-03-26T04:01:40.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:40.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[37501]: osdmap e298: 8 total, 8 up, 6 in 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:39 smithi006 ceph-mon[29446]: osdmap e298: 8 total, 8 up, 6 in 2024-03-26T04:01:40.212 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:40.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:40.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:40.402 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:40.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:40.532 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:40.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:40.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:40.629 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7, 3] out_osds: [6, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:01:40.630 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:01:40.630 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T04:01:40.630 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T04:01:40.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:01:40.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:40.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:40.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:40.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:40.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:40.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:41.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:40 smithi086 ceph-mon[32965]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 7 unknown, 45 peering, 104 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 835 B/s rd, 0 B/s wr, 172 op/s; 9943/274425 objects misplaced (3.623%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:01:41.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.140 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:41.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:40 smithi006 ceph-mon[37501]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 7 unknown, 45 peering, 104 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 835 B/s rd, 0 B/s wr, 172 op/s; 9943/274425 objects misplaced (3.623%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:01:41.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:40 smithi006 ceph-mon[29446]: pgmap v591: 158 pgs: 2 active+remapped+backfilling, 7 unknown, 45 peering, 104 active+clean; 5.6 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 835 B/s rd, 0 B/s wr, 172 op/s; 9943/274425 objects misplaced (3.623%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:01:41.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:41.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:41.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:41.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:41.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:41.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:41.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:41.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:41.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:41.796 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:41.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:41.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:41.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:41.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:41.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:42.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:42.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:41 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:01:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:41 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-03-26T04:01:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:41 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:01:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:41 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-03-26T04:01:42.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:42.218 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:42.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:41 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:01:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:41 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 13 pgs inactive, 15 pgs peering) 2024-03-26T04:01:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:42.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:42.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:42.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:42.630 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:42.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.757 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:42.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:42.898 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:42.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:42.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:43.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:43.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:43.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:43.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:42 smithi006 ceph-mon[37501]: pgmap v592: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 6.4 KiB/s rd, 47 KiB/s wr, 548 op/s; 47905/268746 objects misplaced (17.825%); 34 MiB/s, 558 objects/s recovering 2024-03-26T04:01:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:42 smithi006 ceph-mon[29446]: pgmap v592: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 6.4 KiB/s rd, 47 KiB/s wr, 548 op/s; 47905/268746 objects misplaced (17.825%); 34 MiB/s, 558 objects/s recovering 2024-03-26T04:01:43.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:43.174 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:43.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:43.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:43.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:43.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:42 smithi086 ceph-mon[32965]: pgmap v592: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 6.4 KiB/s rd, 47 KiB/s wr, 548 op/s; 47905/268746 objects misplaced (17.825%); 34 MiB/s, 558 objects/s recovering 2024-03-26T04:01:43.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:43.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:43.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:43.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:43.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:43.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:43.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:43.866 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:43.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:43.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:44.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:44.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:44.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:44.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:43 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:44.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:43 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:43 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:43 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:44.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:44.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:44.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:43 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:44.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:43 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:44.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:44.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:44.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.398 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:44.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:44.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:44.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:44.642 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.665 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:44.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:44.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:44.806 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:44.908 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:44.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:44.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:45.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.127 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:45.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:45.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[37501]: pgmap v593: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 427 op/s; 47905/268746 objects misplaced (17.825%); 26 MiB/s, 434 objects/s recovering 2024-03-26T04:01:45.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[37501]: osdmap e299: 8 total, 8 up, 6 in 2024-03-26T04:01:45.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[37501]: osdmap e300: 8 total, 8 up, 6 in 2024-03-26T04:01:45.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[29446]: pgmap v593: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 427 op/s; 47905/268746 objects misplaced (17.825%); 26 MiB/s, 434 objects/s recovering 2024-03-26T04:01:45.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[29446]: osdmap e299: 8 total, 8 up, 6 in 2024-03-26T04:01:45.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:44 smithi006 ceph-mon[29446]: osdmap e300: 8 total, 8 up, 6 in 2024-03-26T04:01:45.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:45.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:45.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:44 smithi086 ceph-mon[32965]: pgmap v593: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.5 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.0 KiB/s rd, 37 KiB/s wr, 427 op/s; 47905/268746 objects misplaced (17.825%); 26 MiB/s, 434 objects/s recovering 2024-03-26T04:01:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:44 smithi086 ceph-mon[32965]: osdmap e299: 8 total, 8 up, 6 in 2024-03-26T04:01:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:44 smithi086 ceph-mon[32965]: osdmap e300: 8 total, 8 up, 6 in 2024-03-26T04:01:45.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:45.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:45.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:45.526 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:45.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:45.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:45.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:45.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T04:01:45.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:45.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:45.898 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:45.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:45.972 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:45.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:46.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:45 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:46.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:45 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:46.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:46.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:46.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:46.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:46.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:46.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:45 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:46.381 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:46.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:46.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:46.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:46.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:46.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:46.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:46.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:46.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:46.877 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:46.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:46.920 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:01:46.959 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:46.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:46.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:46.983 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[29446]: pgmap v596: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.8 KiB/s rd, 36 KiB/s wr, 802 op/s; 43472/261906 objects misplaced (16.598%); 63 MiB/s, 1.02k objects/s recovering 2024-03-26T04:01:46.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/730967109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:46.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:46.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[37501]: pgmap v596: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.8 KiB/s rd, 36 KiB/s wr, 802 op/s; 43472/261906 objects misplaced (16.598%); 63 MiB/s, 1.02k objects/s recovering 2024-03-26T04:01:46.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/730967109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:46.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:46 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:47.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:47.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:47.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:47.266 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:46 smithi086 ceph-mon[32965]: pgmap v596: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.3 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 5.8 KiB/s rd, 36 KiB/s wr, 802 op/s; 43472/261906 objects misplaced (16.598%); 63 MiB/s, 1.02k objects/s recovering 2024-03-26T04:01:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/730967109' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:47.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:46 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:01:47.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:47.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:47.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:47.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:47.547 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:47.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:47.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:47.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:47.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:47.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:47.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:47.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:47.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:47.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:48.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:48.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:48.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:48.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:48.200 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:48.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:01:48.201 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[37501]: osdmap e301: 8 total, 8 up, 6 in 2024-03-26T04:01:48.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:48.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:01:48.201 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:47 smithi006 ceph-mon[29446]: osdmap e301: 8 total, 8 up, 6 in 2024-03-26T04:01:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:48.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:48.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:47 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:01:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:01:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:47 smithi086 ceph-mon[32965]: osdmap e301: 8 total, 8 up, 6 in 2024-03-26T04:01:48.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:48.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:48.499 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:48.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.649 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:48.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:48.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:48.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:48.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:48.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:48.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:48.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:48.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:49.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:49.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:48 smithi006 ceph-mon[37501]: pgmap v598: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 690 op/s; 40721/256329 objects misplaced (15.886%); 52 MiB/s, 838 objects/s recovering 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1087784341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:48 smithi006 ceph-mon[29446]: pgmap v598: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 690 op/s; 40721/256329 objects misplaced (15.886%); 52 MiB/s, 838 objects/s recovering 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:49.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:49 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1087784341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:49.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:49.180 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:49.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:49.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:49.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:48 smithi086 ceph-mon[32965]: pgmap v598: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.2 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 690 op/s; 40721/256329 objects misplaced (15.886%); 52 MiB/s, 838 objects/s recovering 2024-03-26T04:01:49.339 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:48 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:49.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:48 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:49.340 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:48 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1087784341' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:01:49.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:49.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:49.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:49.581 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:49.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:49.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:49.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:49.848 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:49.950 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:01:49.958 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:01:49.959 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:01:49.993 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:49.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:49.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:50.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:01:50.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:50.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:50.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:50.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: osdmap e302: 8 total, 8 up, 6 in 2024-03-26T04:01:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:50 smithi086 ceph-mon[32965]: pgmap v600: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 957 op/s; 37837/252771 objects misplaced (14.969%); 86 MiB/s, 1.37k objects/s recovering 2024-03-26T04:01:50.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:50.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:50.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:50.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: osdmap e302: 8 total, 8 up, 6 in 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[37501]: pgmap v600: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 957 op/s; 37837/252771 objects misplaced (14.969%); 86 MiB/s, 1.37k objects/s recovering 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: osdmap e302: 8 total, 8 up, 6 in 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:50.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:50 smithi006 ceph-mon[29446]: pgmap v600: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 5.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 957 op/s; 37837/252771 objects misplaced (14.969%); 86 MiB/s, 1.37k objects/s recovering 2024-03-26T04:01:50.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:50.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:50.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.638 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:50.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:50.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:50.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:50.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:50.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:51.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:51.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:51.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:51.159 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:51.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:51.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:51.330 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:51.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:51.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[37501]: osdmap e303: 8 total, 8 up, 6 in 2024-03-26T04:01:51.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:01:51.461 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:01:51.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[29446]: osdmap e303: 8 total, 8 up, 6 in 2024-03-26T04:01:51.461 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:01:51.462 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:51 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:01:51.478 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:51.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:51.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:51 smithi086 ceph-mon[32965]: osdmap e303: 8 total, 8 up, 6 in 2024-03-26T04:01:51.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:51 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:01:51.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:51 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:01:51.620 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:51.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:51.664 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:51.722 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:51.779 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:51.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:51.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:51.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:51.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:52.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:52.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:52.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:52.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:52.185 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:52.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:52.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:52.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:52.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:52.545 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:52 smithi006 ceph-mon[29446]: osdmap e304: 8 total, 8 up, 6 in 2024-03-26T04:01:52.545 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:52 smithi006 ceph-mon[29446]: pgmap v603: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 34105/242925 objects misplaced (14.039%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T04:01:52.545 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:52 smithi006 ceph-mon[37501]: osdmap e304: 8 total, 8 up, 6 in 2024-03-26T04:01:52.545 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:52 smithi006 ceph-mon[37501]: pgmap v603: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 34105/242925 objects misplaced (14.039%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T04:01:52.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:52.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:52 smithi086 ceph-mon[32965]: osdmap e304: 8 total, 8 up, 6 in 2024-03-26T04:01:52.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:52 smithi086 ceph-mon[32965]: pgmap v603: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 982 op/s; 34105/242925 objects misplaced (14.039%); 64 MiB/s, 1.03k objects/s recovering 2024-03-26T04:01:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.594 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:52.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:52.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:52.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:52.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:52.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:52.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:52.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:52.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:53.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:53.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:53.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:53.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:53.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:53.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:53.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:53.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:53.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:53.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:53.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:53.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:53.721 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:53.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:53.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:53.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:53.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:53.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:53.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:54.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:54.127 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:54.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:54.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:54.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:54.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:01:54.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:54.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:54.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[37501]: pgmap v604: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s; 34105/242925 objects misplaced (14.039%); 49 MiB/s, 778 objects/s recovering 2024-03-26T04:01:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:54.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:54.641 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:01:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:01:54.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[29446]: pgmap v604: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s; 34105/242925 objects misplaced (14.039%); 49 MiB/s, 778 objects/s recovering 2024-03-26T04:01:54.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:54.642 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:54.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:54.669 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:01:54.691 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:01:54.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:54.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.814 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:54.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:54 smithi086 ceph-mon[32965]: pgmap v604: 158 pgs: 2 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 149 active+clean; 4.9 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 745 op/s; 34105/242925 objects misplaced (14.039%); 49 MiB/s, 778 objects/s recovering 2024-03-26T04:01:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:01:54.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:01:54.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:54.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:54.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:55.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:55.085 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:55.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:55.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:55.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:55.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:55.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:55.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:01:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:01:55.374 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:55.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:55.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:55.516 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:55.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:55.618 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:55.630 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:55 smithi006 ceph-mon[29446]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:55.630 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:55 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:55.633 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:55 smithi006 ceph-mon[37501]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:55.633 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:55 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:55.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:55.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:55.686 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2024-03-26T04:01:53.457751+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5292133521,"num_objects":80975,"num_object_clones":0,"num_object_copies":242925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34105,"num_objects_unfound":0,"num_objects_dirty":80975,"num_whiteouts":0,"num_read":2260,"num_read_kb":2006,"num_write":185128,"num_write_kb":8477760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12606,"num_bytes_recovered":802588025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62816,"ondisk_log_size":62816,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":16165196,"kb_used_data":13366660,"kb_used_omap":9,"kb_used_meta":2798390,"kb_avail":546305716,"statfs":{"total":575970213888,"available":559417053184,"internally_reserved":0,"allocated":13687459840,"data_stored":13683632603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":2865552037},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,1,1,1,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-292814848,"num_objects":-4468,"num_object_clones":0,"num_object_copies":-13404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6616,"num_objects_unfound":0,"num_objects_dirty":-4468,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4468,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4674,"num_bytes_recovered":306315264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001761"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":523,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041769+0000","last_change":"2024-03-26T04:01:35.798275+0000","last_active":"2024-03-26T04:01:49.041769+0000","last_peered":"2024-03-26T04:01:49.041769+0000","last_clean":"2024-03-26T04:01:49.041769+0000","last_became_active":"2024-03-26T04:01:35.797821+0000","last_became_peered":"2024-03-26T04:01:35.797821+0000","last_unstale":"2024-03-26T04:01:49.041769+0000","last_undegraded":"2024-03-26T04:01:49.041769+0000","last_fullsized":"2024-03-26T04:01:49.041769+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:28.210127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:28.210127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:53.740436+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":527,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190284+0000","last_change":"2024-03-26T04:01:35.794331+0000","last_active":"2024-03-26T04:01:50.190284+0000","last_peered":"2024-03-26T04:01:50.190284+0000","last_clean":"2024-03-26T04:01:50.190284+0000","last_became_active":"2024-03-26T04:01:35.793522+0000","last_became_peered":"2024-03-26T04:01:35.793522+0000","last_unstale":"2024-03-26T04:01:50.190284+0000","last_undegraded":"2024-03-26T04:01:50.190284+0000","last_fullsized":"2024-03-26T04:01:50.190284+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:33.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.560099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":538,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197977+0000","last_change":"2024-03-26T04:01:35.812533+0000","last_active":"2024-03-26T04:01:50.197977+0000","last_peered":"2024-03-26T04:01:50.197977+0000","last_clean":"2024-03-26T04:01:50.197977+0000","last_became_active":"2024-03-26T04:01:35.812150+0000","last_became_peered":"2024-03-26T04:01:35.812150+0000","last_unstale":"2024-03-26T04:01:50.197977+0000","last_undegraded":"2024-03-26T04:01:50.197977+0000","last_fullsized":"2024-03-26T04:01:50.197977+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:07.415513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:07.415513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":607,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041466+0000","last_change":"2024-03-26T04:01:20.107349+0000","last_active":"2024-03-26T04:01:49.041466+0000","last_peered":"2024-03-26T04:01:49.041466+0000","last_clean":"2024-03-26T04:01:49.041466+0000","last_became_active":"2024-03-26T03:57:49.359916+0000","last_became_peered":"2024-03-26T03:57:49.359916+0000","last_unstale":"2024-03-26T04:01:49.041466+0000","last_undegraded":"2024-03-26T04:01:49.041466+0000","last_fullsized":"2024-03-26T04:01:49.041466+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:01:20.107206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.107206+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:21.791954+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190090+0000","last_change":"2024-03-26T04:00:26.733786+0000","last_active":"2024-03-26T04:01:50.190090+0000","last_peered":"2024-03-26T04:01:50.190090+0000","last_clean":"2024-03-26T04:01:50.190090+0000","last_became_active":"2024-03-26T03:57:49.377225+0000","last_became_peered":"2024-03-26T03:57:49.377225+0000","last_unstale":"2024-03-26T04:01:50.190090+0000","last_undegraded":"2024-03-26T04:01:50.190090+0000","last_fullsized":"2024-03-26T04:01:50.190090+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.733284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.733284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":518,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041416+0000","last_change":"2024-03-26T04:01:21.069930+0000","last_active":"2024-03-26T04:01:49.041416+0000","last_peered":"2024-03-26T04:01:49.041416+0000","last_clean":"2024-03-26T04:01:49.041416+0000","last_became_active":"2024-03-26T03:50:45.598225+0000","last_became_peered":"2024-03-26T03:50:45.598225+0000","last_unstale":"2024-03-26T04:01:49.041416+0000","last_undegraded":"2024-03-26T04:01:49.041416+0000","last_fullsized":"2024-03-26T04:01:49.041416+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.069804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.069804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:24.662487+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":196,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.188906+0000","last_change":"2024-03-26T04:00:32.445510+0000","last_active":"2024-03-26T04:01:50.188906+0000","last_peered":"2024-03-26T04:01:50.188906+0000","last_clean":"2024-03-26T04:01:50.188906+0000","last_became_active":"2024-03-26T03:57:49.382061+0000","last_became_peered":"2024-03-26T03:57:49.382061+0000","last_unstale":"2024-03-26T04:01:50.188906+0000","last_undegraded":"2024-03-26T04:01:50.188906+0000","last_fullsized":"2024-03-26T04:01:50.188906+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:00:32.445390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":243,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.057430+0000","last_change":"2024-03-26T04:01:20.776481+0000","last_active":"2024-03-26T04:01:49.057430+0000","last_peered":"2024-03-26T04:01:49.057430+0000","last_clean":"2024-03-26T04:01:49.057430+0000","last_became_active":"2024-03-26T03:57:49.380257+0000","last_became_peered":"2024-03-26T03:57:49.380257+0000","last_unstale":"2024-03-26T04:01:49.057430+0000","last_undegraded":"2024-03-26T04:01:49.057430+0000","last_fullsized":"2024-03-26T04:01:49.057430+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:01:20.776205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.776205+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:50.280179+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":79,"num_read_kb":66,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":510,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.200845+0000","last_change":"2024-03-26T04:00:26.435067+0000","last_active":"2024-03-26T04:01:50.200845+0000","last_peered":"2024-03-26T04:01:50.200845+0000","last_clean":"2024-03-26T04:01:50.200845+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:01:50.200845+0000","last_undegraded":"2024-03-26T04:01:50.200845+0000","last_fullsized":"2024-03-26T04:01:50.200845+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.434899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.434899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":271,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197426+0000","last_change":"2024-03-26T04:01:35.803788+0000","last_active":"2024-03-26T04:01:50.197426+0000","last_peered":"2024-03-26T04:01:50.197426+0000","last_clean":"2024-03-26T04:01:50.197426+0000","last_became_active":"2024-03-26T04:01:35.802827+0000","last_became_peered":"2024-03-26T04:01:35.802827+0000","last_unstale":"2024-03-26T04:01:50.197426+0000","last_undegraded":"2024-03-26T04:01:50.197426+0000","last_fullsized":"2024-03-26T04:01:50.197426+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:00:25.328566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":219,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043064+0000","last_change":"2024-03-26T04:01:35.797022+0000","last_active":"2024-03-26T04:01:49.043064+0000","last_peered":"2024-03-26T04:01:49.043064+0000","last_clean":"2024-03-26T04:01:49.043064+0000","last_became_active":"2024-03-26T04:01:35.796601+0000","last_became_peered":"2024-03-26T04:01:35.796601+0000","last_unstale":"2024-03-26T04:01:49.043064+0000","last_undegraded":"2024-03-26T04:01:49.043064+0000","last_fullsized":"2024-03-26T04:01:49.043064+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:16.169903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:01:16.169903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:44.191934+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":606,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041927+0000","last_change":"2024-03-26T03:59:21.358672+0000","last_active":"2024-03-26T04:01:49.041927+0000","last_peered":"2024-03-26T04:01:49.041927+0000","last_clean":"2024-03-26T04:01:49.041927+0000","last_became_active":"2024-03-26T03:57:49.360453+0000","last_became_peered":"2024-03-26T03:57:49.360453+0000","last_unstale":"2024-03-26T04:01:49.041927+0000","last_undegraded":"2024-03-26T04:01:49.041927+0000","last_fullsized":"2024-03-26T04:01:49.041927+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T03:59:21.358504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:21.358504+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":315,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.192376+0000","last_change":"2024-03-26T04:01:35.793861+0000","last_active":"2024-03-26T04:01:50.192376+0000","last_peered":"2024-03-26T04:01:50.192376+0000","last_clean":"2024-03-26T04:01:50.192376+0000","last_became_active":"2024-03-26T04:01:35.793091+0000","last_became_peered":"2024-03-26T04:01:35.793091+0000","last_unstale":"2024-03-26T04:01:50.192376+0000","last_undegraded":"2024-03-26T04:01:50.192376+0000","last_fullsized":"2024-03-26T04:01:50.192376+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.396869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.396869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046505+0000","last_change":"2024-03-26T04:01:27.830143+0000","last_active":"2024-03-26T04:01:49.046505+0000","last_peered":"2024-03-26T04:01:49.046505+0000","last_clean":"2024-03-26T04:01:49.046505+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:01:49.046505+0000","last_undegraded":"2024-03-26T04:01:49.046505+0000","last_fullsized":"2024-03-26T04:01:49.046505+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:27.830027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:01:27.830027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:49.553241+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":244,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042150+0000","last_change":"2024-03-26T04:01:35.797717+0000","last_active":"2024-03-26T04:01:49.042150+0000","last_peered":"2024-03-26T04:01:49.042150+0000","last_clean":"2024-03-26T04:01:49.042150+0000","last_became_active":"2024-03-26T04:01:35.797214+0000","last_became_peered":"2024-03-26T04:01:35.797214+0000","last_unstale":"2024-03-26T04:01:49.042150+0000","last_undegraded":"2024-03-26T04:01:49.042150+0000","last_fullsized":"2024-03-26T04:01:49.042150+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:18.326616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.326616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":572,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042334+0000","last_change":"2024-03-26T04:01:08.181397+0000","last_active":"2024-03-26T04:01:49.042334+0000","last_peered":"2024-03-26T04:01:49.042334+0000","last_clean":"2024-03-26T04:01:49.042334+0000","last_became_active":"2024-03-26T03:57:49.363679+0000","last_became_peered":"2024-03-26T03:57:49.363679+0000","last_unstale":"2024-03-26T04:01:49.042334+0000","last_undegraded":"2024-03-26T04:01:49.042334+0000","last_fullsized":"2024-03-26T04:01:49.042334+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:01:08.181298+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.181298+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:19.836563+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":542,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043100+0000","last_change":"2024-03-26T04:01:35.800531+0000","last_active":"2024-03-26T04:01:49.043100+0000","last_peered":"2024-03-26T04:01:49.043100+0000","last_clean":"2024-03-26T04:01:49.043100+0000","last_became_active":"2024-03-26T04:01:35.799946+0000","last_became_peered":"2024-03-26T04:01:35.799946+0000","last_unstale":"2024-03-26T04:01:49.043100+0000","last_undegraded":"2024-03-26T04:01:49.043100+0000","last_fullsized":"2024-03-26T04:01:49.043100+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.162521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.162521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:59.452185+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":512,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196977+0000","last_change":"2024-03-26T04:00:10.517958+0000","last_active":"2024-03-26T04:01:50.196977+0000","last_peered":"2024-03-26T04:01:50.196977+0000","last_clean":"2024-03-26T04:01:50.196977+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:01:50.196977+0000","last_undegraded":"2024-03-26T04:01:50.196977+0000","last_fullsized":"2024-03-26T04:01:50.196977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.517839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.517839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":530,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197656+0000","last_change":"2024-03-26T04:00:27.363899+0000","last_active":"2024-03-26T04:01:50.197656+0000","last_peered":"2024-03-26T04:01:50.197656+0000","last_clean":"2024-03-26T04:01:50.197656+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:01:50.197656+0000","last_undegraded":"2024-03-26T04:01:50.197656+0000","last_fullsized":"2024-03-26T04:01:50.197656+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.363567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.363567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":627,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045670+0000","last_change":"2024-03-26T04:00:12.045147+0000","last_active":"2024-03-26T04:01:49.045670+0000","last_peered":"2024-03-26T04:01:49.045670+0000","last_clean":"2024-03-26T04:01:49.045670+0000","last_became_active":"2024-03-26T03:57:49.363368+0000","last_became_peered":"2024-03-26T03:57:49.363368+0000","last_unstale":"2024-03-26T04:01:49.045670+0000","last_undegraded":"2024-03-26T04:01:49.045670+0000","last_fullsized":"2024-03-26T04:01:49.045670+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:12.044640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:12.044640+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":207,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.058174+0000","last_change":"2024-03-26T04:01:35.823180+0000","last_active":"2024-03-26T04:01:49.058174+0000","last_peered":"2024-03-26T04:01:49.058174+0000","last_clean":"2024-03-26T04:01:49.058174+0000","last_became_active":"2024-03-26T04:01:35.821619+0000","last_became_peered":"2024-03-26T04:01:35.821619+0000","last_unstale":"2024-03-26T04:01:49.058174+0000","last_undegraded":"2024-03-26T04:01:49.058174+0000","last_fullsized":"2024-03-26T04:01:49.058174+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:22.809763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.809763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:39.818657+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042494+0000","last_change":"2024-03-26T04:00:26.341422+0000","last_active":"2024-03-26T04:01:49.042494+0000","last_peered":"2024-03-26T04:01:49.042494+0000","last_clean":"2024-03-26T04:01:49.042494+0000","last_became_active":"2024-03-26T03:57:49.359728+0000","last_became_peered":"2024-03-26T03:57:49.359728+0000","last_unstale":"2024-03-26T04:01:49.042494+0000","last_undegraded":"2024-03-26T04:01:49.042494+0000","last_fullsized":"2024-03-26T04:01:49.042494+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.341297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.341297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":520,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041960+0000","last_change":"2024-03-26T03:59:23.332947+0000","last_active":"2024-03-26T04:01:49.041960+0000","last_peered":"2024-03-26T04:01:49.041960+0000","last_clean":"2024-03-26T04:01:49.041960+0000","last_became_active":"2024-03-26T03:48:44.013713+0000","last_became_peered":"2024-03-26T03:48:44.013713+0000","last_unstale":"2024-03-26T04:01:49.041960+0000","last_undegraded":"2024-03-26T04:01:49.041960+0000","last_fullsized":"2024-03-26T04:01:49.041960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:23.332861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:23.332861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":218,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189832+0000","last_change":"2024-03-26T03:59:13.198844+0000","last_active":"2024-03-26T04:01:50.189832+0000","last_peered":"2024-03-26T04:01:50.189832+0000","last_clean":"2024-03-26T04:01:50.189832+0000","last_became_active":"2024-03-26T03:57:49.366519+0000","last_became_peered":"2024-03-26T03:57:49.366519+0000","last_unstale":"2024-03-26T04:01:50.189832+0000","last_undegraded":"2024-03-26T04:01:50.189832+0000","last_fullsized":"2024-03-26T04:01:50.189832+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T03:59:13.198735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:13.198735+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":77,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045546+0000","last_change":"2024-03-26T04:01:35.822556+0000","last_active":"2024-03-26T04:01:49.045546+0000","last_peered":"2024-03-26T04:01:49.045546+0000","last_clean":"2024-03-26T04:01:49.045546+0000","last_became_active":"2024-03-26T04:01:35.805505+0000","last_became_peered":"2024-03-26T04:01:35.805505+0000","last_unstale":"2024-03-26T04:01:49.045546+0000","last_undegraded":"2024-03-26T04:01:49.045546+0000","last_fullsized":"2024-03-26T04:01:49.045546+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:00:05.475152+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:00:05.475152+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"100'9","reported_seq":619,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043122+0000","last_change":"2024-03-26T04:00:35.094959+0000","last_active":"2024-03-26T04:01:49.043122+0000","last_peered":"2024-03-26T04:01:49.043122+0000","last_clean":"2024-03-26T04:01:49.043122+0000","last_became_active":"2024-03-26T03:50:45.592681+0000","last_became_peered":"2024-03-26T03:50:45.592681+0000","last_unstale":"2024-03-26T04:01:49.043122+0000","last_undegraded":"2024-03-26T04:01:49.043122+0000","last_fullsized":"2024-03-26T04:01:49.043122+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:35.094793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.094793+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":550,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190179+0000","last_change":"2024-03-26T04:01:25.417161+0000","last_active":"2024-03-26T04:01:50.190179+0000","last_peered":"2024-03-26T04:01:50.190179+0000","last_clean":"2024-03-26T04:01:50.190179+0000","last_became_active":"2024-03-26T03:57:49.383175+0000","last_became_peered":"2024-03-26T03:57:49.383175+0000","last_unstale":"2024-03-26T04:01:50.190179+0000","last_undegraded":"2024-03-26T04:01:50.190179+0000","last_fullsized":"2024-03-26T04:01:50.190179+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:01:25.417004+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.417004+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:27.568892+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"241'26","reported_seq":601,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198468+0000","last_change":"2024-03-26T04:00:11.288905+0000","last_active":"2024-03-26T04:01:50.198468+0000","last_peered":"2024-03-26T04:01:50.198468+0000","last_clean":"2024-03-26T04:01:50.198468+0000","last_became_active":"2024-03-26T03:57:49.360466+0000","last_became_peered":"2024-03-26T03:57:49.360466+0000","last_unstale":"2024-03-26T04:01:50.198468+0000","last_undegraded":"2024-03-26T04:01:50.198468+0000","last_fullsized":"2024-03-26T04:01:50.198468+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:00:11.288774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.288774+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":523,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196513+0000","last_change":"2024-03-26T04:01:35.816885+0000","last_active":"2024-03-26T04:01:50.196513+0000","last_peered":"2024-03-26T04:01:50.196513+0000","last_clean":"2024-03-26T04:01:50.196513+0000","last_became_active":"2024-03-26T04:01:35.809445+0000","last_became_peered":"2024-03-26T04:01:35.809445+0000","last_unstale":"2024-03-26T04:01:50.196513+0000","last_undegraded":"2024-03-26T04:01:50.196513+0000","last_fullsized":"2024-03-26T04:01:50.196513+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:21.633920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.633920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":515,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198737+0000","last_change":"2024-03-26T04:00:10.317133+0000","last_active":"2024-03-26T04:01:50.198737+0000","last_peered":"2024-03-26T04:01:50.198737+0000","last_clean":"2024-03-26T04:01:50.198737+0000","last_became_active":"2024-03-26T03:48:48.020581+0000","last_became_peered":"2024-03-26T03:48:48.020581+0000","last_unstale":"2024-03-26T04:01:50.198737+0000","last_undegraded":"2024-03-26T04:01:50.198737+0000","last_fullsized":"2024-03-26T04:01:50.198737+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.317007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.317007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190852+0000","last_change":"2024-03-26T04:00:12.417129+0000","last_active":"2024-03-26T04:01:50.190852+0000","last_peered":"2024-03-26T04:01:50.190852+0000","last_clean":"2024-03-26T04:01:50.190852+0000","last_became_active":"2024-03-26T03:57:49.398303+0000","last_became_peered":"2024-03-26T03:57:49.398303+0000","last_unstale":"2024-03-26T04:01:50.190852+0000","last_undegraded":"2024-03-26T04:01:50.190852+0000","last_fullsized":"2024-03-26T04:01:50.190852+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:12.416999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:12.416999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":435,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045619+0000","last_change":"2024-03-26T04:01:36.230619+0000","last_active":"2024-03-26T04:01:49.045619+0000","last_peered":"2024-03-26T04:01:49.045619+0000","last_clean":"2024-03-26T04:01:49.045619+0000","last_became_active":"2024-03-26T04:01:35.820296+0000","last_became_peered":"2024-03-26T04:01:35.820296+0000","last_unstale":"2024-03-26T04:01:49.045619+0000","last_undegraded":"2024-03-26T04:01:49.045619+0000","last_fullsized":"2024-03-26T04:01:49.045619+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:46.925726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":524,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046601+0000","last_change":"2024-03-26T04:01:32.737948+0000","last_active":"2024-03-26T04:01:49.046601+0000","last_peered":"2024-03-26T04:01:49.046601+0000","last_clean":"2024-03-26T04:01:49.046601+0000","last_became_active":"2024-03-26T03:48:50.029657+0000","last_became_peered":"2024-03-26T03:48:50.029657+0000","last_unstale":"2024-03-26T04:01:49.046601+0000","last_undegraded":"2024-03-26T04:01:49.046601+0000","last_fullsized":"2024-03-26T04:01:49.046601+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.737823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:32.737823+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.737823+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:55.005405+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":450,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041769+0000","last_change":"2024-03-26T04:00:23.343567+0000","last_active":"2024-03-26T04:01:49.041769+0000","last_peered":"2024-03-26T04:01:49.041769+0000","last_clean":"2024-03-26T04:01:49.041769+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:01:49.041769+0000","last_undegraded":"2024-03-26T04:01:49.041769+0000","last_fullsized":"2024-03-26T04:01:49.041769+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:23.343421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.343421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:49.204689+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":549,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198202+0000","last_change":"2024-03-26T04:01:35.807303+0000","last_active":"2024-03-26T04:01:50.198202+0000","last_peered":"2024-03-26T04:01:50.198202+0000","last_clean":"2024-03-26T04:01:50.198202+0000","last_became_active":"2024-03-26T04:01:35.806963+0000","last_became_peered":"2024-03-26T04:01:35.806963+0000","last_unstale":"2024-03-26T04:01:50.198202+0000","last_undegraded":"2024-03-26T04:01:50.198202+0000","last_fullsized":"2024-03-26T04:01:50.198202+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:16.271586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:16.271586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":215,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190808+0000","last_change":"2024-03-26T04:00:38.478725+0000","last_active":"2024-03-26T04:01:50.190808+0000","last_peered":"2024-03-26T04:01:50.190808+0000","last_clean":"2024-03-26T04:01:50.190808+0000","last_became_active":"2024-03-26T03:57:49.363380+0000","last_became_peered":"2024-03-26T03:57:49.363380+0000","last_unstale":"2024-03-26T04:01:50.190808+0000","last_undegraded":"2024-03-26T04:01:50.190808+0000","last_fullsized":"2024-03-26T04:01:50.190808+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:00:38.458453+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.458453+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189984+0000","last_change":"2024-03-26T04:00:29.635294+0000","last_active":"2024-03-26T04:01:50.189984+0000","last_peered":"2024-03-26T04:01:50.189984+0000","last_clean":"2024-03-26T04:01:50.189984+0000","last_became_active":"2024-03-26T03:57:49.382620+0000","last_became_peered":"2024-03-26T03:57:49.382620+0000","last_unstale":"2024-03-26T04:01:50.189984+0000","last_undegraded":"2024-03-26T04:01:50.189984+0000","last_fullsized":"2024-03-26T04:01:50.189984+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.635002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.635002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":128,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046198+0000","last_change":"2024-03-26T04:01:33.725805+0000","last_active":"2024-03-26T04:01:49.046198+0000","last_peered":"2024-03-26T04:01:49.046198+0000","last_clean":"2024-03-26T04:01:49.046198+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:01:49.046198+0000","last_undegraded":"2024-03-26T04:01:49.046198+0000","last_fullsized":"2024-03-26T04:01:49.046198+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:33.725685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:33.725685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:38.045836+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":254,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.193428+0000","last_change":"2024-03-26T04:01:35.809409+0000","last_active":"2024-03-26T04:01:50.193428+0000","last_peered":"2024-03-26T04:01:50.193428+0000","last_clean":"2024-03-26T04:01:50.193428+0000","last_became_active":"2024-03-26T04:01:35.804476+0000","last_became_peered":"2024-03-26T04:01:35.804476+0000","last_unstale":"2024-03-26T04:01:50.193428+0000","last_undegraded":"2024-03-26T04:01:50.193428+0000","last_fullsized":"2024-03-26T04:01:50.193428+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.384487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.384487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":247,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.057571+0000","last_change":"2024-03-26T04:01:35.845345+0000","last_active":"2024-03-26T04:01:49.057571+0000","last_peered":"2024-03-26T04:01:49.057571+0000","last_clean":"2024-03-26T04:01:49.057571+0000","last_became_active":"2024-03-26T04:01:35.812701+0000","last_became_peered":"2024-03-26T04:01:35.812701+0000","last_unstale":"2024-03-26T04:01:49.057571+0000","last_undegraded":"2024-03-26T04:01:49.057571+0000","last_fullsized":"2024-03-26T04:01:49.057571+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:01:19.700471+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.700471+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:48.801117+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":525,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.199175+0000","last_change":"2024-03-26T04:01:35.817149+0000","last_active":"2024-03-26T04:01:50.199175+0000","last_peered":"2024-03-26T04:01:50.199175+0000","last_clean":"2024-03-26T04:01:50.199175+0000","last_became_active":"2024-03-26T04:01:35.816809+0000","last_became_peered":"2024-03-26T04:01:35.816809+0000","last_unstale":"2024-03-26T04:01:50.199175+0000","last_undegraded":"2024-03-26T04:01:50.199175+0000","last_fullsized":"2024-03-26T04:01:50.199175+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:09.430539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:09.430539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189898+0000","last_change":"2024-03-26T04:00:30.491018+0000","last_active":"2024-03-26T04:01:50.189898+0000","last_peered":"2024-03-26T04:01:50.189898+0000","last_clean":"2024-03-26T04:01:50.189898+0000","last_became_active":"2024-03-26T03:57:49.377134+0000","last_became_peered":"2024-03-26T03:57:49.377134+0000","last_unstale":"2024-03-26T04:01:50.189898+0000","last_undegraded":"2024-03-26T04:01:50.189898+0000","last_fullsized":"2024-03-26T04:01:50.189898+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:30.490887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:30.490887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":548,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047564+0000","last_change":"2024-03-26T04:01:34.763366+0000","last_active":"2024-03-26T04:01:49.047564+0000","last_peered":"2024-03-26T04:01:49.047564+0000","last_clean":"2024-03-26T04:01:49.047564+0000","last_became_active":"2024-03-26T03:57:49.346605+0000","last_became_peered":"2024-03-26T03:57:49.346605+0000","last_unstale":"2024-03-26T04:01:49.047564+0000","last_undegraded":"2024-03-26T04:01:49.047564+0000","last_fullsized":"2024-03-26T04:01:49.047564+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:34.763231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.763231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:56.663274+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":577,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.202470+0000","last_change":"2024-03-26T04:01:36.382207+0000","last_active":"2024-03-26T04:01:50.202470+0000","last_peered":"2024-03-26T04:01:50.202470+0000","last_clean":"2024-03-26T04:01:50.202470+0000","last_became_active":"2024-03-26T03:50:45.592128+0000","last_became_peered":"2024-03-26T03:50:45.592128+0000","last_unstale":"2024-03-26T04:01:50.202470+0000","last_undegraded":"2024-03-26T04:01:50.202470+0000","last_fullsized":"2024-03-26T04:01:50.202470+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:01:36.382121+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.382121+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:48.290652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":207,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.057340+0000","last_change":"2024-03-26T04:01:35.811070+0000","last_active":"2024-03-26T04:01:49.057340+0000","last_peered":"2024-03-26T04:01:49.057340+0000","last_clean":"2024-03-26T04:01:49.057340+0000","last_became_active":"2024-03-26T04:01:35.810507+0000","last_became_peered":"2024-03-26T04:01:35.810507+0000","last_unstale":"2024-03-26T04:01:49.057340+0000","last_undegraded":"2024-03-26T04:01:49.057340+0000","last_fullsized":"2024-03-26T04:01:49.057340+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.709457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.709457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:31.489829+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":539,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.202300+0000","last_change":"2024-03-26T04:01:35.809351+0000","last_active":"2024-03-26T04:01:50.202300+0000","last_peered":"2024-03-26T04:01:50.202300+0000","last_clean":"2024-03-26T04:01:50.202300+0000","last_became_active":"2024-03-26T04:01:35.808859+0000","last_became_peered":"2024-03-26T04:01:35.808859+0000","last_unstale":"2024-03-26T04:01:50.202300+0000","last_undegraded":"2024-03-26T04:01:50.202300+0000","last_fullsized":"2024-03-26T04:01:50.202300+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.560382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.560382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042242+0000","last_change":"2024-03-26T04:01:35.104781+0000","last_active":"2024-03-26T04:01:49.042242+0000","last_peered":"2024-03-26T04:01:49.042242+0000","last_clean":"2024-03-26T04:01:49.042242+0000","last_became_active":"2024-03-26T03:50:45.592467+0000","last_became_peered":"2024-03-26T03:50:45.592467+0000","last_unstale":"2024-03-26T04:01:49.042242+0000","last_undegraded":"2024-03-26T04:01:49.042242+0000","last_fullsized":"2024-03-26T04:01:49.042242+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:01:35.104690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:46.043427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":584,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042534+0000","last_change":"2024-03-26T04:01:35.834001+0000","last_active":"2024-03-26T04:01:49.042534+0000","last_peered":"2024-03-26T04:01:49.042534+0000","last_clean":"2024-03-26T04:01:49.042534+0000","last_became_active":"2024-03-26T04:01:35.811833+0000","last_became_peered":"2024-03-26T04:01:35.811833+0000","last_unstale":"2024-03-26T04:01:49.042534+0000","last_undegraded":"2024-03-26T04:01:49.042534+0000","last_fullsized":"2024-03-26T04:01:49.042534+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T03:59:18.367180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:18.367180+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":522,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198186+0000","last_change":"2024-03-26T04:01:12.414169+0000","last_active":"2024-03-26T04:01:50.198186+0000","last_peered":"2024-03-26T04:01:50.198186+0000","last_clean":"2024-03-26T04:01:50.198186+0000","last_became_active":"2024-03-26T03:57:49.359746+0000","last_became_peered":"2024-03-26T03:57:49.359746+0000","last_unstale":"2024-03-26T04:01:50.198186+0000","last_undegraded":"2024-03-26T04:01:50.198186+0000","last_fullsized":"2024-03-26T04:01:50.198186+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:12.414053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:12.414053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:41.255840+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"294'11","reported_seq":99,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196875+0000","last_change":"2024-03-26T04:01:35.849570+0000","last_active":"2024-03-26T04:01:50.196875+0000","last_peered":"2024-03-26T04:01:50.196875+0000","last_clean":"2024-03-26T04:01:50.196875+0000","last_became_active":"2024-03-26T04:01:35.803203+0000","last_became_peered":"2024-03-26T04:01:35.803203+0000","last_unstale":"2024-03-26T04:01:50.196875+0000","last_undegraded":"2024-03-26T04:01:50.196875+0000","last_fullsized":"2024-03-26T04:01:50.196875+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:23.341352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.341352+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":536,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046365+0000","last_change":"2024-03-26T04:00:36.999119+0000","last_active":"2024-03-26T04:01:49.046365+0000","last_peered":"2024-03-26T04:01:49.046365+0000","last_clean":"2024-03-26T04:01:49.046365+0000","last_became_active":"2024-03-26T03:48:44.016059+0000","last_became_peered":"2024-03-26T03:48:44.016059+0000","last_unstale":"2024-03-26T04:01:49.046365+0000","last_undegraded":"2024-03-26T04:01:49.046365+0000","last_fullsized":"2024-03-26T04:01:49.046365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:36.998999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.998999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:00.860008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":224,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.062635+0000","last_change":"2024-03-26T04:00:17.439081+0000","last_active":"2024-03-26T04:01:49.062635+0000","last_peered":"2024-03-26T04:01:49.062635+0000","last_clean":"2024-03-26T04:01:49.062635+0000","last_became_active":"2024-03-26T03:57:49.366246+0000","last_became_peered":"2024-03-26T03:57:49.366246+0000","last_unstale":"2024-03-26T04:01:49.062635+0000","last_undegraded":"2024-03-26T04:01:49.062635+0000","last_fullsized":"2024-03-26T04:01:49.062635+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-03-26T04:00:17.438826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.438826+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":47,"num_read_kb":40,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":523,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043269+0000","last_change":"2024-03-26T04:01:35.811101+0000","last_active":"2024-03-26T04:01:49.043269+0000","last_peered":"2024-03-26T04:01:49.043269+0000","last_clean":"2024-03-26T04:01:49.043269+0000","last_became_active":"2024-03-26T04:01:35.810636+0000","last_became_peered":"2024-03-26T04:01:35.810636+0000","last_unstale":"2024-03-26T04:01:49.043269+0000","last_undegraded":"2024-03-26T04:01:49.043269+0000","last_fullsized":"2024-03-26T04:01:49.043269+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.428232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.428232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:55.710579+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"215'10","reported_seq":246,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.062233+0000","last_change":"2024-03-26T04:00:20.443100+0000","last_active":"2024-03-26T04:01:49.062233+0000","last_peered":"2024-03-26T04:01:49.062233+0000","last_clean":"2024-03-26T04:01:49.062233+0000","last_became_active":"2024-03-26T03:57:49.386057+0000","last_became_peered":"2024-03-26T03:57:49.386057+0000","last_unstale":"2024-03-26T04:01:49.062233+0000","last_undegraded":"2024-03-26T04:01:49.062233+0000","last_fullsized":"2024-03-26T04:01:49.062233+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'10","last_scrub_stamp":"2024-03-26T04:00:20.442965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:20.442965+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":560,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197182+0000","last_change":"2024-03-26T04:00:31.362526+0000","last_active":"2024-03-26T04:01:50.197182+0000","last_peered":"2024-03-26T04:01:50.197182+0000","last_clean":"2024-03-26T04:01:50.197182+0000","last_became_active":"2024-03-26T03:57:49.361076+0000","last_became_peered":"2024-03-26T03:57:49.361076+0000","last_unstale":"2024-03-26T04:01:50.197182+0000","last_undegraded":"2024-03-26T04:01:50.197182+0000","last_fullsized":"2024-03-26T04:01:50.197182+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:00:31.362386+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T03:50:35.194368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:31.362386+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":459,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047763+0000","last_change":"2024-03-26T04:01:35.843413+0000","last_active":"2024-03-26T04:01:49.047763+0000","last_peered":"2024-03-26T04:01:49.047763+0000","last_clean":"2024-03-26T04:01:49.047763+0000","last_became_active":"2024-03-26T04:01:35.800188+0000","last_became_peered":"2024-03-26T04:01:35.800188+0000","last_unstale":"2024-03-26T04:01:49.047763+0000","last_undegraded":"2024-03-26T04:01:49.047763+0000","last_fullsized":"2024-03-26T04:01:49.047763+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:00:28.238100+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:00:28.238100+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":534,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046051+0000","last_change":"2024-03-26T04:01:13.279444+0000","last_active":"2024-03-26T04:01:49.046051+0000","last_peered":"2024-03-26T04:01:49.046051+0000","last_clean":"2024-03-26T04:01:49.046051+0000","last_became_active":"2024-03-26T03:48:50.033229+0000","last_became_peered":"2024-03-26T03:48:50.033229+0000","last_unstale":"2024-03-26T04:01:49.046051+0000","last_undegraded":"2024-03-26T04:01:49.046051+0000","last_fullsized":"2024-03-26T04:01:49.046051+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:01:13.279325+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:01:13.279325+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:13.788202+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":193,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.058460+0000","last_change":"2024-03-26T04:00:22.523367+0000","last_active":"2024-03-26T04:01:49.058460+0000","last_peered":"2024-03-26T04:01:49.058460+0000","last_clean":"2024-03-26T04:01:49.058460+0000","last_became_active":"2024-03-26T03:57:49.368529+0000","last_became_peered":"2024-03-26T03:57:49.368529+0000","last_unstale":"2024-03-26T04:01:49.058460+0000","last_undegraded":"2024-03-26T04:01:49.058460+0000","last_fullsized":"2024-03-26T04:01:49.058460+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.523245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.523245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":229,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042124+0000","last_change":"2024-03-26T04:01:25.141498+0000","last_active":"2024-03-26T04:01:49.042124+0000","last_peered":"2024-03-26T04:01:49.042124+0000","last_clean":"2024-03-26T04:01:49.042124+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:01:49.042124+0000","last_undegraded":"2024-03-26T04:01:49.042124+0000","last_fullsized":"2024-03-26T04:01:49.042124+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:25.141383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.141383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:28.737009+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":607,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198570+0000","last_change":"2024-03-26T04:00:25.344041+0000","last_active":"2024-03-26T04:01:50.198570+0000","last_peered":"2024-03-26T04:01:50.198570+0000","last_clean":"2024-03-26T04:01:50.198570+0000","last_became_active":"2024-03-26T03:57:49.360081+0000","last_became_peered":"2024-03-26T03:57:49.360081+0000","last_unstale":"2024-03-26T04:01:50.198570+0000","last_undegraded":"2024-03-26T04:01:50.198570+0000","last_fullsized":"2024-03-26T04:01:50.198570+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:25.343896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:25.343896+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:52.529323+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":537,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047470+0000","last_change":"2024-03-26T04:01:35.800351+0000","last_active":"2024-03-26T04:01:49.047470+0000","last_peered":"2024-03-26T04:01:49.047470+0000","last_clean":"2024-03-26T04:01:49.047470+0000","last_became_active":"2024-03-26T04:01:35.799877+0000","last_became_peered":"2024-03-26T04:01:35.799877+0000","last_unstale":"2024-03-26T04:01:49.047470+0000","last_undegraded":"2024-03-26T04:01:49.047470+0000","last_fullsized":"2024-03-26T04:01:49.047470+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.808761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.808761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:26.948732+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":319,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.188379+0000","last_change":"2024-03-26T04:01:35.807971+0000","last_active":"2024-03-26T04:01:50.188379+0000","last_peered":"2024-03-26T04:01:50.188379+0000","last_clean":"2024-03-26T04:01:50.188379+0000","last_became_active":"2024-03-26T04:01:35.807545+0000","last_became_peered":"2024-03-26T04:01:35.807545+0000","last_unstale":"2024-03-26T04:01:50.188379+0000","last_undegraded":"2024-03-26T04:01:50.188379+0000","last_fullsized":"2024-03-26T04:01:50.188379+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:26.216940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:29.537750+0000","last_clean_scrub_stamp":"2024-03-26T04:01:26.216940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:31.339419+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":479,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.200689+0000","last_change":"2024-03-26T04:01:35.813158+0000","last_active":"2024-03-26T04:01:50.200689+0000","last_peered":"2024-03-26T04:01:50.200689+0000","last_clean":"2024-03-26T04:01:50.200689+0000","last_became_active":"2024-03-26T04:01:35.812801+0000","last_became_peered":"2024-03-26T04:01:35.812801+0000","last_unstale":"2024-03-26T04:01:50.200689+0000","last_undegraded":"2024-03-26T04:01:50.200689+0000","last_fullsized":"2024-03-26T04:01:50.200689+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:20.739710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T03:59:20.739710+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":212,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.058269+0000","last_change":"2024-03-26T04:00:21.475401+0000","last_active":"2024-03-26T04:01:49.058269+0000","last_peered":"2024-03-26T04:01:49.058269+0000","last_clean":"2024-03-26T04:01:49.058269+0000","last_became_active":"2024-03-26T03:57:49.381431+0000","last_became_peered":"2024-03-26T03:57:49.381431+0000","last_unstale":"2024-03-26T04:01:49.058269+0000","last_undegraded":"2024-03-26T04:01:49.058269+0000","last_fullsized":"2024-03-26T04:01:49.058269+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:00:21.475271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.475271+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":22,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"303'11631","reported_seq":13746,"reported_epoch":303,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:50.497401+0000","last_change":"2024-03-26T04:01:37.042112+0000","last_active":"2024-03-26T04:01:50.497401+0000","last_peered":"2024-03-26T04:01:50.497401+0000","last_clean":"2024-03-26T04:01:34.763738+0000","last_became_active":"2024-03-26T04:01:36.929399+0000","last_became_peered":"2024-03-26T04:01:36.929399+0000","last_unstale":"2024-03-26T04:01:50.497401+0000","last_undegraded":"2024-03-26T04:01:50.497401+0000","last_fullsized":"2024-03-26T04:01:50.497401+0000","mapping_epoch":294,"log_start":"283'7712","ondisk_log_start":"283'7712","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":3919,"log_dups_size":0,"ondisk_log_size":3919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":336134144,"num_objects":5129,"num_object_clones":0,"num_object_copies":15387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4203,"num_objects_unfound":0,"num_objects_dirty":5129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11631,"num_write_kb":536320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1000,"num_bytes_recovered":65404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":5129}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":238,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197249+0000","last_change":"2024-03-26T04:01:35.815085+0000","last_active":"2024-03-26T04:01:50.197249+0000","last_peered":"2024-03-26T04:01:50.197249+0000","last_clean":"2024-03-26T04:01:50.197249+0000","last_became_active":"2024-03-26T04:01:35.813343+0000","last_became_peered":"2024-03-26T04:01:35.813343+0000","last_unstale":"2024-03-26T04:01:50.197249+0000","last_undegraded":"2024-03-26T04:01:50.197249+0000","last_fullsized":"2024-03-26T04:01:50.197249+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:31.506978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:31.506978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":540,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047757+0000","last_change":"2024-03-26T04:00:17.997376+0000","last_active":"2024-03-26T04:01:49.047757+0000","last_peered":"2024-03-26T04:01:49.047757+0000","last_clean":"2024-03-26T04:01:49.047757+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:01:49.047757+0000","last_undegraded":"2024-03-26T04:01:49.047757+0000","last_fullsized":"2024-03-26T04:01:49.047757+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T04:00:17.997292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.997292+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":475,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046673+0000","last_change":"2024-03-26T04:01:35.829757+0000","last_active":"2024-03-26T04:01:49.046673+0000","last_peered":"2024-03-26T04:01:49.046673+0000","last_clean":"2024-03-26T04:01:49.046673+0000","last_became_active":"2024-03-26T04:01:35.786917+0000","last_became_peered":"2024-03-26T04:01:35.786917+0000","last_unstale":"2024-03-26T04:01:49.046673+0000","last_undegraded":"2024-03-26T04:01:49.046673+0000","last_fullsized":"2024-03-26T04:01:49.046673+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:00:11.391188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.391188+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":704,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046961+0000","last_change":"2024-03-26T04:00:34.970757+0000","last_active":"2024-03-26T04:01:49.046961+0000","last_peered":"2024-03-26T04:01:49.046961+0000","last_clean":"2024-03-26T04:01:49.046961+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:01:49.046961+0000","last_undegraded":"2024-03-26T04:01:49.046961+0000","last_fullsized":"2024-03-26T04:01:49.046961+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:34.970608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.970608+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:56.195694+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"303'11370","reported_seq":11566,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.443188+0000","last_change":"2024-03-26T04:01:11.223850+0000","last_active":"2024-03-26T04:01:50.443188+0000","last_peered":"2024-03-26T04:01:50.443188+0000","last_clean":"2024-03-26T04:01:50.443188+0000","last_became_active":"2024-03-26T03:59:23.931611+0000","last_became_peered":"2024-03-26T03:59:23.931611+0000","last_unstale":"2024-03-26T04:01:50.443188+0000","last_undegraded":"2024-03-26T04:01:50.443188+0000","last_fullsized":"2024-03-26T04:01:50.443188+0000","mapping_epoch":253,"log_start":"284'7830","ondisk_log_start":"284'7830","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'8942","last_scrub_stamp":"2024-03-26T04:01:11.223688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:11.223688+0000","objects_scrubbed":7593,"log_size":3540,"log_dups_size":0,"ondisk_log_size":3540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:23.251990+0000","scrub_duration":2827,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11370,"num_write_kb":525760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189752+0000","last_change":"2024-03-26T03:59:30.159583+0000","last_active":"2024-03-26T04:01:50.189752+0000","last_peered":"2024-03-26T04:01:50.189752+0000","last_clean":"2024-03-26T04:01:50.189752+0000","last_became_active":"2024-03-26T03:57:49.369109+0000","last_became_peered":"2024-03-26T03:57:49.369109+0000","last_unstale":"2024-03-26T04:01:50.189752+0000","last_undegraded":"2024-03-26T04:01:50.189752+0000","last_fullsized":"2024-03-26T04:01:50.189752+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.159481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.159481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":528,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047427+0000","last_change":"2024-03-26T04:00:13.964333+0000","last_active":"2024-03-26T04:01:49.047427+0000","last_peered":"2024-03-26T04:01:49.047427+0000","last_clean":"2024-03-26T04:01:49.047427+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:01:49.047427+0000","last_undegraded":"2024-03-26T04:01:49.047427+0000","last_fullsized":"2024-03-26T04:01:49.047427+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:13.964232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:13.964232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":617,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042423+0000","last_change":"2024-03-26T04:00:36.093635+0000","last_active":"2024-03-26T04:01:49.042423+0000","last_peered":"2024-03-26T04:01:49.042423+0000","last_clean":"2024-03-26T04:01:49.042423+0000","last_became_active":"2024-03-26T03:57:49.360144+0000","last_became_peered":"2024-03-26T03:57:49.360144+0000","last_unstale":"2024-03-26T04:01:49.042423+0000","last_undegraded":"2024-03-26T04:01:49.042423+0000","last_fullsized":"2024-03-26T04:01:49.042423+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:00:36.093314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.093314+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":537,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047436+0000","last_change":"2024-03-26T04:01:11.271682+0000","last_active":"2024-03-26T04:01:49.047436+0000","last_peered":"2024-03-26T04:01:49.047436+0000","last_clean":"2024-03-26T04:01:49.047436+0000","last_became_active":"2024-03-26T03:48:48.018746+0000","last_became_peered":"2024-03-26T03:48:48.018746+0000","last_unstale":"2024-03-26T04:01:49.047436+0000","last_undegraded":"2024-03-26T04:01:49.047436+0000","last_fullsized":"2024-03-26T04:01:49.047436+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:11.271561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:11.271561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:19.191605+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"302'11718","reported_seq":15236,"reported_epoch":302,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:49.424251+0000","last_change":"2024-03-26T04:01:37.087409+0000","last_active":"2024-03-26T04:01:49.424251+0000","last_peered":"2024-03-26T04:01:49.424251+0000","last_clean":"2024-03-26T04:01:34.755730+0000","last_became_active":"2024-03-26T04:01:37.050718+0000","last_became_peered":"2024-03-26T04:01:37.050718+0000","last_unstale":"2024-03-26T04:01:49.424251+0000","last_undegraded":"2024-03-26T04:01:49.424251+0000","last_fullsized":"2024-03-26T04:01:49.424251+0000","mapping_epoch":294,"log_start":"283'7813","ondisk_log_start":"283'7813","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'10834","last_scrub_stamp":"2024-03-26T04:01:32.215037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.215037+0000","objects_scrubbed":6128,"log_size":3905,"log_dups_size":0,"ondisk_log_size":3905,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1983,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338821120,"num_objects":5170,"num_object_clones":0,"num_object_copies":15510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3614,"num_objects_unfound":0,"num_objects_dirty":5170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11718,"num_write_kb":540416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1667,"num_bytes_recovered":109248512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,5],"avail_no_missing":["7","3","5"],"object_location_counts":[{"shards":"3,5,7","objects":5170}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":487,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047598+0000","last_change":"2024-03-26T04:01:19.379887+0000","last_active":"2024-03-26T04:01:49.047598+0000","last_peered":"2024-03-26T04:01:49.047598+0000","last_clean":"2024-03-26T04:01:49.047598+0000","last_became_active":"2024-03-26T03:58:40.502368+0000","last_became_peered":"2024-03-26T03:58:40.502368+0000","last_unstale":"2024-03-26T04:01:49.047598+0000","last_undegraded":"2024-03-26T04:01:49.047598+0000","last_fullsized":"2024-03-26T04:01:49.047598+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:19.379772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.379772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:36.242146+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":543,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046442+0000","last_change":"2024-03-26T04:01:35.823449+0000","last_active":"2024-03-26T04:01:49.046442+0000","last_peered":"2024-03-26T04:01:49.046442+0000","last_clean":"2024-03-26T04:01:49.046442+0000","last_became_active":"2024-03-26T04:01:35.823058+0000","last_became_peered":"2024-03-26T04:01:35.823058+0000","last_unstale":"2024-03-26T04:01:49.046442+0000","last_undegraded":"2024-03-26T04:01:49.046442+0000","last_fullsized":"2024-03-26T04:01:49.046442+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:26.026626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:26.026626+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":387,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189190+0000","last_change":"2024-03-26T04:01:35.795739+0000","last_active":"2024-03-26T04:01:50.189190+0000","last_peered":"2024-03-26T04:01:50.189190+0000","last_clean":"2024-03-26T04:01:50.189190+0000","last_became_active":"2024-03-26T04:01:35.795390+0000","last_became_peered":"2024-03-26T04:01:35.795390+0000","last_unstale":"2024-03-26T04:01:50.189190+0000","last_undegraded":"2024-03-26T04:01:50.189190+0000","last_fullsized":"2024-03-26T04:01:50.189190+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.277958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.277958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.a","version":"241'41","reported_seq":605,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.202141+0000","last_change":"2024-03-26T04:01:35.859946+0000","last_active":"2024-03-26T04:01:50.202141+0000","last_peered":"2024-03-26T04:01:50.202141+0000","last_clean":"2024-03-26T04:01:50.202141+0000","last_became_active":"2024-03-26T04:01:35.818705+0000","last_became_peered":"2024-03-26T04:01:35.818705+0000","last_unstale":"2024-03-26T04:01:50.202141+0000","last_undegraded":"2024-03-26T04:01:50.202141+0000","last_fullsized":"2024-03-26T04:01:50.202141+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'41","last_scrub_stamp":"2024-03-26T04:00:36.459640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:36.459640+0000","objects_scrubbed":10,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.6","version":"302'11651","reported_seq":14585,"reported_epoch":302,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:50.047445+0000","last_change":"2024-03-26T04:01:37.090720+0000","last_active":"2024-03-26T04:01:50.047445+0000","last_peered":"2024-03-26T04:01:50.047445+0000","last_clean":"2024-03-26T04:01:34.770866+0000","last_became_active":"2024-03-26T04:01:36.894612+0000","last_became_peered":"2024-03-26T04:01:36.894612+0000","last_unstale":"2024-03-26T04:01:50.047445+0000","last_undegraded":"2024-03-26T04:01:50.047445+0000","last_fullsized":"2024-03-26T04:01:50.047445+0000","mapping_epoch":294,"log_start":"282'7212","ondisk_log_start":"282'7212","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'9411","last_scrub_stamp":"2024-03-26T04:01:15.567924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:15.567924+0000","objects_scrubbed":7385,"log_size":4439,"log_dups_size":0,"ondisk_log_size":4439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2457,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332070912,"num_objects":5067,"num_object_clones":0,"num_object_copies":15201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3786,"num_objects_unfound":0,"num_objects_dirty":5067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11651,"num_write_kb":534976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1366,"num_bytes_recovered":89456640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,5],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":5067}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":511,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198470+0000","last_change":"2024-03-26T04:00:22.410543+0000","last_active":"2024-03-26T04:01:50.198470+0000","last_peered":"2024-03-26T04:01:50.198470+0000","last_clean":"2024-03-26T04:01:50.198470+0000","last_became_active":"2024-03-26T03:48:50.040386+0000","last_became_peered":"2024-03-26T03:48:50.040386+0000","last_unstale":"2024-03-26T04:01:50.198470+0000","last_undegraded":"2024-03-26T04:01:50.198470+0000","last_fullsized":"2024-03-26T04:01:50.198470+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.410420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.410420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":548,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.050224+0000","last_change":"2024-03-26T04:01:08.274062+0000","last_active":"2024-03-26T04:01:49.050224+0000","last_peered":"2024-03-26T04:01:49.050224+0000","last_clean":"2024-03-26T04:01:49.050224+0000","last_became_active":"2024-03-26T03:57:49.359810+0000","last_became_peered":"2024-03-26T03:57:49.359810+0000","last_unstale":"2024-03-26T04:01:49.050224+0000","last_undegraded":"2024-03-26T04:01:49.050224+0000","last_fullsized":"2024-03-26T04:01:49.050224+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:08.273802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.273802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:27.052015+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":339,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.061415+0000","last_change":"2024-03-26T04:01:35.809546+0000","last_active":"2024-03-26T04:01:49.061415+0000","last_peered":"2024-03-26T04:01:49.061415+0000","last_clean":"2024-03-26T04:01:49.061415+0000","last_became_active":"2024-03-26T04:01:35.808936+0000","last_became_peered":"2024-03-26T04:01:35.808936+0000","last_unstale":"2024-03-26T04:01:49.061415+0000","last_undegraded":"2024-03-26T04:01:49.061415+0000","last_fullsized":"2024-03-26T04:01:49.061415+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:09.385412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:00:09.385412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":608,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197877+0000","last_change":"2024-03-26T04:01:35.846596+0000","last_active":"2024-03-26T04:01:50.197877+0000","last_peered":"2024-03-26T04:01:50.197877+0000","last_clean":"2024-03-26T04:01:50.197877+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:01:50.197877+0000","last_undegraded":"2024-03-26T04:01:50.197877+0000","last_fullsized":"2024-03-26T04:01:50.197877+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:01:08.323515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.323515+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:10.921654+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":324,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196589+0000","last_change":"2024-03-26T04:01:38.806163+0000","last_active":"2024-03-26T04:01:50.196589+0000","last_peered":"2024-03-26T04:01:50.196589+0000","last_clean":"2024-03-26T04:01:50.196589+0000","last_became_active":"2024-03-26T04:01:38.805798+0000","last_became_peered":"2024-03-26T04:01:38.805798+0000","last_unstale":"2024-03-26T04:01:50.196589+0000","last_undegraded":"2024-03-26T04:01:50.196589+0000","last_fullsized":"2024-03-26T04:01:50.196589+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.306506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.306506+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":161010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"303'11398","reported_seq":11589,"reported_epoch":303,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T04:01:50.450981+0000","last_change":"2024-03-26T04:01:37.031730+0000","last_active":"2024-03-26T04:01:50.450981+0000","last_peered":"2024-03-26T04:01:50.450981+0000","last_clean":"2024-03-26T04:01:34.771338+0000","last_became_active":"2024-03-26T04:01:36.919241+0000","last_became_peered":"2024-03-26T04:01:36.919241+0000","last_unstale":"2024-03-26T04:01:50.450981+0000","last_undegraded":"2024-03-26T04:01:50.450981+0000","last_fullsized":"2024-03-26T04:01:50.450981+0000","mapping_epoch":294,"log_start":"283'7474","ondisk_log_start":"283'7474","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"290'9954","last_scrub_stamp":"2024-03-26T04:01:24.854541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:24.854541+0000","objects_scrubbed":6467,"log_size":3924,"log_dups_size":0,"ondisk_log_size":3924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":322437120,"num_objects":4920,"num_object_clones":0,"num_object_copies":14760,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4920,"num_objects_unfound":0,"num_objects_dirty":4920,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11398,"num_write_kb":522176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,5],"avail_no_missing":["0","5","7"],"object_location_counts":[{"shards":"0,5,7","objects":4920}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.188789+0000","last_change":"2024-03-26T04:00:19.502445+0000","last_active":"2024-03-26T04:01:50.188789+0000","last_peered":"2024-03-26T04:01:50.188789+0000","last_clean":"2024-03-26T04:01:50.188789+0000","last_became_active":"2024-03-26T03:57:49.359862+0000","last_became_peered":"2024-03-26T03:57:49.359862+0000","last_unstale":"2024-03-26T04:01:50.188789+0000","last_undegraded":"2024-03-26T04:01:50.188789+0000","last_fullsized":"2024-03-26T04:01:50.188789+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.502323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.502323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":407,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.200266+0000","last_change":"2024-03-26T04:01:35.840015+0000","last_active":"2024-03-26T04:01:50.200266+0000","last_peered":"2024-03-26T04:01:50.200266+0000","last_clean":"2024-03-26T04:01:50.200266+0000","last_became_active":"2024-03-26T04:01:35.808367+0000","last_became_peered":"2024-03-26T04:01:35.808367+0000","last_unstale":"2024-03-26T04:01:50.200266+0000","last_undegraded":"2024-03-26T04:01:50.200266+0000","last_fullsized":"2024-03-26T04:01:50.200266+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:01:24.259809+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:01:24.259809+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:28.841391+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":527,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042181+0000","last_change":"2024-03-26T04:01:12.096789+0000","last_active":"2024-03-26T04:01:49.042181+0000","last_peered":"2024-03-26T04:01:49.042181+0000","last_clean":"2024-03-26T04:01:49.042181+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:01:49.042181+0000","last_undegraded":"2024-03-26T04:01:49.042181+0000","last_fullsized":"2024-03-26T04:01:49.042181+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:12.096623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:01:12.096623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:21.781455+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"241'48","reported_seq":821,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.195113+0000","last_change":"2024-03-26T04:01:23.451265+0000","last_active":"2024-03-26T04:01:50.195113+0000","last_peered":"2024-03-26T04:01:50.195113+0000","last_clean":"2024-03-26T04:01:50.195113+0000","last_became_active":"2024-03-26T03:57:49.382066+0000","last_became_peered":"2024-03-26T03:57:49.382066+0000","last_unstale":"2024-03-26T04:01:50.195113+0000","last_undegraded":"2024-03-26T04:01:50.195113+0000","last_fullsized":"2024-03-26T04:01:50.195113+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T04:01:23.451122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.451122+0000","objects_scrubbed":11,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:31.549653+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":523,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198506+0000","last_change":"2024-03-26T04:01:18.456710+0000","last_active":"2024-03-26T04:01:50.198506+0000","last_peered":"2024-03-26T04:01:50.198506+0000","last_clean":"2024-03-26T04:01:50.198506+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:01:50.198506+0000","last_undegraded":"2024-03-26T04:01:50.198506+0000","last_fullsized":"2024-03-26T04:01:50.198506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.456594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.456594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:35.001532+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":365,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196875+0000","last_change":"2024-03-26T04:00:34.281218+0000","last_active":"2024-03-26T04:01:50.196875+0000","last_peered":"2024-03-26T04:01:50.196875+0000","last_clean":"2024-03-26T04:01:50.196875+0000","last_became_active":"2024-03-26T04:00:32.425416+0000","last_became_peered":"2024-03-26T04:00:32.425416+0000","last_unstale":"2024-03-26T04:01:50.196875+0000","last_undegraded":"2024-03-26T04:01:50.196875+0000","last_fullsized":"2024-03-26T04:01:50.196875+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.281069+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"303'11389","reported_seq":13376,"reported_epoch":303,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:50.587521+0000","last_change":"2024-03-26T04:01:36.928667+0000","last_active":"2024-03-26T04:01:50.587521+0000","last_peered":"2024-03-26T04:01:50.587521+0000","last_clean":"2024-03-26T04:01:34.761797+0000","last_became_active":"2024-03-26T04:01:36.911256+0000","last_became_peered":"2024-03-26T04:01:36.911256+0000","last_unstale":"2024-03-26T04:01:50.587521+0000","last_undegraded":"2024-03-26T04:01:50.587521+0000","last_fullsized":"2024-03-26T04:01:50.587521+0000","mapping_epoch":294,"log_start":"284'7775","ondisk_log_start":"284'7775","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":3614,"log_dups_size":0,"ondisk_log_size":3614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326172672,"num_objects":4977,"num_object_clones":0,"num_object_copies":14931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4112,"num_objects_unfound":0,"num_objects_dirty":4977,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11389,"num_write_kb":523712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":61276160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,5],"avail_no_missing":["1","5","7"],"object_location_counts":[{"shards":"1,5,7","objects":4977}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":544,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047779+0000","last_change":"2024-03-26T04:00:15.989927+0000","last_active":"2024-03-26T04:01:49.047779+0000","last_peered":"2024-03-26T04:01:49.047779+0000","last_clean":"2024-03-26T04:01:49.047779+0000","last_became_active":"2024-03-26T03:57:49.346614+0000","last_became_peered":"2024-03-26T03:57:49.346614+0000","last_unstale":"2024-03-26T04:01:49.047779+0000","last_undegraded":"2024-03-26T04:01:49.047779+0000","last_fullsized":"2024-03-26T04:01:49.047779+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:15.989815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.989815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":535,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.193778+0000","last_change":"2024-03-26T03:59:30.302063+0000","last_active":"2024-03-26T04:01:50.193778+0000","last_peered":"2024-03-26T04:01:50.193778+0000","last_clean":"2024-03-26T04:01:50.193778+0000","last_became_active":"2024-03-26T03:50:45.591475+0000","last_became_peered":"2024-03-26T03:50:45.591475+0000","last_unstale":"2024-03-26T04:01:50.193778+0000","last_undegraded":"2024-03-26T04:01:50.193778+0000","last_fullsized":"2024-03-26T04:01:50.193778+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:30.301949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:30.301949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":566,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045423+0000","last_change":"2024-03-26T04:01:27.325433+0000","last_active":"2024-03-26T04:01:49.045423+0000","last_peered":"2024-03-26T04:01:49.045423+0000","last_clean":"2024-03-26T04:01:49.045423+0000","last_became_active":"2024-03-26T03:57:49.359690+0000","last_became_peered":"2024-03-26T03:57:49.359690+0000","last_unstale":"2024-03-26T04:01:49.045423+0000","last_undegraded":"2024-03-26T04:01:49.045423+0000","last_fullsized":"2024-03-26T04:01:49.045423+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:27.325285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:01:27.325285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:31.741302+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"241'23","reported_seq":591,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.194759+0000","last_change":"2024-03-26T04:01:35.836345+0000","last_active":"2024-03-26T04:01:50.194759+0000","last_peered":"2024-03-26T04:01:50.194759+0000","last_clean":"2024-03-26T04:01:50.194759+0000","last_became_active":"2024-03-26T04:01:35.795965+0000","last_became_peered":"2024-03-26T04:01:35.795965+0000","last_unstale":"2024-03-26T04:01:50.194759+0000","last_undegraded":"2024-03-26T04:01:50.194759+0000","last_fullsized":"2024-03-26T04:01:50.194759+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'23","last_scrub_stamp":"2024-03-26T04:00:11.534660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:11.534660+0000","objects_scrubbed":6,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":544,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.053438+0000","last_change":"2024-03-26T04:01:35.816437+0000","last_active":"2024-03-26T04:01:49.053438+0000","last_peered":"2024-03-26T04:01:49.053438+0000","last_clean":"2024-03-26T04:01:49.053438+0000","last_became_active":"2024-03-26T04:01:35.814520+0000","last_became_peered":"2024-03-26T04:01:35.814520+0000","last_unstale":"2024-03-26T04:01:49.053438+0000","last_undegraded":"2024-03-26T04:01:49.053438+0000","last_fullsized":"2024-03-26T04:01:49.053438+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.247532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.247532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:34.722888+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"302'11473","reported_seq":11587,"reported_epoch":303,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T04:01:50.190499+0000","last_change":"2024-03-26T04:01:36.886781+0000","last_active":"2024-03-26T04:01:50.190499+0000","last_peered":"2024-03-26T04:01:50.190499+0000","last_clean":"2024-03-26T04:01:34.762029+0000","last_became_active":"2024-03-26T04:01:36.856353+0000","last_became_peered":"2024-03-26T04:01:36.856353+0000","last_unstale":"2024-03-26T04:01:50.190499+0000","last_undegraded":"2024-03-26T04:01:50.190499+0000","last_fullsized":"2024-03-26T04:01:50.190499+0000","mapping_epoch":294,"log_start":"284'7855","ondisk_log_start":"284'7855","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":3618,"log_dups_size":0,"ondisk_log_size":3618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":15339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5113,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11473,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":5113}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.058370+0000","last_change":"2024-03-26T04:01:35.822560+0000","last_active":"2024-03-26T04:01:49.058370+0000","last_peered":"2024-03-26T04:01:49.058370+0000","last_clean":"2024-03-26T04:01:49.058370+0000","last_became_active":"2024-03-26T04:01:35.821091+0000","last_became_peered":"2024-03-26T04:01:35.821091+0000","last_unstale":"2024-03-26T04:01:49.058370+0000","last_undegraded":"2024-03-26T04:01:49.058370+0000","last_fullsized":"2024-03-26T04:01:49.058370+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.313268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:29.313268+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.313268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:57.048386+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":811,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045998+0000","last_change":"2024-03-26T04:00:27.961142+0000","last_active":"2024-03-26T04:01:49.045998+0000","last_peered":"2024-03-26T04:01:49.045998+0000","last_clean":"2024-03-26T04:01:49.045998+0000","last_became_active":"2024-03-26T04:00:27.442487+0000","last_became_peered":"2024-03-26T04:00:27.442487+0000","last_unstale":"2024-03-26T04:01:49.045998+0000","last_undegraded":"2024-03-26T04:01:49.045998+0000","last_fullsized":"2024-03-26T04:01:49.045998+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:27.960979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.960979+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":209,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.191148+0000","last_change":"2024-03-26T04:01:35.808116+0000","last_active":"2024-03-26T04:01:50.191148+0000","last_peered":"2024-03-26T04:01:50.191148+0000","last_clean":"2024-03-26T04:01:50.191148+0000","last_became_active":"2024-03-26T04:01:35.807784+0000","last_became_peered":"2024-03-26T04:01:35.807784+0000","last_unstale":"2024-03-26T04:01:50.191148+0000","last_undegraded":"2024-03-26T04:01:50.191148+0000","last_fullsized":"2024-03-26T04:01:50.191148+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:23.676442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.676442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:49.229565+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":663,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043016+0000","last_change":"2024-03-26T04:01:16.135828+0000","last_active":"2024-03-26T04:01:49.043016+0000","last_peered":"2024-03-26T04:01:49.043016+0000","last_clean":"2024-03-26T04:01:49.043016+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:01:49.043016+0000","last_undegraded":"2024-03-26T04:01:49.043016+0000","last_fullsized":"2024-03-26T04:01:49.043016+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:01:16.135657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:16.135657+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:21.611015+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":425,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046169+0000","last_change":"2024-03-26T04:01:35.814431+0000","last_active":"2024-03-26T04:01:49.046169+0000","last_peered":"2024-03-26T04:01:49.046169+0000","last_clean":"2024-03-26T04:01:49.046169+0000","last_became_active":"2024-03-26T04:01:35.814045+0000","last_became_peered":"2024-03-26T04:01:35.814045+0000","last_unstale":"2024-03-26T04:01:49.046169+0000","last_undegraded":"2024-03-26T04:01:49.046169+0000","last_fullsized":"2024-03-26T04:01:49.046169+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.381181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.381181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:22.870694+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":419,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.195588+0000","last_change":"2024-03-26T04:01:38.812481+0000","last_active":"2024-03-26T04:01:50.195588+0000","last_peered":"2024-03-26T04:01:50.195588+0000","last_clean":"2024-03-26T04:01:50.195588+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:01:50.195588+0000","last_undegraded":"2024-03-26T04:01:50.195588+0000","last_fullsized":"2024-03-26T04:01:50.195588+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:32.592884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:32.592884+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.b","version":"303'11611","reported_seq":13758,"reported_epoch":303,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:50.498697+0000","last_change":"2024-03-26T04:01:37.009811+0000","last_active":"2024-03-26T04:01:50.498697+0000","last_peered":"2024-03-26T04:01:50.498697+0000","last_clean":"2024-03-26T04:01:34.777522+0000","last_became_active":"2024-03-26T04:01:36.915942+0000","last_became_peered":"2024-03-26T04:01:36.915942+0000","last_unstale":"2024-03-26T04:01:50.498697+0000","last_undegraded":"2024-03-26T04:01:50.498697+0000","last_fullsized":"2024-03-26T04:01:50.498697+0000","mapping_epoch":294,"log_start":"284'7740","ondisk_log_start":"284'7740","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'9946","last_scrub_stamp":"2024-03-26T04:01:22.990108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.990108+0000","objects_scrubbed":6720,"log_size":3871,"log_dups_size":0,"ondisk_log_size":3871,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2544,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326041600,"num_objects":4975,"num_object_clones":0,"num_object_copies":14925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4061,"num_objects_unfound":0,"num_objects_dirty":4975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11611,"num_write_kb":530752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":977,"num_bytes_recovered":64028672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,5],"avail_no_missing":["0","2","5"],"object_location_counts":[{"shards":"0,2,5","objects":4975}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":536,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.194772+0000","last_change":"2024-03-26T04:01:35.818474+0000","last_active":"2024-03-26T04:01:50.194772+0000","last_peered":"2024-03-26T04:01:50.194772+0000","last_clean":"2024-03-26T04:01:50.194772+0000","last_became_active":"2024-03-26T04:01:35.811837+0000","last_became_peered":"2024-03-26T04:01:35.811837+0000","last_unstale":"2024-03-26T04:01:50.194772+0000","last_undegraded":"2024-03-26T04:01:50.194772+0000","last_fullsized":"2024-03-26T04:01:50.194772+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.552445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.552445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":552,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197116+0000","last_change":"2024-03-26T04:00:23.326811+0000","last_active":"2024-03-26T04:01:50.197116+0000","last_peered":"2024-03-26T04:01:50.197116+0000","last_clean":"2024-03-26T04:01:50.197116+0000","last_became_active":"2024-03-26T03:57:49.364056+0000","last_became_peered":"2024-03-26T03:57:49.364056+0000","last_unstale":"2024-03-26T04:01:50.197116+0000","last_undegraded":"2024-03-26T04:01:50.197116+0000","last_fullsized":"2024-03-26T04:01:50.197116+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:23.326554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:23.326554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1326,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045922+0000","last_change":"2024-03-26T04:00:28.914240+0000","last_active":"2024-03-26T04:01:49.045922+0000","last_peered":"2024-03-26T04:01:49.045922+0000","last_clean":"2024-03-26T04:01:49.045922+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:01:49.045922+0000","last_undegraded":"2024-03-26T04:01:49.045922+0000","last_fullsized":"2024-03-26T04:01:49.045922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:00:28.914032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:00:28.914032+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":777,"num_read_kb":772,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":555,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198321+0000","last_change":"2024-03-26T04:00:18.345453+0000","last_active":"2024-03-26T04:01:50.198321+0000","last_peered":"2024-03-26T04:01:50.198321+0000","last_clean":"2024-03-26T04:01:50.198321+0000","last_became_active":"2024-03-26T03:48:46.025580+0000","last_became_peered":"2024-03-26T03:48:46.025580+0000","last_unstale":"2024-03-26T04:01:50.198321+0000","last_undegraded":"2024-03-26T04:01:50.198321+0000","last_fullsized":"2024-03-26T04:01:50.198321+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:00:18.345305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.345305+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":228,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.195081+0000","last_change":"2024-03-26T04:01:35.803435+0000","last_active":"2024-03-26T04:01:50.195081+0000","last_peered":"2024-03-26T04:01:50.195081+0000","last_clean":"2024-03-26T04:01:50.195081+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:01:50.195081+0000","last_undegraded":"2024-03-26T04:01:50.195081+0000","last_fullsized":"2024-03-26T04:01:50.195081+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:32.538628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:32.538628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":365,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198086+0000","last_change":"2024-03-26T04:00:33.288471+0000","last_active":"2024-03-26T04:01:50.198086+0000","last_peered":"2024-03-26T04:01:50.198086+0000","last_clean":"2024-03-26T04:01:50.198086+0000","last_became_active":"2024-03-26T04:00:32.425047+0000","last_became_peered":"2024-03-26T04:00:32.425047+0000","last_unstale":"2024-03-26T04:01:50.198086+0000","last_undegraded":"2024-03-26T04:01:50.198086+0000","last_fullsized":"2024-03-26T04:01:50.198086+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.288346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.288346+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.e","version":"302'11174","reported_seq":11361,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.423984+0000","last_change":"2024-03-26T04:01:15.859874+0000","last_active":"2024-03-26T04:01:49.423984+0000","last_peered":"2024-03-26T04:01:49.423984+0000","last_clean":"2024-03-26T04:01:49.423984+0000","last_became_active":"2024-03-26T03:59:23.930042+0000","last_became_peered":"2024-03-26T03:59:23.930042+0000","last_unstale":"2024-03-26T04:01:49.423984+0000","last_undegraded":"2024-03-26T04:01:49.423984+0000","last_fullsized":"2024-03-26T04:01:49.423984+0000","mapping_epoch":253,"log_start":"282'7286","ondisk_log_start":"282'7286","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'9132","last_scrub_stamp":"2024-03-26T04:01:15.859609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:15.859609+0000","objects_scrubbed":7163,"log_size":3888,"log_dups_size":0,"ondisk_log_size":3888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:24.838940+0000","scrub_duration":2039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331743232,"num_objects":5062,"num_object_clones":0,"num_object_copies":15186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5062,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11174,"num_write_kb":519552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":523,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190370+0000","last_change":"2024-03-26T04:00:35.502278+0000","last_active":"2024-03-26T04:01:50.190370+0000","last_peered":"2024-03-26T04:01:50.190370+0000","last_clean":"2024-03-26T04:01:50.190370+0000","last_became_active":"2024-03-26T03:50:45.586163+0000","last_became_peered":"2024-03-26T03:50:45.586163+0000","last_unstale":"2024-03-26T04:01:50.190370+0000","last_undegraded":"2024-03-26T04:01:50.190370+0000","last_fullsized":"2024-03-26T04:01:50.190370+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.502151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":544,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.195019+0000","last_change":"2024-03-26T04:01:35.814214+0000","last_active":"2024-03-26T04:01:50.195019+0000","last_peered":"2024-03-26T04:01:50.195019+0000","last_clean":"2024-03-26T04:01:50.195019+0000","last_became_active":"2024-03-26T04:01:35.803821+0000","last_became_peered":"2024-03-26T04:01:35.803821+0000","last_unstale":"2024-03-26T04:01:50.195019+0000","last_undegraded":"2024-03-26T04:01:50.195019+0000","last_fullsized":"2024-03-26T04:01:50.195019+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:34.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.550087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:59.889055+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"241'35","reported_seq":625,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197738+0000","last_change":"2024-03-26T04:01:35.852945+0000","last_active":"2024-03-26T04:01:50.197738+0000","last_peered":"2024-03-26T04:01:50.197738+0000","last_clean":"2024-03-26T04:01:50.197738+0000","last_became_active":"2024-03-26T04:01:35.824185+0000","last_became_peered":"2024-03-26T04:01:35.824185+0000","last_unstale":"2024-03-26T04:01:50.197738+0000","last_undegraded":"2024-03-26T04:01:50.197738+0000","last_fullsized":"2024-03-26T04:01:50.197738+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'35","last_scrub_stamp":"2024-03-26T04:00:38.314372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.314372+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":468,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197056+0000","last_change":"2024-03-26T04:01:35.813665+0000","last_active":"2024-03-26T04:01:50.197056+0000","last_peered":"2024-03-26T04:01:50.197056+0000","last_clean":"2024-03-26T04:01:50.197056+0000","last_became_active":"2024-03-26T04:01:35.812898+0000","last_became_peered":"2024-03-26T04:01:35.812898+0000","last_unstale":"2024-03-26T04:01:50.197056+0000","last_undegraded":"2024-03-26T04:01:50.197056+0000","last_fullsized":"2024-03-26T04:01:50.197056+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.734176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.734176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":367,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.057695+0000","last_change":"2024-03-26T04:01:38.805264+0000","last_active":"2024-03-26T04:01:49.057695+0000","last_peered":"2024-03-26T04:01:49.057695+0000","last_clean":"2024-03-26T04:01:49.057695+0000","last_became_active":"2024-03-26T04:01:38.804806+0000","last_became_peered":"2024-03-26T04:01:38.804806+0000","last_unstale":"2024-03-26T04:01:49.057695+0000","last_undegraded":"2024-03-26T04:01:49.057695+0000","last_fullsized":"2024-03-26T04:01:49.057695+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.390854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.390854+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"303'11645","reported_seq":13785,"reported_epoch":303,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:50.584569+0000","last_change":"2024-03-26T04:01:37.151708+0000","last_active":"2024-03-26T04:01:50.584569+0000","last_peered":"2024-03-26T04:01:50.584569+0000","last_clean":"2024-03-26T04:01:34.720321+0000","last_became_active":"2024-03-26T04:01:37.055762+0000","last_became_peered":"2024-03-26T04:01:37.055762+0000","last_unstale":"2024-03-26T04:01:50.584569+0000","last_undegraded":"2024-03-26T04:01:50.584569+0000","last_fullsized":"2024-03-26T04:01:50.584569+0000","mapping_epoch":294,"log_start":"284'8094","ondisk_log_start":"284'8094","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'9931","last_scrub_stamp":"2024-03-26T04:01:22.378897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.378897+0000","objects_scrubbed":6899,"log_size":3551,"log_dups_size":0,"ondisk_log_size":3551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334954496,"num_objects":5111,"num_object_clones":0,"num_object_copies":15333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4191,"num_objects_unfound":0,"num_objects_dirty":5111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11645,"num_write_kb":536192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":63569920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,5],"avail_no_missing":["1","3","5"],"object_location_counts":[{"shards":"1,3,5","objects":5111}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":533,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.199366+0000","last_change":"2024-03-26T04:01:10.378155+0000","last_active":"2024-03-26T04:01:50.199366+0000","last_peered":"2024-03-26T04:01:50.199366+0000","last_clean":"2024-03-26T04:01:50.199366+0000","last_became_active":"2024-03-26T03:57:49.377628+0000","last_became_peered":"2024-03-26T03:57:49.377628+0000","last_unstale":"2024-03-26T04:01:50.199366+0000","last_undegraded":"2024-03-26T04:01:50.199366+0000","last_fullsized":"2024-03-26T04:01:50.199366+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:10.378037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:10.378037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:31.018590+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":327,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189102+0000","last_change":"2024-03-26T04:01:35.805280+0000","last_active":"2024-03-26T04:01:50.189102+0000","last_peered":"2024-03-26T04:01:50.189102+0000","last_clean":"2024-03-26T04:01:50.189102+0000","last_became_active":"2024-03-26T04:01:35.804822+0000","last_became_peered":"2024-03-26T04:01:35.804822+0000","last_unstale":"2024-03-26T04:01:50.189102+0000","last_undegraded":"2024-03-26T04:01:50.189102+0000","last_fullsized":"2024-03-26T04:01:50.189102+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:14.456520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:14.456520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":596,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.202382+0000","last_change":"2024-03-26T04:01:35.858079+0000","last_active":"2024-03-26T04:01:50.202382+0000","last_peered":"2024-03-26T04:01:50.202382+0000","last_clean":"2024-03-26T04:01:50.202382+0000","last_became_active":"2024-03-26T04:01:35.819144+0000","last_became_peered":"2024-03-26T04:01:35.819144+0000","last_unstale":"2024-03-26T04:01:50.202382+0000","last_undegraded":"2024-03-26T04:01:50.202382+0000","last_fullsized":"2024-03-26T04:01:50.202382+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:01:19.454295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.454295+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:47.255199+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"294'14","reported_seq":725,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.197517+0000","last_change":"2024-03-26T04:01:35.837549+0000","last_active":"2024-03-26T04:01:50.197517+0000","last_peered":"2024-03-26T04:01:50.197517+0000","last_clean":"2024-03-26T04:01:50.197517+0000","last_became_active":"2024-03-26T04:01:35.824821+0000","last_became_peered":"2024-03-26T04:01:35.824821+0000","last_unstale":"2024-03-26T04:01:50.197517+0000","last_undegraded":"2024-03-26T04:01:50.197517+0000","last_fullsized":"2024-03-26T04:01:50.197517+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'13","last_scrub_stamp":"2024-03-26T03:59:16.761422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T03:59:16.761422+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":371,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.060980+0000","last_change":"2024-03-26T04:01:38.819636+0000","last_active":"2024-03-26T04:01:49.060980+0000","last_peered":"2024-03-26T04:01:49.060980+0000","last_clean":"2024-03-26T04:01:49.060980+0000","last_became_active":"2024-03-26T04:01:38.816283+0000","last_became_peered":"2024-03-26T04:01:38.816283+0000","last_unstale":"2024-03-26T04:01:49.060980+0000","last_undegraded":"2024-03-26T04:01:49.060980+0000","last_fullsized":"2024-03-26T04:01:49.060980+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:34.423937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.423937+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:52.240185+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"302'11626","reported_seq":11735,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.044295+0000","last_change":"2024-03-26T03:59:23.930568+0000","last_active":"2024-03-26T04:01:50.044295+0000","last_peered":"2024-03-26T04:01:50.044295+0000","last_clean":"2024-03-26T04:01:50.044295+0000","last_became_active":"2024-03-26T03:59:23.930275+0000","last_became_peered":"2024-03-26T03:59:23.930275+0000","last_unstale":"2024-03-26T04:01:50.044295+0000","last_undegraded":"2024-03-26T04:01:50.044295+0000","last_fullsized":"2024-03-26T04:01:50.044295+0000","mapping_epoch":253,"log_start":"284'8038","ondisk_log_start":"284'8038","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":3588,"log_dups_size":0,"ondisk_log_size":3588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333053952,"num_objects":5082,"num_object_clones":0,"num_object_copies":15246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11626,"num_write_kb":534656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":531,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046512+0000","last_change":"2024-03-26T04:01:23.897232+0000","last_active":"2024-03-26T04:01:49.046512+0000","last_peered":"2024-03-26T04:01:49.046512+0000","last_clean":"2024-03-26T04:01:49.046512+0000","last_became_active":"2024-03-26T03:50:45.589676+0000","last_became_peered":"2024-03-26T03:50:45.589676+0000","last_unstale":"2024-03-26T04:01:49.046512+0000","last_undegraded":"2024-03-26T04:01:49.046512+0000","last_fullsized":"2024-03-26T04:01:49.046512+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:23.897123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.897123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:41.318220+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":533,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.198372+0000","last_change":"2024-03-26T04:01:34.494867+0000","last_active":"2024-03-26T04:01:50.198372+0000","last_peered":"2024-03-26T04:01:50.198372+0000","last_clean":"2024-03-26T04:01:50.198372+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:01:50.198372+0000","last_undegraded":"2024-03-26T04:01:50.198372+0000","last_fullsized":"2024-03-26T04:01:50.198372+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.494617+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:03:00.110856+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":628,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042276+0000","last_change":"2024-03-26T04:01:35.830125+0000","last_active":"2024-03-26T04:01:49.042276+0000","last_peered":"2024-03-26T04:01:49.042276+0000","last_clean":"2024-03-26T04:01:49.042276+0000","last_became_active":"2024-03-26T04:01:35.799996+0000","last_became_peered":"2024-03-26T04:01:35.799996+0000","last_unstale":"2024-03-26T04:01:49.042276+0000","last_undegraded":"2024-03-26T04:01:49.042276+0000","last_fullsized":"2024-03-26T04:01:49.042276+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:00:29.239892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.239892+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"215'11","reported_seq":249,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190738+0000","last_change":"2024-03-26T04:00:15.479476+0000","last_active":"2024-03-26T04:01:50.190738+0000","last_peered":"2024-03-26T04:01:50.190738+0000","last_clean":"2024-03-26T04:01:50.190738+0000","last_became_active":"2024-03-26T03:57:49.365972+0000","last_became_peered":"2024-03-26T03:57:49.365972+0000","last_unstale":"2024-03-26T04:01:50.190738+0000","last_undegraded":"2024-03-26T04:01:50.190738+0000","last_fullsized":"2024-03-26T04:01:50.190738+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"215'11","last_scrub_stamp":"2024-03-26T04:00:15.479350+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.479350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":367,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042619+0000","last_change":"2024-03-26T04:01:38.804087+0000","last_active":"2024-03-26T04:01:49.042619+0000","last_peered":"2024-03-26T04:01:49.042619+0000","last_clean":"2024-03-26T04:01:49.042619+0000","last_became_active":"2024-03-26T04:01:38.803692+0000","last_became_peered":"2024-03-26T04:01:38.803692+0000","last_unstale":"2024-03-26T04:01:49.042619+0000","last_undegraded":"2024-03-26T04:01:49.042619+0000","last_fullsized":"2024-03-26T04:01:49.042619+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.174993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.174993+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"302'11500","reported_seq":22341,"reported_epoch":302,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:01:49.423143+0000","last_change":"2024-03-26T04:01:36.934296+0000","last_active":"2024-03-26T04:01:49.423143+0000","last_peered":"2024-03-26T04:01:49.423143+0000","last_clean":"2024-03-26T04:01:34.716478+0000","last_became_active":"2024-03-26T04:01:36.850641+0000","last_became_peered":"2024-03-26T04:01:36.850641+0000","last_unstale":"2024-03-26T04:01:49.423143+0000","last_undegraded":"2024-03-26T04:01:49.423143+0000","last_fullsized":"2024-03-26T04:01:49.423143+0000","mapping_epoch":294,"log_start":"283'7622","ondisk_log_start":"283'7622","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"287'9210","last_scrub_stamp":"2024-03-26T04:01:12.877147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:12.877147+0000","objects_scrubbed":7452,"log_size":3878,"log_dups_size":0,"ondisk_log_size":3878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"no scrub is scheduled","scrub_duration":1998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":333119512,"num_objects":5084,"num_object_clones":0,"num_object_copies":15252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":105,"num_objects_unfound":0,"num_objects_dirty":5084,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11500,"num_write_kb":530625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5320,"num_bytes_recovered":348520472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,5],"avail_no_missing":["7","1","5"],"object_location_counts":[{"shards":"1,5,7","objects":5084}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":513,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041800+0000","last_change":"2024-03-26T04:00:10.189235+0000","last_active":"2024-03-26T04:01:49.041800+0000","last_peered":"2024-03-26T04:01:49.041800+0000","last_clean":"2024-03-26T04:01:49.041800+0000","last_became_active":"2024-03-26T03:57:49.359540+0000","last_became_peered":"2024-03-26T03:57:49.359540+0000","last_unstale":"2024-03-26T04:01:49.041800+0000","last_undegraded":"2024-03-26T04:01:49.041800+0000","last_fullsized":"2024-03-26T04:01:49.041800+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.189077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.189077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":193,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.058546+0000","last_change":"2024-03-26T03:59:12.184503+0000","last_active":"2024-03-26T04:01:49.058546+0000","last_peered":"2024-03-26T04:01:49.058546+0000","last_clean":"2024-03-26T04:01:49.058546+0000","last_became_active":"2024-03-26T03:57:49.361303+0000","last_became_peered":"2024-03-26T03:57:49.361303+0000","last_unstale":"2024-03-26T04:01:49.058546+0000","last_undegraded":"2024-03-26T04:01:49.058546+0000","last_fullsized":"2024-03-26T04:01:49.058546+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:12.184357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:12.184357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":105,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047129+0000","last_change":"2024-03-26T04:01:17.362143+0000","last_active":"2024-03-26T04:01:49.047129+0000","last_peered":"2024-03-26T04:01:49.047129+0000","last_clean":"2024-03-26T04:01:49.047129+0000","last_became_active":"2024-03-26T04:00:25.423339+0000","last_became_peered":"2024-03-26T04:00:25.423339+0000","last_unstale":"2024-03-26T04:01:49.047129+0000","last_undegraded":"2024-03-26T04:01:49.047129+0000","last_fullsized":"2024-03-26T04:01:49.047129+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:01:17.361690+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:01:17.361690+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:24.732587+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":525,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041873+0000","last_change":"2024-03-26T04:00:34.128491+0000","last_active":"2024-03-26T04:01:49.041873+0000","last_peered":"2024-03-26T04:01:49.041873+0000","last_clean":"2024-03-26T04:01:49.041873+0000","last_became_active":"2024-03-26T03:57:49.359795+0000","last_became_peered":"2024-03-26T03:57:49.359795+0000","last_unstale":"2024-03-26T04:01:49.041873+0000","last_undegraded":"2024-03-26T04:01:49.041873+0000","last_fullsized":"2024-03-26T04:01:49.041873+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:34.128366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:16.977862+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.128366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:57.367315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":317,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041939+0000","last_change":"2024-03-26T04:01:38.793314+0000","last_active":"2024-03-26T04:01:49.041939+0000","last_peered":"2024-03-26T04:01:49.041939+0000","last_clean":"2024-03-26T04:01:49.041939+0000","last_became_active":"2024-03-26T04:01:38.792965+0000","last_became_peered":"2024-03-26T04:01:38.792965+0000","last_unstale":"2024-03-26T04:01:49.041939+0000","last_undegraded":"2024-03-26T04:01:49.041939+0000","last_fullsized":"2024-03-26T04:01:49.041939+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.340085+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:49.207924+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2","version":"302'11477","reported_seq":11586,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190619+0000","last_change":"2024-03-26T03:59:23.932941+0000","last_active":"2024-03-26T04:01:50.190619+0000","last_peered":"2024-03-26T04:01:50.190619+0000","last_clean":"2024-03-26T04:01:50.190619+0000","last_became_active":"2024-03-26T03:59:23.932638+0000","last_became_peered":"2024-03-26T03:59:23.932638+0000","last_unstale":"2024-03-26T04:01:50.190619+0000","last_undegraded":"2024-03-26T04:01:50.190619+0000","last_fullsized":"2024-03-26T04:01:50.190619+0000","mapping_epoch":253,"log_start":"284'7867","ondisk_log_start":"284'7867","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T03:59:22.496532+0000","objects_scrubbed":0,"log_size":3610,"log_dups_size":0,"ondisk_log_size":3610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325255168,"num_objects":4963,"num_object_clones":0,"num_object_copies":14889,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4963,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11477,"num_write_kb":526080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":206,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.196678+0000","last_change":"2024-03-26T04:01:35.817659+0000","last_active":"2024-03-26T04:01:50.196678+0000","last_peered":"2024-03-26T04:01:50.196678+0000","last_clean":"2024-03-26T04:01:50.196678+0000","last_became_active":"2024-03-26T04:01:35.817144+0000","last_became_peered":"2024-03-26T04:01:35.817144+0000","last_unstale":"2024-03-26T04:01:50.196678+0000","last_undegraded":"2024-03-26T04:01:50.196678+0000","last_fullsized":"2024-03-26T04:01:50.196678+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:25.365447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.365447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:54.948982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"294'11","reported_seq":700,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043229+0000","last_change":"2024-03-26T04:01:35.834880+0000","last_active":"2024-03-26T04:01:49.043229+0000","last_peered":"2024-03-26T04:01:49.043229+0000","last_clean":"2024-03-26T04:01:49.043229+0000","last_became_active":"2024-03-26T04:01:35.811486+0000","last_became_peered":"2024-03-26T04:01:35.811486+0000","last_unstale":"2024-03-26T04:01:49.043229+0000","last_undegraded":"2024-03-26T04:01:49.043229+0000","last_fullsized":"2024-03-26T04:01:49.043229+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:22.364212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.364212+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.190944+0000","last_change":"2024-03-26T03:59:27.171191+0000","last_active":"2024-03-26T04:01:50.190944+0000","last_peered":"2024-03-26T04:01:50.190944+0000","last_clean":"2024-03-26T04:01:50.190944+0000","last_became_active":"2024-03-26T03:57:49.369857+0000","last_became_peered":"2024-03-26T03:57:49.369857+0000","last_unstale":"2024-03-26T04:01:50.190944+0000","last_undegraded":"2024-03-26T04:01:50.190944+0000","last_fullsized":"2024-03-26T04:01:50.190944+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.171113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.171113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":611,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.194808+0000","last_change":"2024-03-26T04:00:15.456301+0000","last_active":"2024-03-26T04:01:50.194808+0000","last_peered":"2024-03-26T04:01:50.194808+0000","last_clean":"2024-03-26T04:01:50.194808+0000","last_became_active":"2024-03-26T03:57:49.380369+0000","last_became_peered":"2024-03-26T03:57:49.380369+0000","last_unstale":"2024-03-26T04:01:50.194808+0000","last_undegraded":"2024-03-26T04:01:50.194808+0000","last_fullsized":"2024-03-26T04:01:50.194808+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:15.456156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:15.456156+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"302'11127","reported_seq":11313,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.387646+0000","last_change":"2024-03-26T04:01:17.733913+0000","last_active":"2024-03-26T04:01:49.387646+0000","last_peered":"2024-03-26T04:01:49.387646+0000","last_clean":"2024-03-26T04:01:49.387646+0000","last_became_active":"2024-03-26T03:59:23.523509+0000","last_became_peered":"2024-03-26T03:59:23.523509+0000","last_unstale":"2024-03-26T04:01:49.387646+0000","last_undegraded":"2024-03-26T04:01:49.387646+0000","last_fullsized":"2024-03-26T04:01:49.387646+0000","mapping_epoch":253,"log_start":"282'7197","ondisk_log_start":"282'7197","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'9252","last_scrub_stamp":"2024-03-26T04:01:17.733731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:17.733731+0000","objects_scrubbed":6953,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:26.511986+0000","scrub_duration":1980,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328663040,"num_objects":5015,"num_object_clones":0,"num_object_copies":15045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5015,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11127,"num_write_kb":516544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":199,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.062339+0000","last_change":"2024-03-26T04:01:35.821346+0000","last_active":"2024-03-26T04:01:49.062339+0000","last_peered":"2024-03-26T04:01:49.062339+0000","last_clean":"2024-03-26T04:01:49.062339+0000","last_became_active":"2024-03-26T04:01:35.820570+0000","last_became_peered":"2024-03-26T04:01:35.820570+0000","last_unstale":"2024-03-26T04:01:49.062339+0000","last_undegraded":"2024-03-26T04:01:49.062339+0000","last_fullsized":"2024-03-26T04:01:49.062339+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.493215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.493215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":246,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043178+0000","last_change":"2024-03-26T04:01:35.810872+0000","last_active":"2024-03-26T04:01:49.043178+0000","last_peered":"2024-03-26T04:01:49.043178+0000","last_clean":"2024-03-26T04:01:49.043178+0000","last_became_active":"2024-03-26T04:01:35.810536+0000","last_became_peered":"2024-03-26T04:01:35.810536+0000","last_unstale":"2024-03-26T04:01:49.043178+0000","last_undegraded":"2024-03-26T04:01:49.043178+0000","last_fullsized":"2024-03-26T04:01:49.043178+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:10.363768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:00:10.363768+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":521,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042025+0000","last_change":"2024-03-26T04:00:17.365750+0000","last_active":"2024-03-26T04:01:49.042025+0000","last_peered":"2024-03-26T04:01:49.042025+0000","last_clean":"2024-03-26T04:01:49.042025+0000","last_became_active":"2024-03-26T03:48:44.015185+0000","last_became_peered":"2024-03-26T03:48:44.015185+0000","last_unstale":"2024-03-26T04:01:49.042025+0000","last_undegraded":"2024-03-26T04:01:49.042025+0000","last_fullsized":"2024-03-26T04:01:49.042025+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":628,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042892+0000","last_change":"2024-03-26T03:59:14.317073+0000","last_active":"2024-03-26T04:01:49.042892+0000","last_peered":"2024-03-26T04:01:49.042892+0000","last_clean":"2024-03-26T04:01:49.042892+0000","last_became_active":"2024-03-26T03:57:49.363785+0000","last_became_peered":"2024-03-26T03:57:49.363785+0000","last_unstale":"2024-03-26T04:01:49.042892+0000","last_undegraded":"2024-03-26T04:01:49.042892+0000","last_fullsized":"2024-03-26T04:01:49.042892+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T03:59:14.316934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:14.316934+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":321,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.047237+0000","last_change":"2024-03-26T04:01:38.812313+0000","last_active":"2024-03-26T04:01:49.047237+0000","last_peered":"2024-03-26T04:01:49.047237+0000","last_clean":"2024-03-26T04:01:49.047237+0000","last_became_active":"2024-03-26T04:01:38.811921+0000","last_became_peered":"2024-03-26T04:01:38.811921+0000","last_unstale":"2024-03-26T04:01:49.047237+0000","last_undegraded":"2024-03-26T04:01:49.047237+0000","last_fullsized":"2024-03-26T04:01:49.047237+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.276421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.276421+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:49.207949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":423697,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"303'11566","reported_seq":11760,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.468424+0000","last_change":"2024-03-26T04:01:14.609880+0000","last_active":"2024-03-26T04:01:50.468424+0000","last_peered":"2024-03-26T04:01:50.468424+0000","last_clean":"2024-03-26T04:01:50.468424+0000","last_became_active":"2024-03-26T03:59:23.931459+0000","last_became_peered":"2024-03-26T03:59:23.931459+0000","last_unstale":"2024-03-26T04:01:50.468424+0000","last_undegraded":"2024-03-26T04:01:50.468424+0000","last_fullsized":"2024-03-26T04:01:50.468424+0000","mapping_epoch":253,"log_start":"284'8021","ondisk_log_start":"284'8021","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"288'9229","last_scrub_stamp":"2024-03-26T04:01:14.609043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:14.609043+0000","objects_scrubbed":7463,"log_size":3545,"log_dups_size":0,"ondisk_log_size":3545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:37.397008+0000","scrub_duration":3147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329252864,"num_objects":5024,"num_object_clones":0,"num_object_copies":15072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11566,"num_write_kb":530880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":544,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.046089+0000","last_change":"2024-03-26T04:00:27.251209+0000","last_active":"2024-03-26T04:01:49.046089+0000","last_peered":"2024-03-26T04:01:49.046089+0000","last_clean":"2024-03-26T04:01:49.046089+0000","last_became_active":"2024-03-26T03:57:49.360928+0000","last_became_peered":"2024-03-26T03:57:49.360928+0000","last_unstale":"2024-03-26T04:01:49.046089+0000","last_undegraded":"2024-03-26T04:01:49.046089+0000","last_fullsized":"2024-03-26T04:01:49.046089+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.251082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.251082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:56.805532+0000","scrub_duration":181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":520,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.195270+0000","last_change":"2024-03-26T04:01:29.373365+0000","last_active":"2024-03-26T04:01:50.195270+0000","last_peered":"2024-03-26T04:01:50.195270+0000","last_clean":"2024-03-26T04:01:50.195270+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:01:50.195270+0000","last_undegraded":"2024-03-26T04:01:50.195270+0000","last_fullsized":"2024-03-26T04:01:50.195270+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:01:29.373227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:54.752886+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":472,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.041873+0000","last_change":"2024-03-26T03:59:27.313345+0000","last_active":"2024-03-26T04:01:49.041873+0000","last_peered":"2024-03-26T04:01:49.041873+0000","last_clean":"2024-03-26T04:01:49.041873+0000","last_became_active":"2024-03-26T03:57:49.359830+0000","last_became_peered":"2024-03-26T03:57:49.359830+0000","last_unstale":"2024-03-26T04:01:49.041873+0000","last_undegraded":"2024-03-26T04:01:49.041873+0000","last_fullsized":"2024-03-26T04:01:49.041873+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T03:59:27.313255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T03:59:27.313255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":565,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043141+0000","last_change":"2024-03-26T03:59:26.340091+0000","last_active":"2024-03-26T04:01:49.043141+0000","last_peered":"2024-03-26T04:01:49.043141+0000","last_clean":"2024-03-26T04:01:49.043141+0000","last_became_active":"2024-03-26T03:48:46.021176+0000","last_became_peered":"2024-03-26T03:48:46.021176+0000","last_unstale":"2024-03-26T04:01:49.043141+0000","last_undegraded":"2024-03-26T04:01:49.043141+0000","last_fullsized":"2024-03-26T04:01:49.043141+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T03:59:26.339621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T03:59:26.339621+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":818,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045926+0000","last_change":"2024-03-26T04:00:33.001695+0000","last_active":"2024-03-26T04:01:49.045926+0000","last_peered":"2024-03-26T04:01:49.045926+0000","last_clean":"2024-03-26T04:01:49.045926+0000","last_became_active":"2024-03-26T04:00:32.428836+0000","last_became_peered":"2024-03-26T04:00:32.428836+0000","last_unstale":"2024-03-26T04:01:49.045926+0000","last_undegraded":"2024-03-26T04:01:49.045926+0000","last_fullsized":"2024-03-26T04:01:49.045926+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:00:33.001580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.001580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:55.639165+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"302'11809","reported_seq":11996,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.028043+0000","last_change":"2024-03-26T04:01:21.209476+0000","last_active":"2024-03-26T04:01:50.028043+0000","last_peered":"2024-03-26T04:01:50.028043+0000","last_clean":"2024-03-26T04:01:50.028043+0000","last_became_active":"2024-03-26T03:59:23.930238+0000","last_became_peered":"2024-03-26T03:59:23.930238+0000","last_unstale":"2024-03-26T04:01:50.028043+0000","last_undegraded":"2024-03-26T04:01:50.028043+0000","last_fullsized":"2024-03-26T04:01:50.028043+0000","mapping_epoch":253,"log_start":"284'8237","ondisk_log_start":"284'8237","created":253,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"289'9927","last_scrub_stamp":"2024-03-26T04:01:21.209345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:59:22.496532+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.209345+0000","objects_scrubbed":6954,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:21.505824+0000","scrub_duration":2566,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327221248,"num_objects":4993,"num_object_clones":0,"num_object_copies":14979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4993,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11809,"num_write_kb":537664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":508,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.043281+0000","last_change":"2024-03-26T04:00:13.290591+0000","last_active":"2024-03-26T04:01:49.043281+0000","last_peered":"2024-03-26T04:01:49.043281+0000","last_clean":"2024-03-26T04:01:49.043281+0000","last_became_active":"2024-03-26T03:48:50.038080+0000","last_became_peered":"2024-03-26T03:48:50.038080+0000","last_unstale":"2024-03-26T04:01:49.043281+0000","last_undegraded":"2024-03-26T04:01:49.043281+0000","last_fullsized":"2024-03-26T04:01:49.043281+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:13.290512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:00:13.290512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":52,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"294'11","reported_seq":724,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.045322+0000","last_change":"2024-03-26T04:01:35.844607+0000","last_active":"2024-03-26T04:01:49.045322+0000","last_peered":"2024-03-26T04:01:49.045322+0000","last_clean":"2024-03-26T04:01:49.045322+0000","last_became_active":"2024-03-26T04:01:35.818800+0000","last_became_peered":"2024-03-26T04:01:35.818800+0000","last_unstale":"2024-03-26T04:01:49.045322+0000","last_undegraded":"2024-03-26T04:01:49.045322+0000","last_fullsized":"2024-03-26T04:01:49.045322+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:01:34.236967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.236967+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:02:36.789179+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":195,"reported_epoch":303,"state":"active+clean","last_fresh":"2024-03-26T04:01:50.189183+0000","last_change":"2024-03-26T04:00:18.449990+0000","last_active":"2024-03-26T04:01:50.189183+0000","last_peered":"2024-03-26T04:01:50.189183+0000","last_clean":"2024-03-26T04:01:50.189183+0000","last_became_active":"2024-03-26T03:57:49.360243+0000","last_became_peered":"2024-03-26T03:57:49.360243+0000","last_unstale":"2024-03-26T04:01:50.189183+0000","last_undegraded":"2024-03-26T04:01:50.189183+0000","last_fullsized":"2024-03-26T04:01:50.189183+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:18.449869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.449869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":600,"reported_epoch":302,"state":"active+clean","last_fresh":"2024-03-26T04:01:49.042401+0000","last_change":"2024-03-26T04:00:37.112949+0000","last_active":"2024-03-26T04:01:49.042401+0000","last_peered":"2024-03-26T04:01:49.042401+0000","last_clean":"2024-03-26T04:01:49.042401+0000","last_became_active":"2024-03-26T03:57:49.363791+0000","last_became_peered":"2024-03-26T03:57:49.363791+0000","last_unstale":"2024-03-26T04:01:49.042401+0000","last_undegraded":"2024-03-26T04:01:49.042401+0000","last_fullsized":"2024-03-26T04:01:49.042401+0000","mapping_epoch":214,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":215,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:00:37.112810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:37.112810+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:01:59.410286+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5291638808,"num_objects":80745,"num_object_clones":0,"num_object_copies":242235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":34105,"num_objects_unfound":0,"num_objects_dirty":80745,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":184165,"num_write_kb":8477057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12236,"num_bytes_recovered":801505304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16226201600,"data_stored":16226189384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60392,"ondisk_log_size":60392,"up":48,"acting":48,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":777,"num_read_kb":772,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":488689,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":54,"num_read_kb":80,"num_write":49,"num_write_kb":658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1073396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2786,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1404,"num_read_kb":1129,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":347,"num_bytes_recovered":6150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":13570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":908,"ondisk_log_size":908,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070625,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3347436,"kb_used_data":2941412,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":90397716,"statfs":{"total":95995035648,"available":92567261184,"internally_reserved":0,"allocated":3012005888,"data_stored":3011378454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":415758794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748364967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575040,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3559720,"kb_used_data":3014872,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":90185432,"statfs":{"total":95995035648,"available":92349882368,"internally_reserved":0,"allocated":3087228928,"data_stored":3086562085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738565,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2155476,"kb_used_data":1695304,"kb_used_omap":1,"kb_used_meta":460158,"kb_avail":91589676,"statfs":{"total":95995035648,"available":93787828224,"internally_reserved":0,"allocated":1735991296,"data_stored":1735341672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":471202245},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1972064,"kb_used_data":1445268,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91773088,"statfs":{"total":95995035648,"available":93975642112,"internally_reserved":0,"allocated":1479954432,"data_stored":1479349043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182581,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3075404,"kb_used_data":2651264,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":90669748,"statfs":{"total":95995035648,"available":92845821952,"internally_reserved":0,"allocated":2714894336,"data_stored":2714246170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":434239943},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953648,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2055096,"kb_used_data":1618540,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":91690056,"statfs":{"total":95995035648,"available":93890617344,"internally_reserved":0,"allocated":1657384960,"data_stored":1656755179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":2024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1734606848,"data_stored":1734606848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3086028800,"data_stored":3086024728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1478361088,"data_stored":1478361088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2705063936,"data_stored":2705063936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1657606144,"data_stored":1657602072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2571829248,"data_stored":2571829248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2992705536,"data_stored":2992701464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:01:55.693 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-26T04:01:55.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:55.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:55.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:55 smithi086 ceph-mon[32965]: from='client.25349 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:01:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:55 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:01:55.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:55.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:55.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:56.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:56.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:56.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:56.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:56.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:56.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:56.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:56.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:56.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:56.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:56.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:56.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.723 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:56.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:01:56.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:56 smithi086 ceph-mon[32965]: pgmap v605: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.06k op/s; 31825/235977 objects misplaced (13.486%); 62 MiB/s, 987 objects/s recovering 2024-03-26T04:01:56.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:56.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:56.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:56.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:56 smithi006 ceph-mon[29446]: pgmap v605: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.06k op/s; 31825/235977 objects misplaced (13.486%); 62 MiB/s, 987 objects/s recovering 2024-03-26T04:01:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:56 smithi006 ceph-mon[37501]: pgmap v605: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.06k op/s; 31825/235977 objects misplaced (13.486%); 62 MiB/s, 987 objects/s recovering 2024-03-26T04:01:56.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:57.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:57.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:57.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:57.183 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:01:57.183 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:01:57.183 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:01:57.183 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:01:57.183 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:01:57.242 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:57.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:57.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:57.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:57.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:57.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:57.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:57.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:01:57.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:57.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:01:57.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:01:57.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.832 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:57.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:01:57.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:57.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:57.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:58.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:58.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:58.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:58.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:58.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.390 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:58.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:58.532 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:58.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:01:58.641 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:01:58 smithi086 ceph-mon[32965]: pgmap v606: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 934 op/s; 28666/230361 objects misplaced (12.444%); 46 MiB/s, 730 objects/s recovering 2024-03-26T04:01:58.671 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:01:58.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:58.807 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:01:58 smithi006 ceph-mon[37501]: pgmap v606: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 934 op/s; 28666/230361 objects misplaced (12.444%); 46 MiB/s, 730 objects/s recovering 2024-03-26T04:01:58.807 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:01:58 smithi006 ceph-mon[29446]: pgmap v606: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.7 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 934 op/s; 28666/230361 objects misplaced (12.444%); 46 MiB/s, 730 objects/s recovering 2024-03-26T04:01:58.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.823 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:58.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:58.966 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:58.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:59.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:59.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:59.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:59.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:59.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:01:59.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:59.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:01:59.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T04:01:59.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:01:59.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:59.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:01:59.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:01:59.623 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:01:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:59.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:01:59.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:01:59.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:01:59.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:01:59.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:01:59.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:01:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:01:59.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:00.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:00.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:00.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:00.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:00.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:00.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:00.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:00.352 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:00.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:00.508 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:00.564 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:02:00.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:00.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:00.694 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7, 3] out_osds: [6, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:02:00.694 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:02:00.694 INFO:tasks.thrashosds.thrasher:Setting full ratio to .001 2024-03-26T04:02:00.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set-full-ratio .001 2024-03-26T04:02:00.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:00.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[37501]: pgmap v607: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 907 op/s; 27843/226695 objects misplaced (12.282%); 40 MiB/s, 646 objects/s recovering 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[29446]: pgmap v607: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 907 op/s; 27843/226695 objects misplaced (12.282%); 40 MiB/s, 646 objects/s recovering 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:00.820 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:00 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:02:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:00 smithi086 ceph-mon[32965]: pgmap v607: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 907 op/s; 27843/226695 objects misplaced (12.282%); 40 MiB/s, 646 objects/s recovering 2024-03-26T04:02:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:00 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:02:00.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:00.936 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:00.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:01.037 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.100 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:01.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:01.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:01.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:01.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:01.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:01.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:01.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:01.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.485 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:01.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:01.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:01.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:01.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:01.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:01.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:01.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:01 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:01.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:01 smithi086 ceph-mon[32965]: osdmap e305: 8 total, 8 up, 6 in 2024-03-26T04:02:01.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[37501]: osdmap e305: 8 total, 8 up, 6 in 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2827613563' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:01 smithi006 ceph-mon[29446]: osdmap e305: 8 total, 8 up, 6 in 2024-03-26T04:02:01.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:01.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:01.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:01.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:02.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:02.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:02.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:02.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:02.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.305 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:02.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:02.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:02.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:02.515 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:02:02.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:02.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[37501]: pgmap v609: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:02.560 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:02:02.565 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:02:02.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:02.606 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:02:02.652 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:02:02.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:02.693 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:02:02.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:02.740 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:02:02.785 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:02:02.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:02.826 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3178774237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[29446]: pgmap v609: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:02.827 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:02 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3178774237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:02:02.832 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:02:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:02 smithi086 ceph-mon[32965]: pgmap v609: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:02:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:02:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:02 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:02.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:02 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3178774237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:02.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:02.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:02.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:02.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:02.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:03.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:03.129 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:03.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:03.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:03.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:03.277 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:03.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:03.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:03.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:03.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:03.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:03.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:03.732 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:03.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:03.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:03.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:03.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:03.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:04.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:04.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:04.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:04.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:04.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:04.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:04.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:04.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:04.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:04.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:04.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:04.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:04 smithi006 ceph-mon[37501]: pgmap v610: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:04.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:04 smithi006 ceph-mon[37501]: osdmap e306: 8 total, 8 up, 6 in 2024-03-26T04:02:04.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:04 smithi006 ceph-mon[29446]: pgmap v610: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:04.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:04 smithi006 ceph-mon[29446]: osdmap e306: 8 total, 8 up, 6 in 2024-03-26T04:02:04.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.654 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:04.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:04.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:04.798 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:04.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:04 smithi086 ceph-mon[32965]: pgmap v610: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 903 op/s; 24333/215832 objects misplaced (11.274%); 39 MiB/s, 631 objects/s recovering 2024-03-26T04:02:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:04 smithi086 ceph-mon[32965]: osdmap e306: 8 total, 8 up, 6 in 2024-03-26T04:02:04.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:04.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:04.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:04.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:05.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:05.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:05.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:02:05.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:05.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:05.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:05.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:05.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:05.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:05.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:05.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:05.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:05.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:05.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:05.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:05.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:05.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:02:05.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:05.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:05 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:05.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:05.863 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:05 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:05.864 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:05 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:05.931 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:05.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:05.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:06.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:06.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:06.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:06.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:06.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:06.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:06.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:06.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:06.486 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:06.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:06.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:06.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:06.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:06.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:06 smithi086 ceph-mon[32965]: pgmap v612: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 938 op/s; 22632/207861 objects misplaced (10.888%); 29 MiB/s, 467 objects/s recovering 2024-03-26T04:02:06.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:06.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:06 smithi006 ceph-mon[37501]: pgmap v612: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 938 op/s; 22632/207861 objects misplaced (10.888%); 29 MiB/s, 467 objects/s recovering 2024-03-26T04:02:06.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:06.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:06.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:06 smithi006 ceph-mon[29446]: pgmap v612: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 938 op/s; 22632/207861 objects misplaced (10.888%); 29 MiB/s, 467 objects/s recovering 2024-03-26T04:02:06.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:07.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:07.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:07.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:07.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:07.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:07.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:07.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:07.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:07.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:07.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:07.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:07.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:07.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:07.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:07.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:08.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:08.088 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:08.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:08.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:08.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:08.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:08.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:08.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:08.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:08.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:08.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.473 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:08.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:08.613 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:08.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:08.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:08.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:08.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:08 smithi086 ceph-mon[32965]: pgmap v613: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 20200/201696 objects misplaced (10.015%); 35 MiB/s, 562 objects/s recovering 2024-03-26T04:02:08.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:08.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:08.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:08.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:08 smithi006 ceph-mon[37501]: pgmap v613: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 20200/201696 objects misplaced (10.015%); 35 MiB/s, 562 objects/s recovering 2024-03-26T04:02:08.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:08 smithi006 ceph-mon[29446]: pgmap v613: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 20200/201696 objects misplaced (10.015%); 35 MiB/s, 562 objects/s recovering 2024-03-26T04:02:08.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:09.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:09.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:09.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:09.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:09.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:09.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:09.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:09.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:09.544 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:09.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.674 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:09.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:09.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:09.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:09.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:09.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:10.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:10.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:10.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:10.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:10.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:10.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:10.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:10.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:10.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:10.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.453 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:10.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:10.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:10.596 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:10.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:10.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:10.729 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:10.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:10 smithi086 ceph-mon[32965]: pgmap v614: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 19524/198540 objects misplaced (9.834%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:02:10.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:10.857 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:10.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:10.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:10 smithi006 ceph-mon[37501]: pgmap v614: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 19524/198540 objects misplaced (9.834%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:02:10.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:10 smithi006 ceph-mon[29446]: pgmap v614: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 778 op/s; 19524/198540 objects misplaced (9.834%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:02:10.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:11.002 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:11.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:11.246 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:11.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:11.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:11.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.577 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:11.678 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:11.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:11.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:11.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:11.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:11 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:11 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:11.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:11.973 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:11.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:12.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:11 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:12.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:12.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:12.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:12.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:12.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:12.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:12.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:12.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:12.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:12.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:12.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:12.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:12.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:12.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:12.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:12.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:12.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:13.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:13.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:12 smithi086 ceph-mon[32965]: pgmap v615: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16674/189705 objects misplaced (8.789%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:02:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:12 smithi006 ceph-mon[37501]: pgmap v615: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16674/189705 objects misplaced (8.789%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:02:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:12 smithi006 ceph-mon[29446]: pgmap v615: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 871 op/s; 16674/189705 objects misplaced (8.789%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:02:13.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:13.298 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:13.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:13.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:13.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:13.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:13.626 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:13.627 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:13.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:13.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:13.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:13.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:13.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:13 smithi006 ceph-mon[37501]: osdmap e307: 8 total, 8 up, 6 in 2024-03-26T04:02:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:13 smithi006 ceph-mon[37501]: osdmap e308: 8 total, 8 up, 6 in 2024-03-26T04:02:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:13 smithi006 ceph-mon[29446]: osdmap e307: 8 total, 8 up, 6 in 2024-03-26T04:02:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:13 smithi006 ceph-mon[29446]: osdmap e308: 8 total, 8 up, 6 in 2024-03-26T04:02:13.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:13.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:13.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:14.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.085 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:14.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:13 smithi086 ceph-mon[32965]: osdmap e307: 8 total, 8 up, 6 in 2024-03-26T04:02:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:13 smithi086 ceph-mon[32965]: osdmap e308: 8 total, 8 up, 6 in 2024-03-26T04:02:14.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:14.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:14.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:14.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:14.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:14.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:14.480 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:14.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.610 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:14.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:14.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:14.711 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:14.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:14.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:14.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:14.881 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:14.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:14.973 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:14 smithi086 ceph-mon[32965]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 964 op/s; 16674/189705 objects misplaced (8.789%); 35 MiB/s, 559 objects/s recovering 2024-03-26T04:02:14.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:15.007 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:15.108 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:14 smithi006 ceph-mon[37501]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 964 op/s; 16674/189705 objects misplaced (8.789%); 35 MiB/s, 559 objects/s recovering 2024-03-26T04:02:15.109 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:14 smithi006 ceph-mon[29446]: pgmap v617: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 3.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 964 op/s; 16674/189705 objects misplaced (8.789%); 35 MiB/s, 559 objects/s recovering 2024-03-26T04:02:15.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:15.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:15.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:15.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:15.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:15.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:15.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:15.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.517 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:15.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:15.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:15.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:15.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:15.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:15.916 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:15.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:15.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:16.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:16.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:15 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:15 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:15 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:16.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:16.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:16.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:16.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:16.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.312 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:16.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:16.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:16.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:16.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:16.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:16.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:16.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:16.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:16.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:16.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:16.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:16.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:16.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:16 smithi006 ceph-mon[37501]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 15553/184677 objects misplaced (8.422%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:02:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:16 smithi006 ceph-mon[29446]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 15553/184677 objects misplaced (8.422%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:02:16.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:16.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:16.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:17.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:16 smithi086 ceph-mon[32965]: pgmap v619: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 710 op/s; 15553/184677 objects misplaced (8.422%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:02:17.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:17.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:17.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:17.237 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:17.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:17.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:17.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:17.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:17.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:17.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:17.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:17.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:17.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:17.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:17.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:17.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:18.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.033 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:18.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:18.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:18.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:18.261 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:18.303 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:18.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:18.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:18.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:18.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:18.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:18.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:18.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:18.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:18.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:18.990 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:18.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:18 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:02:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:18 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:02:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:18 smithi086 ceph-mon[32965]: pgmap v620: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 12768/180816 objects misplaced (7.061%); 39 MiB/s, 626 objects/s recovering 2024-03-26T04:02:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:18 smithi086 ceph-mon[32965]: osdmap e309: 8 total, 8 up, 6 in 2024-03-26T04:02:19.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.118 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:19.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:02:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:02:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[37501]: pgmap v620: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 12768/180816 objects misplaced (7.061%); 39 MiB/s, 626 objects/s recovering 2024-03-26T04:02:19.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[37501]: osdmap e309: 8 total, 8 up, 6 in 2024-03-26T04:02:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:02:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:02:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[29446]: pgmap v620: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.7 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 738 op/s; 12768/180816 objects misplaced (7.061%); 39 MiB/s, 626 objects/s recovering 2024-03-26T04:02:19.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:18 smithi006 ceph-mon[29446]: osdmap e309: 8 total, 8 up, 6 in 2024-03-26T04:02:19.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:19.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.259 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:19.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:19.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:19.384 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:19.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:19.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:19.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.511 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:19.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:19.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:19.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:19.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:19.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:19.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:19.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:19.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:19.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:19.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:19.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:20.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:20.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:20.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.161 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:20.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:20.297 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:20.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:20.425 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:20.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:20.564 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:20.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:20.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:20.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:20.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:20 smithi086 ceph-mon[32965]: osdmap e310: 8 total, 8 up, 6 in 2024-03-26T04:02:20.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:20 smithi086 ceph-mon[32965]: pgmap v623: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 637 op/s; 12768/178248 objects misplaced (7.163%); 33 MiB/s, 532 objects/s recovering 2024-03-26T04:02:20.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:20 smithi006 ceph-mon[37501]: osdmap e310: 8 total, 8 up, 6 in 2024-03-26T04:02:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:20 smithi006 ceph-mon[37501]: pgmap v623: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 637 op/s; 12768/178248 objects misplaced (7.163%); 33 MiB/s, 532 objects/s recovering 2024-03-26T04:02:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:20 smithi006 ceph-mon[29446]: osdmap e310: 8 total, 8 up, 6 in 2024-03-26T04:02:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:20 smithi006 ceph-mon[29446]: pgmap v623: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 637 op/s; 12768/178248 objects misplaced (7.163%); 33 MiB/s, 532 objects/s recovering 2024-03-26T04:02:20.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:20.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:20.946 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:20.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:21.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:21.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:21.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:21.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:21.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:21.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:21.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:21.493 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:21.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:21.619 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:21.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:21.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:21.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:21.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:21.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:21.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:21.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:21.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:22.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:22.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:22.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:22.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:22.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:22.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:22.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:22.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:22.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:22.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:22.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:22.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:22.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:22.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:22.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:22.791 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:22 smithi086 ceph-mon[32965]: osdmap e311: 8 total, 8 up, 6 in 2024-03-26T04:02:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:22 smithi086 ceph-mon[32965]: pgmap v625: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 801 op/s; 10295/170268 objects misplaced (6.046%); 42 MiB/s, 677 objects/s recovering 2024-03-26T04:02:22.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:22 smithi006 ceph-mon[37501]: osdmap e311: 8 total, 8 up, 6 in 2024-03-26T04:02:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:22 smithi006 ceph-mon[37501]: pgmap v625: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 801 op/s; 10295/170268 objects misplaced (6.046%); 42 MiB/s, 677 objects/s recovering 2024-03-26T04:02:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:22 smithi006 ceph-mon[29446]: osdmap e311: 8 total, 8 up, 6 in 2024-03-26T04:02:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:22 smithi006 ceph-mon[29446]: pgmap v625: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 801 op/s; 10295/170268 objects misplaced (6.046%); 42 MiB/s, 677 objects/s recovering 2024-03-26T04:02:22.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:22.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:22.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:23.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.058 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:23.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:23.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:23.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:23.287 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:23.323 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:23.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:23.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:23.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:23.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:23.600 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:23.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:23.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:23.732 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:23 smithi006 ceph-mon[37501]: osdmap e312: 8 total, 8 up, 6 in 2024-03-26T04:02:23.732 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:23 smithi006 ceph-mon[29446]: osdmap e312: 8 total, 8 up, 6 in 2024-03-26T04:02:23.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T04:02:23.757 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:23 smithi086 ceph-mon[32965]: osdmap e312: 8 total, 8 up, 6 in 2024-03-26T04:02:23.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:23.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:23.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:23.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:23.989 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:24.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.000 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:24.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:24.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:24.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:24.150 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:24.157 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:24.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:24.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:24.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:24.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:24.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:24.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.516 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:24.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:24.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:24 smithi006 ceph-mon[29446]: pgmap v627: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 887 B/s rd, 0 B/s wr, 610 op/s; 10295/170268 objects misplaced (6.046%); 19 MiB/s, 304 objects/s recovering 2024-03-26T04:02:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:24.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:24.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:24 smithi006 ceph-mon[37501]: pgmap v627: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 887 B/s rd, 0 B/s wr, 610 op/s; 10295/170268 objects misplaced (6.046%); 19 MiB/s, 304 objects/s recovering 2024-03-26T04:02:24.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:24.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:24.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:24.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:24.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:24.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:24.832 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:24 smithi086 ceph-mon[32965]: pgmap v627: 158 pgs: 1 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 151 active+clean; 3.5 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 887 B/s rd, 0 B/s wr, 610 op/s; 10295/170268 objects misplaced (6.046%); 19 MiB/s, 304 objects/s recovering 2024-03-26T04:02:24.832 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:24.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:24.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:24.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:24.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:25.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:25.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:25.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:25.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:24 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:25.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.198 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:25.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:25.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:25.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:25.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:25.470 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:25.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.612 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:25.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:25.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:25.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:25.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:25.812 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:25 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:25.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:25.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:25.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:25.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:25 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:25 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:25.969 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:26.013 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:26.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:26.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:26.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:26.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:26.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:26.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:26.376 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:26.377 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:26.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:26.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:26.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:26.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:26.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:26.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:26.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:26.705 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:26 smithi086 ceph-mon[32965]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 10014/164181 objects misplaced (6.099%); 19 MiB/s, 297 objects/s recovering 2024-03-26T04:02:26.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:26.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:26.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:26 smithi006 ceph-mon[29446]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 10014/164181 objects misplaced (6.099%); 19 MiB/s, 297 objects/s recovering 2024-03-26T04:02:26.827 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:26 smithi006 ceph-mon[37501]: pgmap v628: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.3 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 919 op/s; 10014/164181 objects misplaced (6.099%); 19 MiB/s, 297 objects/s recovering 2024-03-26T04:02:26.855 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:26.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:26.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:26.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:26.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:27.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T04:02:27.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:27.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:27.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:27.253 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:27.343 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:27.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:27.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:27.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:27.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:27.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:27.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:27.660 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:27.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:27.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:27.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:27.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:27.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:28.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:28.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:28.195 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:28.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:28.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:28.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:28.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:28.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:28.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:28.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:28.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:28.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:28.721 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:28.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:28.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:28 smithi086 ceph-mon[32965]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 6959/159219 objects misplaced (4.371%); 35 MiB/s, 563 objects/s recovering 2024-03-26T04:02:28.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:28.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:28 smithi006 ceph-mon[37501]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 6959/159219 objects misplaced (4.371%); 35 MiB/s, 563 objects/s recovering 2024-03-26T04:02:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:28 smithi006 ceph-mon[29446]: pgmap v629: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 793 op/s; 6959/159219 objects misplaced (4.371%); 35 MiB/s, 563 objects/s recovering 2024-03-26T04:02:28.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:28.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:28.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:29.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:29.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:29.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:29.242 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:29.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:29.336 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:29.336 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:29.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:29.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:29.512 INFO:teuthology.orchestra.run.smithi006.stderr:osd set-full-ratio 0.001 2024-03-26T04:02:29.534 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:29.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.660 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:29.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:29.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:29 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-03-26T04:02:29.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:29 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-03-26T04:02:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:29 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:02:29.490+0000 7fd468790700 -1 log_channel(cluster) log [ERR] : Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-03-26T04:02:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:29 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.001}]: dispatch 2024-03-26T04:02:29.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:29.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:29.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:30.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T04:02:30.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:30.067 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:30.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:30.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:30.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.208 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:30.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:30.341 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:30.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:30.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:30.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:30.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:30.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:30.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:30.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.635 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:30.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:30.722 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:30 smithi086 ceph-mon[32965]: pgmap v630: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 6959/155817 objects misplaced (4.466%); 21 MiB/s, 342 objects/s recovering 2024-03-26T04:02:30.722 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:30 smithi086 ceph-mon[32965]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-03-26T04:02:30.722 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:30 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-03-26T04:02:30.722 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:30 smithi086 ceph-mon[32965]: osdmap e313: 8 total, 8 up, 6 in 2024-03-26T04:02:30.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:30.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:30.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:30.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:30.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:30.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[37501]: pgmap v630: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 6959/155817 objects misplaced (4.466%); 21 MiB/s, 342 objects/s recovering 2024-03-26T04:02:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[37501]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-03-26T04:02:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-03-26T04:02:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[37501]: osdmap e313: 8 total, 8 up, 6 in 2024-03-26T04:02:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[29446]: pgmap v630: 158 pgs: 1 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 152 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 6959/155817 objects misplaced (4.466%); 21 MiB/s, 342 objects/s recovering 2024-03-26T04:02:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[29446]: Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-03-26T04:02:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2192075756' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.001}]': finished 2024-03-26T04:02:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:30 smithi006 ceph-mon[29446]: osdmap e313: 8 total, 8 up, 6 in 2024-03-26T04:02:30.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:31.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:31.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:31.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.140 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:31.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:31.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:31.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:31.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:31.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:31.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:31.519 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:31.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:31.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:31 smithi006 ceph-mon[37501]: osdmap e314: 8 total, 8 up, 6 in 2024-03-26T04:02:31.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:31 smithi006 ceph-mon[29446]: osdmap e314: 8 total, 8 up, 6 in 2024-03-26T04:02:31.652 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:31.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:31.674 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:31 smithi086 ceph-mon[32965]: osdmap e314: 8 total, 8 up, 6 in 2024-03-26T04:02:31.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:31.779 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:31.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:31.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:31.893 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:31.893 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:31.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:31.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:31.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:32.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:32.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.033 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:32.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:32.104 INFO:tasks.thrashosds.thrasher:Setting full ratio back to .95 2024-03-26T04:02:32.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set-full-ratio .95 2024-03-26T04:02:32.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:32.157 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:32.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:32.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.286 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:32.286 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:32.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:32.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:32.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T04:02:32.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:32.538 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:32.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:32.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:32.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:32.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:32.799 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:32 smithi006 ceph-mon[37501]: pgmap v633: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 4652/145539 objects misplaced (3.196%); 36 MiB/s, 568 objects/s recovering 2024-03-26T04:02:32.800 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:32 smithi006 ceph-mon[37501]: osdmap e315: 8 total, 8 up, 6 in 2024-03-26T04:02:32.800 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:32 smithi006 ceph-mon[29446]: pgmap v633: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 4652/145539 objects misplaced (3.196%); 36 MiB/s, 568 objects/s recovering 2024-03-26T04:02:32.800 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:32 smithi006 ceph-mon[29446]: osdmap e315: 8 total, 8 up, 6 in 2024-03-26T04:02:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:32.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:32 smithi086 ceph-mon[32965]: pgmap v633: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 4652/145539 objects misplaced (3.196%); 36 MiB/s, 568 objects/s recovering 2024-03-26T04:02:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:32 smithi086 ceph-mon[32965]: osdmap e315: 8 total, 8 up, 6 in 2024-03-26T04:02:32.838 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:32.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:32.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:32.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:32.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:33.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:33.104 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:33.205 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.247 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:33.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:33.385 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:33.487 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:33.489 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:33.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:33 smithi006 ceph-mon[37501]: osdmap e316: 8 total, 8 up, 6 in 2024-03-26T04:02:33.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:33 smithi006 ceph-mon[29446]: osdmap e316: 8 total, 8 up, 6 in 2024-03-26T04:02:33.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:33.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:33.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:33 smithi086 ceph-mon[32965]: osdmap e316: 8 total, 8 up, 6 in 2024-03-26T04:02:33.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:33.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:33.969 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:33.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:34.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.094 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:34.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:34.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:34.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:34.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:34.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.352 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:34.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:34.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:34.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:34.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:34.567 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:34.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:34 smithi086 ceph-mon[32965]: pgmap v636: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 4652/145539 objects misplaced (3.196%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:02:34.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:34.609 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:34.609 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:34.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:34.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:34.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:34.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:34.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:34.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:34.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:34 smithi006 ceph-mon[37501]: pgmap v636: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 4652/145539 objects misplaced (3.196%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:02:34.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:34 smithi006 ceph-mon[29446]: pgmap v636: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 2.9 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 0 B/s wr, 856 op/s; 4652/145539 objects misplaced (3.196%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:02:34.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:35.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:35.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:35.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:35.170 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:35.245 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:34 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:35.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:35.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T04:02:35.290 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:02:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:35.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:35.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:02:35.378 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:02:35.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:35.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:35.435 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:02:35.479 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:02:35.534 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:35.550 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:02:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:35.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:35.587 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:02:35.589 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:35.632 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:35 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/978700928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:35.633 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:02:35.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:35 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/978700928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:35.638 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:02:35.639 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:02:35.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:35.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:35.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:35 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/978700928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:35.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:35.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:35.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:35.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:36.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:36.127 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:36.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:36.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:36.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:36.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:36.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:36.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:36.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:36.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:36.596 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:02:36.630 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:36.630 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:36 smithi086 ceph-mon[32965]: pgmap v637: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 856 B/s rd, 0 B/s wr, 895 op/s; 4422/139770 objects misplaced (3.164%); 19 MiB/s, 301 objects/s recovering 2024-03-26T04:02:36.631 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:02:36.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:36.669 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:36.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:36.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:36.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[37501]: pgmap v637: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 856 B/s rd, 0 B/s wr, 895 op/s; 4422/139770 objects misplaced (3.164%); 19 MiB/s, 301 objects/s recovering 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[29446]: pgmap v637: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 4 active+remapped+backfilling, 152 active+clean; 2.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 856 B/s rd, 0 B/s wr, 895 op/s; 4422/139770 objects misplaced (3.164%); 19 MiB/s, 301 objects/s recovering 2024-03-26T04:02:36.871 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:02:36.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:36.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:36.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:37.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:37.059 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:37.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:37.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:37.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:37.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:37.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:37.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:37.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:37.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:37.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:37.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:37.567 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:37.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:37.581 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:02:37.581 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:02:37.582 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:02:37.582 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:02:37.582 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:02:37.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:37.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:37.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:37 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:02:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:37 smithi086 ceph-mon[32965]: osdmap e317: 8 total, 8 up, 6 in 2024-03-26T04:02:37.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:37.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:02:37.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[37501]: osdmap e317: 8 total, 8 up, 6 in 2024-03-26T04:02:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:02:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/498421429' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:02:37.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:37 smithi006 ceph-mon[29446]: osdmap e317: 8 total, 8 up, 6 in 2024-03-26T04:02:37.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:37.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:37.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:38.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:38.103 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:38.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:38.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:38.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:38.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T04:02:38.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:38.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:38.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:38.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:38.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:38.586 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:38.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:38.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.631 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:38.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: pgmap v639: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 2768/135009 objects misplaced (2.050%); 17 MiB/s, 272 objects/s recovering 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: osdmap e318: 8 total, 8 up, 6 in 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: 1.1 scrub starts 2024-03-26T04:02:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[29446]: 1.1 scrub ok 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: pgmap v639: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 2768/135009 objects misplaced (2.050%); 17 MiB/s, 272 objects/s recovering 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: osdmap e318: 8 total, 8 up, 6 in 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: 1.1 scrub starts 2024-03-26T04:02:38.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:38 smithi006 ceph-mon[37501]: 1.1 scrub ok 2024-03-26T04:02:38.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:38.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:38.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:38.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:38.927 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:38.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:38.966 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: pgmap v639: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 2768/135009 objects misplaced (2.050%); 17 MiB/s, 272 objects/s recovering 2024-03-26T04:02:38.966 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: osdmap e318: 8 total, 8 up, 6 in 2024-03-26T04:02:38.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T04:02:38.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T04:02:38.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: 1.1 scrub starts 2024-03-26T04:02:38.967 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:38 smithi086 ceph-mon[32965]: 1.1 scrub ok 2024-03-26T04:02:39.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.058 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:39.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:39.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:39.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.367 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:39.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:39.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:39.493 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:39.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:02:39.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:39.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:39.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[37501]: 1.a scrub starts 2024-03-26T04:02:39.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[37501]: 1.a scrub ok 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[37501]: 9.5 scrub starts 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[29446]: 1.a scrub starts 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[29446]: 1.a scrub ok 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[29446]: 9.5 scrub starts 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:39.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:39.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:39.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:39.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:39.876 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:39.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:39.935 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:39 smithi086 ceph-mon[32965]: 1.a scrub starts 2024-03-26T04:02:39.935 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:39 smithi086 ceph-mon[32965]: 1.a scrub ok 2024-03-26T04:02:39.935 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:39 smithi086 ceph-mon[32965]: 9.5 scrub starts 2024-03-26T04:02:39.935 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:39.935 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:39.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:40.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:40.046 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:40.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:40.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:40.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:40.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:40.329 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:02:40.329 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:02:40.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:02:40.329 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-03-26T04:02:40.329 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:02:40.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:40.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:02:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:40.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:40.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.586 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:40.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:40.631 INFO:teuthology.orchestra.run.smithi006.stderr:osd set-full-ratio 0.95 2024-03-26T04:02:40.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:40.748 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:40.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:40.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:40.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:40.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[37501]: pgmap v641: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 679 op/s; 2768/131439 objects misplaced (2.106%); 15 MiB/s, 235 objects/s recovering 2024-03-26T04:02:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[37501]: 1.8 scrub starts 2024-03-26T04:02:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[37501]: 9.5 scrub ok 2024-03-26T04:02:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/761679635' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:40.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[29446]: pgmap v641: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 679 op/s; 2768/131439 objects misplaced (2.106%); 15 MiB/s, 235 objects/s recovering 2024-03-26T04:02:40.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[29446]: 1.8 scrub starts 2024-03-26T04:02:40.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[29446]: 9.5 scrub ok 2024-03-26T04:02:40.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/761679635' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:40.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:40 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:40.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:41.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:41.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:40 smithi086 ceph-mon[32965]: pgmap v641: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 156 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 679 op/s; 2768/131439 objects misplaced (2.106%); 15 MiB/s, 235 objects/s recovering 2024-03-26T04:02:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:40 smithi086 ceph-mon[32965]: 1.8 scrub starts 2024-03-26T04:02:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:40 smithi086 ceph-mon[32965]: 9.5 scrub ok 2024-03-26T04:02:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/761679635' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:40 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set-full-ratio", "ratio": 0.95}]: dispatch 2024-03-26T04:02:41.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:41.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:41.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T04:02:41.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.297 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:41.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:41.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:41.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:41.463 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:41.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:41.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:41.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:41.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:41.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:41.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:41.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:41.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: 1.8 scrub ok 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: 9.a scrub starts 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: osdmap e319: 8 total, 8 up, 6 in 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T04:02:41.851 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:41 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T04:02:41.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: 1.8 scrub ok 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: 9.a scrub starts 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: osdmap e319: 8 total, 8 up, 6 in 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: 1.8 scrub ok 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: 9.a scrub starts 2024-03-26T04:02:41.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: Health check cleared: OSD_OUT_OF_ORDER_FULL (was: full ratio(s) out of order) 2024-03-26T04:02:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set-full-ratio", "ratio": 0.95}]': finished 2024-03-26T04:02:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: osdmap e319: 8 total, 8 up, 6 in 2024-03-26T04:02:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T04:02:41.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:41 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T04:02:41.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:41.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:41.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:41.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:42.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:42.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:42.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:42.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:42.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:42.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:42.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:42.570 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:42.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:42.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[37501]: pgmap v643: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 975 op/s; 1740/122061 objects misplaced (1.426%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[37501]: 9.a scrub ok 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[29446]: pgmap v643: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 975 op/s; 1740/122061 objects misplaced (1.426%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[29446]: 9.a scrub ok 2024-03-26T04:02:42.692 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:42 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:02:42.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:42.727 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:42.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:42.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:42.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:42.846 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:02:42.886 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:02:42.923 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:42 smithi086 ceph-mon[32965]: pgmap v643: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 975 op/s; 1740/122061 objects misplaced (1.426%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:02:42.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:42 smithi086 ceph-mon[32965]: 9.a scrub ok 2024-03-26T04:02:42.924 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:42 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:02:42.930 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:02:42.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:42.976 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:02:42.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:42.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:43.015 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:02:43.056 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:02:43.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:43.097 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:02:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:43.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:43.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:02:43.154 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:02:43.200 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:43.200 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:43.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:43.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:43.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:43.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.403 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:43.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:43.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:43.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:43.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:43.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:43.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:43.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:43.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.816 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:43.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2698445762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T04:02:43.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2698445762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:43.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T04:02:43.917 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:43.956 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:43.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:43.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:44.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:43 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T04:02:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:43 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T04:02:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2698445762' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.099 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:44.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:44.148 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:44.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:44.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:44.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:44.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:44.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:44.528 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:44.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:44.584 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:44.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:44.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:44.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:44.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:44.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:44.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:44 smithi086 ceph-mon[32965]: pgmap v644: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 1740/122061 objects misplaced (1.426%); 7.2 MiB/s, 116 objects/s recovering 2024-03-26T04:02:44.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:44 smithi086 ceph-mon[32965]: osdmap e320: 8 total, 8 up, 6 in 2024-03-26T04:02:44.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:44 smithi006 ceph-mon[37501]: pgmap v644: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 1740/122061 objects misplaced (1.426%); 7.2 MiB/s, 116 objects/s recovering 2024-03-26T04:02:44.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:44 smithi006 ceph-mon[37501]: osdmap e320: 8 total, 8 up, 6 in 2024-03-26T04:02:44.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:44 smithi006 ceph-mon[29446]: pgmap v644: 158 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 156 active+clean; 2.5 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 0 B/s wr, 619 op/s; 1740/122061 objects misplaced (1.426%); 7.2 MiB/s, 116 objects/s recovering 2024-03-26T04:02:44.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:44 smithi006 ceph-mon[29446]: osdmap e320: 8 total, 8 up, 6 in 2024-03-26T04:02:44.890 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:44.930 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:44.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:44.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:45.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:45.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:45.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:45.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:45.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:45.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:45.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:45.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:45.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:45.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:45.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:45.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:45.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:45.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.592 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:45.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:45.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:45.730 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:45.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:45.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:45.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:45.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:45 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:02:45.873 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:45 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:02:45.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:45 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:02:45.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:45 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:02:45.938 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:45 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:02:45.939 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:45 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:02:45.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:45.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:45.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:46.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:46.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:46.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:46.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:46.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:46.263 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:46.263 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:46.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:46.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:46.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:46.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:02:46.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:46.533 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:46.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:46.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:46.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:46.669 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:02:46.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:46.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:46.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:46.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T04:02:46.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:46.942 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: pgmap v646: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 932 op/s; 811/112845 objects misplaced (0.719%); 12 MiB/s, 197 objects/s recovering 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: osdmap e321: 8 total, 8 up, 6 in 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T04:02:46.943 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:46 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T04:02:46.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:46.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:47.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T04:02:47.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: pgmap v646: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 932 op/s; 811/112845 objects misplaced (0.719%); 12 MiB/s, 197 objects/s recovering 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: osdmap e321: 8 total, 8 up, 6 in 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: pgmap v646: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 156 active+clean; 2.3 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 0 B/s wr, 932 op/s; 811/112845 objects misplaced (0.719%); 12 MiB/s, 197 objects/s recovering 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: osdmap e321: 8 total, 8 up, 6 in 2024-03-26T04:02:47.039 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:02:47.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T04:02:47.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:46 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T04:02:47.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:47.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:47.187 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7, 3] out_osds: [6, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:02:47.187 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:02:47.187 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 4, 7, 3] 2024-03-26T04:02:47.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd out 3 2024-03-26T04:02:47.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:47.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:47.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:47.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:47.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:47.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:47.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:47.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:47.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.654 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:47.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:47.664 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:47.689 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:02:47.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:47.800 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:02:47.800 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:02:47.800 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: Cluster is now healthy 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: osdmap e322: 8 total, 8 up, 6 in 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:02:47.801 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: Cluster is now healthy 2024-03-26T04:02:47.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:47.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: osdmap e322: 8 total, 8 up, 6 in 2024-03-26T04:02:47.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T04:02:47.802 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:47 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T04:02:47.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:47.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:47.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:47.944 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:47.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:48.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: Cluster is now healthy 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2171202227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:02:48.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: osdmap e322: 8 total, 8 up, 6 in 2024-03-26T04:02:48.076 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T04:02:48.076 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:47 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T04:02:48.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:48.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:48.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:48.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:48.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:48.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:48.650 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:48.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:48.789 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:48.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T04:02:48.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T04:02:48.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: pgmap v649: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 930 op/s; 811/105303 objects misplaced (0.770%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:48.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: 5.b deep-scrub starts 2024-03-26T04:02:48.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T04:02:48.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T04:02:48.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T04:02:48.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T04:02:48.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: pgmap v649: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 930 op/s; 811/105303 objects misplaced (0.770%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:48.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: 5.b deep-scrub starts 2024-03-26T04:02:48.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T04:02:48.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:48 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T04:02:48.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:48.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:48.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:48.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:49.014 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T04:02:49.015 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T04:02:49.015 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: pgmap v649: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.1 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 930 op/s; 811/105303 objects misplaced (0.770%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:49.015 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: 5.b deep-scrub starts 2024-03-26T04:02:49.015 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T04:02:49.015 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:48 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T04:02:49.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:49.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:49.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:49.071 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:49.071 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:49.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:49.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:49.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.368 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:49.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:49.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:49.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:49.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:02:49.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:49.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:49.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.754 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:49.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:49.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:49 smithi006 ceph-mon[29446]: 5.b deep-scrub ok 2024-03-26T04:02:49.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:49 smithi006 ceph-mon[29446]: 6.f deep-scrub starts 2024-03-26T04:02:49.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:49 smithi006 ceph-mon[37501]: 5.b deep-scrub ok 2024-03-26T04:02:49.893 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:49 smithi006 ceph-mon[37501]: 6.f deep-scrub starts 2024-03-26T04:02:49.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:49.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:49.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:49.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:49 smithi086 ceph-mon[32965]: 5.b deep-scrub ok 2024-03-26T04:02:49.912 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:49 smithi086 ceph-mon[32965]: 6.f deep-scrub starts 2024-03-26T04:02:49.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T04:02:50.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:50.051 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:50.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:50.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:50.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:50.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:50.259 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:50.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:50.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:50.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:50.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:50.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:50.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:50.594 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:50.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:50.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:50.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:50.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:50.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:50.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:50.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: 6.f deep-scrub ok 2024-03-26T04:02:50.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T04:02:50.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: pgmap v650: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 811/100863 objects misplaced (0.804%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: 6.f deep-scrub ok 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: pgmap v650: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 811/100863 objects misplaced (0.804%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T04:02:50.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:50 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T04:02:51.010 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:51.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: 6.f deep-scrub ok 2024-03-26T04:02:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T04:02:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T04:02:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: pgmap v650: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 2.0 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.18k op/s; 811/100863 objects misplaced (0.804%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:02:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T04:02:51.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:50 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T04:02:51.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:51.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:51.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:51.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:51.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:51.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:51.464 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.3. 2024-03-26T04:02:51.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:51.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:51.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:51.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.671 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:51.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:51.707 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:02:51.419+0000 7fd465f8b700 -1 mon.a@0(leader).osd e322 definitely_dead 0 2024-03-26T04:02:51.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:51.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:51.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:51.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:51.923 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:51.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:52.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:52.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:02:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:02:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1265152138' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T04:02:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T04:02:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:51 smithi086 ceph-mon[32965]: osdmap e323: 8 total, 8 up, 5 in 2024-03-26T04:02:52.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1265152138' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[29446]: osdmap e323: 8 total, 8 up, 5 in 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1265152138' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-26T04:02:52.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-26T04:02:52.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-26T04:02:52.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:51 smithi006 ceph-mon[37501]: osdmap e323: 8 total, 8 up, 5 in 2024-03-26T04:02:52.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:52.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:52.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:52.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:52.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:52.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:52.435 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:52.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:52.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:52.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:52.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:52.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:52.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:52.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:52.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:52.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:52.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:52.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:52.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:52.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:53.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:53.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:53.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: pgmap v652: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.37k op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T04:02:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T04:02:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T04:02:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T04:02:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T04:02:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T04:02:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:52 smithi086 ceph-mon[32965]: osdmap e324: 8 total, 8 up, 5 in 2024-03-26T04:02:53.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: pgmap v652: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.37k op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[29446]: osdmap e324: 8 total, 8 up, 5 in 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: pgmap v652: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.37k op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T04:02:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T04:02:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T04:02:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T04:02:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T04:02:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T04:02:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:52 smithi006 ceph-mon[37501]: osdmap e324: 8 total, 8 up, 5 in 2024-03-26T04:02:53.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.203 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:53.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:53.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:53.329 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:53.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:53.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:53.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:53.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:53.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:53.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:53.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:53.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:53.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:53.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:53.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:53.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:53.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:53 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T04:02:53.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:53 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T04:02:53.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:53 smithi086 ceph-mon[32965]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 322'14827 2024-03-26T04:02:53.990 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:53 smithi086 ceph-mon[32965]: osdmap e325: 8 total, 8 up, 5 in 2024-03-26T04:02:53.990 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:53 smithi086 ceph-mon[32965]: 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 322'14568 2024-03-26T04:02:54.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:54.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[37501]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 322'14827 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[37501]: osdmap e325: 8 total, 8 up, 5 in 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[37501]: 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 322'14568 2024-03-26T04:02:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T04:02:54.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T04:02:54.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[29446]: 9.8 starting backfill to osd.1 from (0'0,0'0] MAX to 322'14827 2024-03-26T04:02:54.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[29446]: osdmap e325: 8 total, 8 up, 5 in 2024-03-26T04:02:54.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:53 smithi006 ceph-mon[29446]: 9.9 starting backfill to osd.4 from (0'0,0'0] MAX to 322'14568 2024-03-26T04:02:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:54.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:54.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.273 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:54.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:54.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:54.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:54.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:54.444 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:02:54.444 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:02:54.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:54.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:54.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:54.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:02:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:02:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:54.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:54.799 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:54.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:54.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:54.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:54.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:54.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:55.018 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 6.b deep-scrub starts 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 6.b deep-scrub ok 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 322'14943 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 322'14736 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.a starting backfill to osd.2 from (0'0,0'0] MAX to 322'14672 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.3 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14410 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 322'15183 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 322'15064 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.f starting backfill to osd.4 from (0'0,0'0] MAX to 322'14968 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14659 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: pgmap v655: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: osdmap e326: 8 total, 8 up, 5 in 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:55.019 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:55.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:55.084 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:55.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 6.b deep-scrub starts 2024-03-26T04:02:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 6.b deep-scrub ok 2024-03-26T04:02:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 322'14943 2024-03-26T04:02:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 322'14736 2024-03-26T04:02:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.a starting backfill to osd.2 from (0'0,0'0] MAX to 322'14672 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.3 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14410 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 322'15183 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 322'15064 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.f starting backfill to osd.4 from (0'0,0'0] MAX to 322'14968 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14659 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: pgmap v655: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: osdmap e326: 8 total, 8 up, 5 in 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 6.b deep-scrub starts 2024-03-26T04:02:55.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 6.b deep-scrub ok 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.c starting backfill to osd.0 from (0'0,0'0] MAX to 322'14943 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.2 starting backfill to osd.7 from (0'0,0'0] MAX to 322'14736 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.a starting backfill to osd.2 from (0'0,0'0] MAX to 322'14672 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.3 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14410 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.7 starting backfill to osd.1 from (0'0,0'0] MAX to 322'15183 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.1 starting backfill to osd.2 from (0'0,0'0] MAX to 322'15064 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.f starting backfill to osd.4 from (0'0,0'0] MAX to 322'14968 2024-03-26T04:02:55.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: 9.4 starting backfill to osd.0 from (0'0,0'0] MAX to 322'14659 2024-03-26T04:02:55.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: pgmap v655: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 1.8 GiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 954 op/s; 38/88131 objects misplaced (0.043%); 7.4 MiB/s, 118 objects/s recovering 2024-03-26T04:02:55.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: osdmap e326: 8 total, 8 up, 5 in 2024-03-26T04:02:55.145 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:02:55.146 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:02:55.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:55.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T04:02:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.214 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:55.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:55.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:55.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:02:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:02:55.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:55.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:55.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:55.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:55.479 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:55.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:55.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:55.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:02:55.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:55.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:55.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:55.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:55.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:55.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:55.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:56.007 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:56.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:56.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:56.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:56.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:56.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:56.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:56.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:56.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:56.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:56.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:56.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:56.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:56 smithi086 ceph-mon[32965]: osdmap e327: 8 total, 8 up, 5 in 2024-03-26T04:02:56.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:56 smithi086 ceph-mon[32965]: pgmap v658: 158 pgs: 1 active+remapped+backfill_toofull, 4 peering, 4 active+remapped+backfilling, 149 active+clean; 1.7 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 290 op/s; 6502/84756 objects misplaced (7.671%); 4.8 MiB/s, 93 objects/s recovering 2024-03-26T04:02:56.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:56.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:56.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.836 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:56.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[37501]: osdmap e327: 8 total, 8 up, 5 in 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[37501]: pgmap v658: 158 pgs: 1 active+remapped+backfill_toofull, 4 peering, 4 active+remapped+backfilling, 149 active+clean; 1.7 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 290 op/s; 6502/84756 objects misplaced (7.671%); 4.8 MiB/s, 93 objects/s recovering 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[29446]: osdmap e327: 8 total, 8 up, 5 in 2024-03-26T04:02:56.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:56 smithi006 ceph-mon[29446]: pgmap v658: 158 pgs: 1 active+remapped+backfill_toofull, 4 peering, 4 active+remapped+backfilling, 149 active+clean; 1.7 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 290 op/s; 6502/84756 objects misplaced (7.671%); 4.8 MiB/s, 93 objects/s recovering 2024-03-26T04:02:56.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:56.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:56.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:57.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:57.104 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:57.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:57.236 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:57.334 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:02:57.334 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:02:57.334 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:02:57.334 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:02:57.334 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:02:57.338 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:57.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:57.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.505 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:57 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/954841557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:57 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:02:57.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:57 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:02:57.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:02:57.650 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:02:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:02:57.680 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:02:57.681 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:02:57.682 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:02:57.682 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:02:57.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:57.781 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:57.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/954841557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:02:57.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/954841557' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:02:57.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:02:57.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:57 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:02:57.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:57.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:58.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:58.051 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:58.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:58.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T04:02:58.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:58.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:58.292 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:58.332 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:58.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:58.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:58.438 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:02:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:58.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:58.578 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:58.616 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:58.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:58.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:58.731 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:58 smithi086 ceph-mon[32965]: osdmap e328: 8 total, 8 up, 5 in 2024-03-26T04:02:58.731 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:02:58 smithi086 ceph-mon[32965]: pgmap v660: 158 pgs: 4 active+remapped+backfill_toofull, 4 peering, 6 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 632 op/s; 15903/80589 objects misplaced (19.733%); 12 MiB/s, 220 objects/s recovering 2024-03-26T04:02:58.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:58.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:02:58.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:58.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:02:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:58 smithi006 ceph-mon[37501]: osdmap e328: 8 total, 8 up, 5 in 2024-03-26T04:02:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:02:58 smithi006 ceph-mon[37501]: pgmap v660: 158 pgs: 4 active+remapped+backfill_toofull, 4 peering, 6 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 632 op/s; 15903/80589 objects misplaced (19.733%); 12 MiB/s, 220 objects/s recovering 2024-03-26T04:02:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:58 smithi006 ceph-mon[29446]: osdmap e328: 8 total, 8 up, 5 in 2024-03-26T04:02:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:02:58 smithi006 ceph-mon[29446]: pgmap v660: 158 pgs: 4 active+remapped+backfill_toofull, 4 peering, 6 active+remapped+backfilling, 144 active+clean; 1.6 GiB data, 6.5 GiB used, 441 GiB / 447 GiB avail; 7.0 KiB/s rd, 55 KiB/s wr, 632 op/s; 15903/80589 objects misplaced (19.733%); 12 MiB/s, 220 objects/s recovering 2024-03-26T04:02:58.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:59.018 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:59.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:59.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:59.164 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:59.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:02:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:02:59.393 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:02:59.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:02:59.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:02:59.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:02:59.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:02:59.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:02:59.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:02:59.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:02:59.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:59.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:02:59.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:02:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:02:59.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:02:59.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:00.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:00.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:00.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:00.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215934 8.9 8:9b840472:::benchmark_data_smithi006_103169_object86003:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 311.01227902199997, 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9143611719999996, 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.354 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215934 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337353+0000", 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.313e-06 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337361+0000", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6170000000000001e-06 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337370+0000", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9700000000000005e-06 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337379+0000", 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7999999999999993e-06 2024-03-26T04:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337438+0000", 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8943000000000001e-05 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337451+0000", 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2081e-05 2024-03-26T04:03:00.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.758498+0000", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4210475499999999 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.761407+0000", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029086649999999999 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.854995+0000", 2024-03-26T04:03:00.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.093588355999999998 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.857945+0000", 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002950275 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.192696+0000", 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.33475033199999998 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.195884+0000", 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003187835 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.201543+0000", 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056593379999999999 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.223734+0000", 2024-03-26T04:03:00.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022191072999999999 2024-03-26T04:03:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.240769+0000", 2024-03-26T04:03:00.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017035136999999999 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.241861+0000", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001091987 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.244275+0000", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002413512 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.245980+0000", 2024-03-26T04:03:00.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001704999 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.251713+0000", 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057333890000000002 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215951 8.9 8:9b9ea6e8:::benchmark_data_smithi006_103169_object86020:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 311.01217043000003, 2024-03-26T04:03:00.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9678481669999996, 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215951 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337461+0000", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2099999999999996e-07 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337464+0000", 2024-03-26T04:03:00.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0759999999999999e-06 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337470+0000", 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9819999999999996e-06 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337476+0000", 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2060000000000004e-06 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337537+0000", 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0900999999999998e-05 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:03:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337548+0000", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0538e-05 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.768786+0000", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4312377959999996 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.771602+0000", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028160630000000002 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.869578+0000", 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097976928000000005 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.872463+0000", 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028841299999999999 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.262539+0000", 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.39007594200000001 2024-03-26T04:03:00.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.265078+0000", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025392700000000002 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.269554+0000", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044757060000000003 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.288933+0000", 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019379251 2024-03-26T04:03:00.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.289948+0000", 2024-03-26T04:03:00.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010155539999999999 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.290994+0000", 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010457400000000001 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.298118+0000", 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0071244020000000002 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.299493+0000", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013744079999999999 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.305309+0000", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058156529999999996 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215967 8.9 8:95a535a3:::benchmark_data_smithi006_103169_object86036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:03:00.375 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.86819090900002, 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.625424701, 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215967 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481442+0000", 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.747e-06 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481455+0000", 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3691e-05 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481468+0000", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2838e-05 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481478+0000", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6439999999999995e-06 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.771896+0000", 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.290418598 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.774772+0000", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002875739 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.791709+0000", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016937124000000001 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.794579+0000", 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028692420000000001 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.065543+0000", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.27096404400000002 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.067925+0000", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023821580000000001 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.072181+0000", 2024-03-26T04:03:00.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042559770000000002 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.094623+0000", 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022442397999999999 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.095642+0000", 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010186749999999999 2024-03-26T04:03:00.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.097572+0000", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001930576 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.100437+0000", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028645430000000002 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.101719+0000", 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012818549999999999 2024-03-26T04:03:00.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.106865+0000", 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051458520000000002 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215972 8.9 8:96cf40d7:::benchmark_data_smithi006_103169_object86041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.858749639, 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6251629169999999, 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215972 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:03:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490882+0000", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5900000000000005e-07 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490886+0000", 2024-03-26T04:03:00.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1099999999999996e-06 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490896+0000", 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4099999999999997e-06 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490903+0000", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8150000000000006e-06 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.775083+0000", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.284179795 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.778005+0000", 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029216049999999999 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.805286+0000", 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027281401 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.808134+0000", 2024-03-26T04:03:00.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028481230000000001 2024-03-26T04:03:00.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.083016+0000", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.27488142500000001 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.085364+0000", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023482640000000001 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.089565+0000", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042006719999999999 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.099908+0000", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010342953 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.107116+0000", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0072084599999999999 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.108024+0000", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090759 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.109164+0000", 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00114007 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.110733+0000", 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015694420000000001 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.116044+0000", 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053108230000000001 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216001 8.9 8:9303bcb3:::benchmark_data_smithi006_103169_object86070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.78904137500001, 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8087078779999999, 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216001 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.399 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560590+0000", 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0359999999999999e-06 2024-03-26T04:03:00.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560595+0000", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4379999999999997e-06 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560604+0000", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1959999999999997e-06 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560609+0000", 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.31e-06 2024-03-26T04:03:00.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.778310+0000", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2177009320000001 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.781165+0000", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028549529999999999 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.883967+0000", 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10280199399999999 2024-03-26T04:03:00.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.886904+0000", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002937042 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.312272+0000", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.425368146 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.314685+0000", 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024124049999999998 2024-03-26T04:03:00.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.318932+0000", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042475569999999999 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.341790+0000", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022857747000000001 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.358289+0000", 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016499092 2024-03-26T04:03:00.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.360596+0000", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023069330000000002 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.361690+0000", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010932889999999999 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.363266+0000", 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015763660000000001 2024-03-26T04:03:00.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.369297+0000", 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060314419999999997 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.406 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216007 8.9 8:9fef8228:::benchmark_data_smithi006_103169_object86076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.78016136899998, 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5957876789999998, 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216007 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569470+0000", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.02e-06 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569476+0000", 2024-03-26T04:03:00.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2979999999999999e-06 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569488+0000", 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1822e-05 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569496+0000", 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6859999999999995e-06 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.809721+0000", 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2402245949999999 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.812495+0000", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027742750000000001 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.826749+0000", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014254195000000001 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.829604+0000", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002855064 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.123487+0000", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.29388293300000001 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.125888+0000", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002401065 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.130205+0000", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043169109999999997 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.155137+0000", 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024931531 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.155665+0000", 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00052799200000000002 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.156456+0000", 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00079165599999999996 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.157703+0000", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012468920000000001 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.159481+0000", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001777947 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.165257+0000", 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005775797 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.415 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216021 8.9 8:9cfec2e3:::benchmark_data_smithi006_103169_object86090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.76130706800001, 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5884961029999998, 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.416 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216021 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:03:00.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588325+0000", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0640000000000001e-06 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588329+0000", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4660000000000001e-06 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588340+0000", 2024-03-26T04:03:00.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0958000000000001e-05 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588349+0000", 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6109999999999994e-06 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.813568+0000", 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.225218693 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.816433+0000", 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002865795 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.840953+0000", 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024520088999999998 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.843796+0000", 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028421879999999998 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.142384+0000", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.298587936 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.144752+0000", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023685759999999998 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.149021+0000", 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042687619999999997 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.165580+0000", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016558661999999998 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.166607+0000", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010269770000000001 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.167951+0000", 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013445040000000001 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.169092+0000", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011412499999999999 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.170635+0000", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001542615 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.176820+0000", 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061849569999999996 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216026 8.9 8:9d812808:::benchmark_data_smithi006_103169_object86095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.75492754800001, 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8086321540000001, 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216026 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.424 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594704+0000", 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.085e-06 2024-03-26T04:03:00.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594709+0000", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3279999999999999e-06 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594716+0000", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7929999999999998e-06 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594723+0000", 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9759999999999998e-06 2024-03-26T04:03:00.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.888541+0000", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.293817561 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.891566+0000", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030245459999999999 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.907254+0000", 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015688698000000001 2024-03-26T04:03:00.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.910169+0000", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029149839999999998 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.376477+0000", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.46630807899999999 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.379058+0000", 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025804579999999999 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.383452+0000", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043940469999999999 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.393537+0000", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010085213000000001 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.394133+0000", 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059573000000000004 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.394972+0000", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00083899799999999998 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.396250+0000", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001277893 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.397862+0000", 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001612589 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.403335+0000", 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054731759999999997 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216052 8.9 8:9883e2bd:::benchmark_data_smithi006_103169_object86121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.67304976000003, 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7761615759999998, 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.431 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216052 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676582+0000", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6299999999999993e-07 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676585+0000", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4410000000000002e-06 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676595+0000", 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2390000000000008e-06 2024-03-26T04:03:00.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676602+0000", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9020000000000002e-06 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.892626+0000", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2160241689999998 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.895561+0000", 2024-03-26T04:03:00.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029351799999999999 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.921143+0000", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025582229000000001 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.924014+0000", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028710910000000001 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.411625+0000", 2024-03-26T04:03:00.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.48761071099999997 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.414079+0000", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002453616 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.418292+0000", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042136939999999996 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.441672+0000", 2024-03-26T04:03:00.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0233796 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.442545+0000", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00087358399999999995 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.443631+0000", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001085997 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.444972+0000", 2024-03-26T04:03:00.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001340911 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.446539+0000", 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001566743 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.452743+0000", 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0062035060000000001 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216058 8.9 8:907609bc:::benchmark_data_smithi006_103169_object86127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.65503113300002, 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7699544309999999, 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216058 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694601+0000", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2099999999999995e-07 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694605+0000", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.267e-06 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694615+0000", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0131e-05 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694622+0000", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6200000000000001e-06 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.930307+0000", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2356848830000002 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.933185+0000", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002878219 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.951929+0000", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018744690000000001 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.954809+0000", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002879716 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.430784+0000", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.47597526899999998 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.433190+0000", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024058730000000002 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.437392+0000", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042020180000000001 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.444178+0000", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006786177 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.453940+0000", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0097615389999999996 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.455080+0000", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011395349999999999 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.456458+0000", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001378867 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.458538+0000", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002080087 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.464554+0000", 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060156189999999998 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216060 8.9 8:911f1304:::benchmark_data_smithi006_103169_object86129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 308.97381353100002, 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.139370537, 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.446 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216060 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375818+0000", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0500000000000003e-07 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375821+0000", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1700000000000001e-06 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375827+0000", 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2890000000000003e-06 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375832+0000", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4750000000000004e-06 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.934247+0000", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5584147580000001 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.937162+0000", 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029153260000000002 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.967070+0000", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029908481000000001 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.970070+0000", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00300008 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.472966+0000", 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.50289568900000003 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.475312+0000", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002345499 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.479421+0000", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00410955 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.504198+0000", 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024776461 2024-03-26T04:03:00.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.505405+0000", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012075269999999999 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.506060+0000", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00065503600000000001 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.507216+0000", 2024-03-26T04:03:00.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011562530000000001 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.509194+0000", 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019774689999999999 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.515188+0000", 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059938689999999998 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.453 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216086 8.9 8:95431b70:::benchmark_data_smithi006_103169_object86155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 308.94058198699997, 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1168013650000002, 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216086 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409050+0000", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4600000000000003e-07 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.455 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409054+0000", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5990000000000004e-06 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409065+0000", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0984e-05 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409073+0000", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7829999999999999e-06 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.456 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.937475+0000", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5284018110000002 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.940431+0000", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029556610000000001 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.982173+0000", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041742082 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.985166+0000", 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002992818 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.490278+0000", 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.50511281900000005 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.493071+0000", 2024-03-26T04:03:00.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027928800000000002 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.498275+0000", 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052038709999999997 2024-03-26T04:03:00.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.506403+0000", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081282049999999995 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.516183+0000", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0097795950000000003 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.517490+0000", 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013068489999999999 2024-03-26T04:03:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.518574+0000", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010838829999999999 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.520234+0000", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016606259999999999 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.525850+0000", 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056159529999999999 2024-03-26T04:03:00.462 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216096 8.9 8:96b54a59:::benchmark_data_smithi006_103169_object86165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 308.89565791400003, 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1427821639999998, 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.463 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216096 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453974+0000", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1800000000000005e-07 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453977+0000", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.422e-06 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453985+0000", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1950000000000003e-06 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453990+0000", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7589999999999997e-06 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.985562+0000", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5315718939999998 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.988447+0000", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002884583 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.466 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.004127+0000", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015680555999999998 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.006941+0000", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002814212 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.541439+0000", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.534497577 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.544323+0000", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028842519999999999 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.548578+0000", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042547619999999996 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.568435+0000", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019856780000000001 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.585897+0000", 2024-03-26T04:03:00.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017462485999999999 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.587101+0000", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012040950000000001 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.589232+0000", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002131029 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.590881+0000", 2024-03-26T04:03:00.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001648341 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.596755+0000", 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058744030000000003 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216125 8.9 8:9586e5b7:::benchmark_data_smithi006_103169_object86194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 307.92365176499999, 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2384351320000002, 2024-03-26T04:03:00.470 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216125 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425980+0000", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2399999999999996e-07 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425984+0000", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8089999999999999e-06 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425992+0000", 2024-03-26T04:03:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3410000000000003e-06 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425997+0000", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2959999999999998e-06 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.989600+0000", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5636028149999999 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.992579+0000", 2024-03-26T04:03:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029790649999999999 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.019346+0000", 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026766503000000001 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.022354+0000", 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030078790000000002 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.611057+0000", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.58870329300000002 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.613926+0000", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028689359999999999 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.618353+0000", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044269069999999999 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.637880+0000", 2024-03-26T04:03:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019527013999999999 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.653783+0000", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015903098000000001 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.654853+0000", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010697739999999999 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.656744+0000", 2024-03-26T04:03:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001891055 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.658330+0000", 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001585927 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.664414+0000", 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006084496 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216143 8.9 8:9157ba64:::benchmark_data_smithi006_103169_object86212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 307.87963702399998, 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2590048, 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216143 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469995+0000", 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1209999999999999e-06 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469999+0000", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9949999999999999e-06 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.470011+0000", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1939999999999999e-05 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.470018+0000", 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0940000000000002e-06 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.022753+0000", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.55273555 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.025701+0000", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00294751 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.041508+0000", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015806875000000001 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.044487+0000", 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029794130000000002 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.678561+0000", 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.63407396900000002 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.681234+0000", 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026731569999999998 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.685340+0000", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041054120000000001 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.714580+0000", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029240051999999999 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.719298+0000", 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047185259999999998 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.720597+0000", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298565 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.722146+0000", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015487260000000001 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.723755+0000", 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016092509999999999 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.728999+0000", 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052436440000000004 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216152 8.9 8:9b1864ad:::benchmark_data_smithi006_103169_object86221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 307.79868495300002, 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2448713520000001, 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216152 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:03:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550947+0000", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.031e-06 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550952+0000", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8289999999999998e-06 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550962+0000", 2024-03-26T04:03:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550968+0000", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5930000000000002e-06 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.026438+0000", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.475469897 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.029344+0000", 2024-03-26T04:03:00.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029057240000000002 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.055621+0000", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026276711000000001 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.058605+0000", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029844210000000001 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.743804+0000", 2024-03-26T04:03:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.68519932800000005 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.746158+0000", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023533019999999998 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.750518+0000", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043598029999999998 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.781267+0000", 2024-03-26T04:03:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030749739000000002 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.785314+0000", 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0040463369999999997 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.786421+0000", 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001107547 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.788464+0000", 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020427940000000001 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.790342+0000", 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001877952 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.795817+0000", 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054752560000000004 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216241 8.9 8:9d5583f9:::benchmark_data_smithi006_103169_object86310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 305.93917342200001, 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1741758330000001, 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216241 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410458+0000", 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1300000000000001e-07 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410460+0000", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.322e-06 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410465+0000", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9049999999999996e-06 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410469+0000", 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8210000000000003e-06 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.464870+0000", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054400873000000002 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.466194+0000", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001324684 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.015018+0000", 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.54882358899999995 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.017951+0000", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029328179999999998 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.545176+0000", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.52722475800000002 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.547506+0000", 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023308679999999998 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.551679+0000", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004172327 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.574926+0000", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023247431999999998 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.575425+0000", 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049900400000000005 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.576344+0000", 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00091846699999999996 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.577460+0000", 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001116314 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.578946+0000", 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014856380000000001 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.584633+0000", 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056876000000000001 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216821 8.9 8:9c89ad88:::benchmark_data_smithi006_103169_object86890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 304.18891786099999, 2024-03-26T04:03:00.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1976572089999999, 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216821 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:03:00.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160714+0000", 2024-03-26T04:03:00.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4200000000000005e-07 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160717+0000", 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3670000000000001e-06 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160725+0000", 2024-03-26T04:03:00.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8220000000000007e-06 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160730+0000", 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3730000000000001e-06 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.195041+0000", 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034310933000000002 2024-03-26T04:03:00.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.196339+0000", 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00129762 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.745873+0000", 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.54953434800000001 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.748690+0000", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028172319999999998 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.316439+0000", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.56774893599999998 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.318912+0000", 2024-03-26T04:03:00.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002472798 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.323900+0000", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049881480000000004 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.348124+0000", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024223176999999999 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.349422+0000", 2024-03-26T04:03:00.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012988159999999999 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.349961+0000", 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000539008 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.350738+0000", 2024-03-26T04:03:00.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00077698199999999995 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.352285+0000", 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001546541 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.358370+0000", 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060852659999999998 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.511 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:218800 8.9 8:9f4e53b9:::benchmark_data_smithi006_103169_object88869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-03-26T04:03:00.511 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:03:00.511 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.15709232099999, 2024-03-26T04:03:00.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1631977849999999, 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 218800 2024-03-26T04:03:00.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192540+0000", 2024-03-26T04:03:00.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.062e-06 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192544+0000", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5340000000000001e-06 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192553+0000", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0529999999999996e-06 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192559+0000", 2024-03-26T04:03:00.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1020000000000002e-06 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.237081+0000", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044522220000000001 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.238406+0000", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013245279999999999 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.736273+0000", 2024-03-26T04:03:00.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.49786729099999999 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.739003+0000", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027294670000000002 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.322123+0000", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.58312056099999998 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.324262+0000", 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021390210000000001 2024-03-26T04:03:00.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.328111+0000", 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003849037 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.345681+0000", 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017569544999999999 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.346173+0000", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049171700000000002 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.347475+0000", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001302115 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.348740+0000", 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012655310000000001 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.350131+0000", 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013907139999999999 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.355736+0000", 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056052869999999996 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:219696 8.9 8:9e2d1e9d:::benchmark_data_smithi006_103169_object89765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-26T04:03:00.519 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 294.925416783, 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.143167109, 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 219696 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:03:00.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424215+0000", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5899999999999995e-07 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424218+0000", 2024-03-26T04:03:00.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2030000000000001e-06 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424226+0000", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7889999999999995e-06 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424231+0000", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3229999999999997e-06 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.475754+0000", 2024-03-26T04:03:00.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051523173999999998 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.476935+0000", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011807969999999999 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.992045+0000", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.51511001700000003 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.994655+0000", 2024-03-26T04:03:00.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002610064 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.537108+0000", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.542452657 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.539400+0000", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022919250000000002 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.543629+0000", 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042293779999999998 2024-03-26T04:03:00.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.551304+0000", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0076750179999999996 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.559469+0000", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008164391 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.560475+0000", 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001006175 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.525 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.561404+0000", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000928833 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.562774+0000", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001370263 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.567381+0000", 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046072429999999996 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.526 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:03:00.527 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:03:00.527 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:03:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.527 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:00.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:00.628 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:03:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:00.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:03:00.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:00.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:00.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:00 smithi086 ceph-mon[32965]: pgmap v661: 158 pgs: 4 active+remapped+backfill_toofull, 2 peering, 6 active+remapped+backfilling, 146 active+clean; 1.5 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 682 op/s; 13140/75969 objects misplaced (17.297%); 29 MiB/s, 477 objects/s recovering 2024-03-26T04:03:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:00 smithi006 ceph-mon[37501]: pgmap v661: 158 pgs: 4 active+remapped+backfill_toofull, 2 peering, 6 active+remapped+backfilling, 146 active+clean; 1.5 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 682 op/s; 13140/75969 objects misplaced (17.297%); 29 MiB/s, 477 objects/s recovering 2024-03-26T04:03:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:00 smithi006 ceph-mon[29446]: pgmap v661: 158 pgs: 4 active+remapped+backfill_toofull, 2 peering, 6 active+remapped+backfilling, 146 active+clean; 1.5 GiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 5.5 KiB/s rd, 37 KiB/s wr, 682 op/s; 13140/75969 objects misplaced (17.297%); 29 MiB/s, 477 objects/s recovering 2024-03-26T04:03:00.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:00.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:00.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:01.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:01.082 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:01.083 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:01.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:01.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:03:01.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:03:01.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.385 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:01.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:01.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:01.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T04:03:01.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:01.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:01.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:01.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:01.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:01.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:01.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:01.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:01.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:02.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:02.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:02.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:02.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:02.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:02.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:02.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.324 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:02.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:02.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.477 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:02.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:02.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:02.728 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:03:02.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:02.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:02 smithi086 ceph-mon[32965]: pgmap v662: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 986 op/s; 10351/67398 objects misplaced (15.358%); 35 MiB/s, 577 objects/s recovering 2024-03-26T04:03:02.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:03:02.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:02.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:02 smithi006 ceph-mon[37501]: pgmap v662: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 986 op/s; 10351/67398 objects misplaced (15.358%); 35 MiB/s, 577 objects/s recovering 2024-03-26T04:03:02.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:03:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:02 smithi006 ceph-mon[29446]: pgmap v662: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 4.7 KiB/s rd, 31 KiB/s wr, 986 op/s; 10351/67398 objects misplaced (15.358%); 35 MiB/s, 577 objects/s recovering 2024-03-26T04:03:02.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:03:02.902 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:02.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:02.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:03.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:03.036 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:03.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:03.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:03:03.171 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:03.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:03:03.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:03.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:03.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:03.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.429 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:03.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:03.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:03.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:03.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:03.691 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:03.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:03.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:03.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:03.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:03 smithi086 ceph-mon[32965]: osdmap e329: 8 total, 8 up, 5 in 2024-03-26T04:03:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:03 smithi006 ceph-mon[37501]: osdmap e329: 8 total, 8 up, 5 in 2024-03-26T04:03:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:03 smithi006 ceph-mon[29446]: osdmap e329: 8 total, 8 up, 5 in 2024-03-26T04:03:03.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:03.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:03.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:03.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:04.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.101 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:04.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:04.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:03:04.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:04.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:03:04.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:04.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:04.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:04.508 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:04.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:04.575 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:04.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:04.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:04.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.781 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:04.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: pgmap v664: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 127 B/s rd, 0 B/s wr, 723 op/s; 10351/67398 objects misplaced (15.358%); 28 MiB/s, 460 objects/s recovering 2024-03-26T04:03:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: osdmap e330: 8 total, 8 up, 5 in 2024-03-26T04:03:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T04:03:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T04:03:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:04.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:04 smithi086 ceph-mon[32965]: osdmap e331: 8 total, 8 up, 5 in 2024-03-26T04:03:04.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:04.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: pgmap v664: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 127 B/s rd, 0 B/s wr, 723 op/s; 10351/67398 objects misplaced (15.358%); 28 MiB/s, 460 objects/s recovering 2024-03-26T04:03:04.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: osdmap e330: 8 total, 8 up, 5 in 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[37501]: osdmap e331: 8 total, 8 up, 5 in 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: pgmap v664: 158 pgs: 4 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 1.4 GiB data, 6.0 GiB used, 441 GiB / 447 GiB avail; 127 B/s rd, 0 B/s wr, 723 op/s; 10351/67398 objects misplaced (15.358%); 28 MiB/s, 460 objects/s recovering 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: osdmap e330: 8 total, 8 up, 5 in 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:04 smithi006 ceph-mon[29446]: osdmap e331: 8 total, 8 up, 5 in 2024-03-26T04:03:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:04.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:04.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:05.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:05.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:05.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:05.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:05.168 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:05.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:05.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:05.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.301 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:05.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:05.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:05.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:05.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:05.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:05.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:05.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:05.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:05.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:05.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:05.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:05.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:05.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:05.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:06.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.067 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:06.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:06.073 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7] out_osds: [6, 5, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:03:06.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:03:06.074 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 6, 7, 5] 2024-03-26T04:03:06.074 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:03:06.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:03:06.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:06.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:06.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:06.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:06.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:06.389 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:06 smithi006 systemd[1]: Stopping Ceph osd.2 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:03:06.419 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:06.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:06.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:06.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:06.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:06.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:06.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.698 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:06.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:06.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:06.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:06.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:06 smithi086 ceph-mon[32965]: osdmap e332: 8 total, 8 up, 5 in 2024-03-26T04:03:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:06 smithi086 ceph-mon[32965]: pgmap v668: 158 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 1.2 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 7903/61971 objects misplaced (12.753%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:03:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:06 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:06 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[37501]: osdmap e332: 8 total, 8 up, 5 in 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[37501]: pgmap v668: 158 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 1.2 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 7903/61971 objects misplaced (12.753%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[29446]: osdmap e332: 8 total, 8 up, 5 in 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[29446]: pgmap v668: 158 pgs: 4 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 1.2 GiB data, 5.8 GiB used, 441 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 453 op/s; 7903/61971 objects misplaced (12.753%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:06 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:06.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:06.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:06.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:07.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:07.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:07.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:07.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:07.200 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:07.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:07.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:07.427 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:07.452 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:07.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:07.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:07.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:07.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:07.724 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:07.734 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:03:07.734 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:03:07.734 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T04:03:07.734 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[37501]: osdmap e333: 8 total, 8 up, 5 in 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T04:03:07.735 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:07 smithi006 ceph-mon[29446]: osdmap e333: 8 total, 8 up, 5 in 2024-03-26T04:03:07.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:07 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:03:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:07 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:03:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:07 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T04:03:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:07 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T04:03:07.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:07 smithi086 ceph-mon[32965]: osdmap e333: 8 total, 8 up, 5 in 2024-03-26T04:03:07.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:07.953 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:07.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:07.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:07.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:07.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:08.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:08.113 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:08.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:08.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:08.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:08.372 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:08.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:08.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:08.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:08.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T04:03:08.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: osdmap e334: 8 total, 8 up, 5 in 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: pgmap v671: 158 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 1.1 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 963 op/s; 4741/56121 objects misplaced (8.448%); 67 MiB/s, 1.06k objects/s recovering 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 5.14 deep-scrub starts 2024-03-26T04:03:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:08 smithi086 ceph-mon[32965]: 5.14 deep-scrub ok 2024-03-26T04:03:08.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:08.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:08.735 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: osdmap e334: 8 total, 8 up, 5 in 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: pgmap v671: 158 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 1.1 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 963 op/s; 4741/56121 objects misplaced (8.448%); 67 MiB/s, 1.06k objects/s recovering 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 5.14 deep-scrub starts 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[37501]: 5.14 deep-scrub ok 2024-03-26T04:03:08.736 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: osdmap e334: 8 total, 8 up, 5 in 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: pgmap v671: 158 pgs: 4 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 1.1 GiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 963 op/s; 4741/56121 objects misplaced (8.448%); 67 MiB/s, 1.06k objects/s recovering 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 5.14 deep-scrub starts 2024-03-26T04:03:08.737 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:08 smithi006 ceph-mon[29446]: 5.14 deep-scrub ok 2024-03-26T04:03:08.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:08.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:08.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:08.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:08.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:08.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:09.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:09.073 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:09.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:09.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:09.214 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:09.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:09.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:09.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:09.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:09.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:09.487 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:09.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:09.502 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:03:09.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:09.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.620 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:09.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:09.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:03:09.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:03:09.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:03:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:03:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:03:09.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:03:09.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.759 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:09.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:03:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:03:09.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:03:09.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:09.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:09.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:09.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:10.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.059 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:10.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:10.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:10.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:10.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.318 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:10.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:10.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:10.577 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:10.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:10.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:10.804 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:03:10.804 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: 9.8 scrub starts 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: pgmap v672: 158 pgs: 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 152 active+clean; 1.1 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 970 op/s; 4469/52668 objects misplaced (8.485%); 52 MiB/s, 839 objects/s recovering 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: osdmap e335: 8 total, 8 up, 5 in 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:03:10.805 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: 9.8 scrub starts 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: pgmap v672: 158 pgs: 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 152 active+clean; 1.1 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 970 op/s; 4469/52668 objects misplaced (8.485%); 52 MiB/s, 839 objects/s recovering 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: osdmap e335: 8 total, 8 up, 5 in 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:10.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:10.806 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:10.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: 9.8 scrub starts 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: pgmap v672: 158 pgs: 4 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 152 active+clean; 1.1 GiB data, 5.5 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 970 op/s; 4469/52668 objects misplaced (8.485%); 52 MiB/s, 839 objects/s recovering 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/675837995' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: osdmap e335: 8 total, 8 up, 5 in 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:10.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.846 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:10.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:10.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:10.972 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:10.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:11.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.101 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:11.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:11.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.240 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:11.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:11.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:11.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:11.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:11.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:11.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:11.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:11.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:11.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:11.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:11.772 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:11.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:11.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:11.899 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:11.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:11.950 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[52984]: 2024-03-26T04:03:11.685+0000 7f75800bc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:03:11.950 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[52984]: 2024-03-26T04:03:11.685+0000 7f75800bc700 -1 osd.2 335 *** Got signal Terminated *** 2024-03-26T04:03:11.950 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[52984]: 2024-03-26T04:03:11.685+0000 7f75800bc700 -1 osd.2 335 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:03:12.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:12.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:12.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:12.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:12.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:12.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:12.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:12.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:12.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:12.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:12.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:12.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:12.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.796 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:12.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:12 smithi086 ceph-mon[32965]: pgmap v674: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 949 op/s; 3532/44823 objects misplaced (7.880%); 25 MiB/s, 402 objects/s recovering 2024-03-26T04:03:12.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:12 smithi086 ceph-mon[32965]: osd.2 marked itself down and dead 2024-03-26T04:03:12.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:12 smithi006 ceph-mon[29446]: pgmap v674: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 949 op/s; 3532/44823 objects misplaced (7.880%); 25 MiB/s, 402 objects/s recovering 2024-03-26T04:03:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:12 smithi006 ceph-mon[29446]: osd.2 marked itself down and dead 2024-03-26T04:03:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:12 smithi006 ceph-mon[37501]: pgmap v674: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 949 op/s; 3532/44823 objects misplaced (7.880%); 25 MiB/s, 402 objects/s recovering 2024-03-26T04:03:12.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:12 smithi006 ceph-mon[37501]: osd.2 marked itself down and dead 2024-03-26T04:03:12.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:12.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:12.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:13.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:13.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.046 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:13.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:13.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:13.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:13.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:13.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:13.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:13.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:13.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:13.424 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:13.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:13.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:13.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:13.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:13.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:03:13.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:13.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:13.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:13.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:13 smithi086 ceph-mon[32965]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:03:13.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:13 smithi086 ceph-mon[32965]: osdmap e336: 8 total, 7 up, 5 in 2024-03-26T04:03:13.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.863 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:13.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:13 smithi006 ceph-mon[37501]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:03:13.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:13 smithi006 ceph-mon[37501]: osdmap e336: 8 total, 7 up, 5 in 2024-03-26T04:03:13.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:13 smithi006 ceph-mon[29446]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-26T04:03:13.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:13 smithi006 ceph-mon[29446]: osdmap e336: 8 total, 7 up, 5 in 2024-03-26T04:03:13.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:13.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:13.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:13.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:14.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:14.161 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:14.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:14.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:14.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:14.320 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:14 smithi006 podman[152909]: 2024-03-26 04:03:14.100284938 +0000 UTC m=+5.694755958 container died 6146c67e6437223f6aa192be0041e7d787ad5282e0c70219825adb4db80ee835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-26T04:03:14.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:14.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:14.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:14.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: pgmap v676: 158 pgs: 21 stale+active+clean, 4 active+remapped+backfill_toofull, 133 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 621 op/s; 3532/44823 objects misplaced (7.880%); 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:03:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: osdmap e337: 8 total, 7 up, 5 in 2024-03-26T04:03:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: 9.7 continuing backfill to osd.1 from (306'12991,335'16159] MIN to 335'16159 2024-03-26T04:03:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: 9.f continuing backfill to osd.4 from (309'13111,335'16019] MIN to 335'16019 2024-03-26T04:03:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: 9.2 continuing backfill to osd.7 from (308'12779,335'15723] MIN to 335'15723 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[29446]: osdmap e338: 8 total, 7 up, 5 in 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: pgmap v676: 158 pgs: 21 stale+active+clean, 4 active+remapped+backfill_toofull, 133 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 621 op/s; 3532/44823 objects misplaced (7.880%); 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: osdmap e337: 8 total, 7 up, 5 in 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: 9.7 continuing backfill to osd.1 from (306'12991,335'16159] MIN to 335'16159 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: 9.f continuing backfill to osd.4 from (309'13111,335'16019] MIN to 335'16019 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: 9.2 continuing backfill to osd.7 from (308'12779,335'15723] MIN to 335'15723 2024-03-26T04:03:14.641 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:14 smithi006 ceph-mon[37501]: osdmap e338: 8 total, 7 up, 5 in 2024-03-26T04:03:14.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:14.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:14.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: pgmap v676: 158 pgs: 21 stale+active+clean, 4 active+remapped+backfill_toofull, 133 active+clean; 920 MiB data, 5.1 GiB used, 442 GiB / 447 GiB avail; 0 B/s wr, 621 op/s; 3532/44823 objects misplaced (7.880%); 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:03:14.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: osdmap e337: 8 total, 7 up, 5 in 2024-03-26T04:03:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: 9.7 continuing backfill to osd.1 from (306'12991,335'16159] MIN to 335'16159 2024-03-26T04:03:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: 9.f continuing backfill to osd.4 from (309'13111,335'16019] MIN to 335'16019 2024-03-26T04:03:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: 9.2 continuing backfill to osd.7 from (308'12779,335'15723] MIN to 335'15723 2024-03-26T04:03:14.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:14 smithi086 ceph-mon[32965]: osdmap e338: 8 total, 7 up, 5 in 2024-03-26T04:03:14.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:14.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:14.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:14.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:14.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:14.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:15.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:15.084 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:14 smithi006 podman[152909]: 2024-03-26 04:03:14.648097951 +0000 UTC m=+6.242568970 container cleanup 6146c67e6437223f6aa192be0041e7d787ad5282e0c70219825adb4db80ee835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T04:03:15.084 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:14 smithi006 bash[152909]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:03:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:15.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:15.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:15.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:15.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:15.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:15.389 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:15 smithi006 podman[153266]: 2024-03-26 04:03:15.083983442 +0000 UTC m=+1.880563170 container remove 6146c67e6437223f6aa192be0041e7d787ad5282e0c70219825adb4db80ee835 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T04:03:15.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:15.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:15.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:15.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:15.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:15.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:15.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:15 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:15.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:15 smithi006 ceph-mon[37501]: Health check failed: Degraded data redundancy: 1290/38769 objects degraded (3.327%), 11 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:15.861 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:15 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:15.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:15 smithi006 ceph-mon[29446]: Health check failed: Degraded data redundancy: 1290/38769 objects degraded (3.327%), 11 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:15.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:15.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:15.905 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:15.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:15.949 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:03:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:03:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:256823 9.8 9:160c8922:::benchmark_data_smithi006_132277_object124366:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-26T04:03:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.869312+0000", 2024-03-26T04:03:15.950 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.010300683, 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01033533, 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 256823 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.951 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869312+0000", 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869312+0000", 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.952 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.953 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.953 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.953 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869314+0000", 2024-03-26T04:03:15.953 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.587e-06 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869320+0000", 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2580000000000001e-06 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.954 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869332+0000", 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2072000000000001e-05 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869340+0000", 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9829999999999996e-06 2024-03-26T04:03:15.955 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869731+0000", 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000390968 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869853+0000", 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121519 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.956 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,7", 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869970+0000", 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011691199999999999 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877887+0000", 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079171569999999993 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.957 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:256824 9.8 9:1911be26:::benchmark_data_smithi006_132277_object124367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.869615+0000", 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0099980940000000008, 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010138380000000001, 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:03:15.958 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 256824 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869615+0000", 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.959 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869615+0000", 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869616+0000", 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6000000000000003e-07 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869620+0000", 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.25e-06 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.960 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869629+0000", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2110000000000004e-06 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.869637+0000", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.503e-06 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.872062+0000", 2024-03-26T04:03:15.961 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024251239999999999 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.872170+0000", 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010774500000000001 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,7", 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.872278+0000", 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000108083 2024-03-26T04:03:15.962 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15312.0:256805 9.8 e338/333 ondisk, result = 0)", 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.873093+0000", 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0065203040000000002, 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067476330000000003, 2024-03-26T04:03:15.963 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.964 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.964 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:03:15.964 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873093+0000", 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873093+0000", 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.965 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873093+0000", 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8400000000000005e-07 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873097+0000", 2024-03-26T04:03:15.966 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4649999999999999e-06 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873104+0000", 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0419999999999997e-06 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873109+0000", 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5289999999999999e-06 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.967 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877985+0000", 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048758650000000001 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878004+0000", 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9000000000000001e-05 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.968 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15312.0:256819 9.0 e338/336 9:030e38c3:::benchmark_data_smithi006_132277_object124362:head v 338'16084, mlcod=338'16081)", 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.873126+0000", 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0064874959999999997, 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0067881549999999997, 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873126+0000", 2024-03-26T04:03:15.969 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873126+0000", 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873126+0000", 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1300000000000001e-07 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.970 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873130+0000", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4700000000000004e-06 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873137+0000", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2720000000000003e-06 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.873142+0000", 2024-03-26T04:03:15.971 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0139999999999998e-06 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877925+0000", 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047831690000000003 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877946+0000", 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0992000000000002e-05 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.972 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15312.0:256816 9.8 e338/333)", 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.875151+0000", 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0044623700000000002, 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048307769999999996, 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.973 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875151+0000", 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875151+0000", 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.974 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875151+0000", 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1099999999999995e-07 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875156+0000", 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4750000000000004e-06 2024-03-26T04:03:15.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875166+0000", 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6129999999999993e-06 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875173+0000", 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0859999999999996e-06 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.976 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15312.0:256825 9.0 e338/336)", 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.875187+0000", 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0044260219999999999, 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048531809999999998, 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.977 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875187+0000", 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875187+0000", 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875187+0000", 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.41e-07 2024-03-26T04:03:15.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875191+0000", 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9779999999999999e-06 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875198+0000", 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1600000000000003e-06 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.979 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875203+0000", 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2610000000000001e-06 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15312.0:256824 9.8 e338/333)", 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.875956+0000", 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0036571759999999998, 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041403680000000002, 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.980 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875956+0000", 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875956+0000", 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.981 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875956+0000", 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7099999999999997e-07 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875959+0000", 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6699999999999998e-06 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.982 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875965+0000", 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8850000000000004e-06 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.875983+0000", 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7975e-05 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15312.0:256823 9.8 e338/333)", 2024-03-26T04:03:15.983 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.877884+0000", 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0017290230000000001, 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022638110000000001, 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877884+0000", 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.984 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877884+0000", 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877885+0000", 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.85e-07 2024-03-26T04:03:15.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877890+0000", 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4859999999999997e-06 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877899+0000", 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0650000000000005e-06 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.986 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877907+0000", 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6599999999999995e-06 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop_reply(client.15312.0:256824 9.8 e338/333)", 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.877920+0000", 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0016928449999999999, 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022981120000000002, 2024-03-26T04:03:15.987 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877920+0000", 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.988 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877920+0000", 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877921+0000", 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8700000000000002e-07 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.989 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877923+0000", 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.689e-06 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877929+0000", 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4659999999999998e-06 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.990 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.877934+0000", 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0889999999999999e-06 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:256830 9.8 9.72d92908 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.878343+0000", 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00126965, 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019293839999999999, 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.991 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "queued for pg", 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 256830 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878343+0000", 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.992 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878343+0000", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878345+0000", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.279e-06 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878351+0000", 2024-03-26T04:03:15.993 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9510000000000003e-06 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878675+0000", 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00032443899999999999 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.878686+0000", 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1287000000000001e-05 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:15.994 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15312.0:256831 9.c e338/332 9:3d9dcacf:::benchmark_data_smithi006_132277_object124374:head v 338'16192, mlcod=338'16190)", 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.879133+0000", 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00048033800000000002, 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012057400000000001, 2024-03-26T04:03:15.995 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879133+0000", 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.996 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.997 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.997 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:15.997 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879133+0000", 2024-03-26T04:03:15.997 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879134+0000", 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7199999999999997e-07 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879140+0000", 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4099999999999996e-06 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.998 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879154+0000", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3437e-05 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879160+0000", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8009999999999998e-06 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879226+0000", 2024-03-26T04:03:15.999 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5508999999999999e-05 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879245+0000", 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.889e-05 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:256832 9.9 9:9e9b360f:::benchmark_data_smithi006_132277_object124375:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-26T04:03:16.000 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.879458+0000", 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.00015516, 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00095308199999999995, 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:03:16.001 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 256832 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879458+0000", 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879458+0000", 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879459+0000", 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2900000000000002e-07 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:16.003 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879464+0000", 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5679999999999999e-06 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879475+0000", 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0094e-05 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.004 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879482+0000", 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1659999999999997e-06 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879537+0000", 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4963000000000002e-05 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:16.005 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879657+0000", 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120635 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.879772+0000", 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114261 2024-03-26T04:03:16.006 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15312.0:256806 9.0 e338/336 9:07b3c8b7:::benchmark_data_smithi006_132277_object124349:head v 338'16083, mlcod=338'16081)", 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.859938+0000", 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.019674844, 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.020559121999999999, 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:16.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859938+0000", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859938+0000", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859939+0000", 2024-03-26T04:03:16.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4899999999999995e-07 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859944+0000", 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0590000000000002e-06 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859952+0000", 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5800000000000003e-06 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.009 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.859957+0000", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8170000000000004e-06 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862850+0000", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028921860000000001 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862872+0000", 2024-03-26T04:03:16.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1963e-05 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:256816 9.8 9:17a75f5c:::benchmark_data_smithi006_132277_object124359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e338)", 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:15.862965+0000", 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.016647694000000001, 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017603305999999999, 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:03:16.011 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 256816 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862965+0000", 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:03:16.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862965+0000", 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862967+0000", 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2219999999999999e-06 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862972+0000", 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6180000000000003e-06 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862981+0000", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8789999999999992e-06 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.862988+0000", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0740000000000004e-06 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.863092+0000", 2024-03-26T04:03:16.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000103546 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.863207+0000", 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011502699999999999 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,7", 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.863318+0000", 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111261 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.866271+0000", 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029527580000000002 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:15.866311+0000", 2024-03-26T04:03:16.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9799000000000003e-05 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 14 2024-03-26T04:03:16.017 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:03:16.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:16.058 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:16.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:15 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:15 smithi086 ceph-mon[32965]: Health check failed: Degraded data redundancy: 1290/38769 objects degraded (3.327%), 11 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:16.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:16.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:16.294 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:16.332 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:16.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:16.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:16.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:16.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:16.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:16.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:16.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:16.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:16.868 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:16.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:16.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:16.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:16.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:16 smithi006 ceph-mon[37501]: pgmap v679: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 21 active+undersized, 13 stale+active+clean, 3 active+remapped+backfill_toofull, 110 active+clean; 794 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 784 op/s; 1290/38769 objects degraded (3.327%); 3200/38769 objects misplaced (8.254%); 2.1 MiB/s, 33 objects/s recovering 2024-03-26T04:03:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:16.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:16 smithi006 ceph-mon[29446]: pgmap v679: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 21 active+undersized, 13 stale+active+clean, 3 active+remapped+backfill_toofull, 110 active+clean; 794 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 784 op/s; 1290/38769 objects degraded (3.327%); 3200/38769 objects misplaced (8.254%); 2.1 MiB/s, 33 objects/s recovering 2024-03-26T04:03:16.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:17.006 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:17.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:16 smithi086 ceph-mon[32965]: pgmap v679: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 10 active+undersized+degraded, 21 active+undersized, 13 stale+active+clean, 3 active+remapped+backfill_toofull, 110 active+clean; 794 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 9.7 KiB/s rd, 73 KiB/s wr, 784 op/s; 1290/38769 objects degraded (3.327%); 3200/38769 objects misplaced (8.254%); 2.1 MiB/s, 33 objects/s recovering 2024-03-26T04:03:17.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:17.139 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:17.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:17.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T04:03:17.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:17.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:17.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:17.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:17.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:17.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:17.591 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:17.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:17.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:17.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:17.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.864 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:17.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:17.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:17.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:17.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:18.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:18.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:18.139 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:18.185 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:18.198 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:03:18.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:18.220 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:03:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.240 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:18.241 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:03:18.260 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:03:18.280 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:03:18.300 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:03:18.320 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:03:18.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:03:18.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:18.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:18.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:18.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:18.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:18.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:18.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:18.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[37501]: pgmap v680: 158 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 567 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 953 op/s; 3632/27864 objects degraded (13.035%); 2098/27864 objects misplaced (7.529%); 3.7 MiB/s, 53 objects/s recovering 2024-03-26T04:03:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[37501]: osdmap e339: 8 total, 7 up, 5 in 2024-03-26T04:03:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3077620891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[29446]: pgmap v680: 158 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 567 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 953 op/s; 3632/27864 objects degraded (13.035%); 2098/27864 objects misplaced (7.529%); 3.7 MiB/s, 53 objects/s recovering 2024-03-26T04:03:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[29446]: osdmap e339: 8 total, 7 up, 5 in 2024-03-26T04:03:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:18 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3077620891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:18.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:18.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:19.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:18 smithi086 ceph-mon[32965]: pgmap v680: 158 pgs: 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 567 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 9.5 KiB/s rd, 73 KiB/s wr, 953 op/s; 3632/27864 objects degraded (13.035%); 2098/27864 objects misplaced (7.529%); 3.7 MiB/s, 53 objects/s recovering 2024-03-26T04:03:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:18 smithi086 ceph-mon[32965]: osdmap e339: 8 total, 7 up, 5 in 2024-03-26T04:03:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:18 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3077620891' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:19.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:19.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:19.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:19.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:19.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:19.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:19.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:19.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:19.383 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:19.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:19.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:19.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:19.644 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:19.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:19.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:19.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.786 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:19.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:19.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:19.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[29446]: osdmap e340: 8 total, 7 up, 5 in 2024-03-26T04:03:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:03:19.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:03:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[37501]: osdmap e340: 8 total, 7 up, 5 in 2024-03-26T04:03:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:03:19.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:19 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:03:19.927 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:19.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:20.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:20.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:20.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:19 smithi086 ceph-mon[32965]: osdmap e340: 8 total, 7 up, 5 in 2024-03-26T04:03:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:19 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:03:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:19 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:03:20.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:20.201 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:20.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:20.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:20.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:20.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:20.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:20.470 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:20.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:20.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:20.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:20.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:20.733 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:20.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:20.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:20.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:20.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:20.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:20.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[29446]: pgmap v683: 158 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 441 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 1.30k op/s; 2707/21849 objects degraded (12.390%); 1449/21849 objects misplaced (6.632%); 7.7 MiB/s, 118 objects/s recovering 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[29446]: osdmap e341: 8 total, 7 up, 5 in 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[37501]: pgmap v683: 158 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 441 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 1.30k op/s; 2707/21849 objects degraded (12.390%); 1449/21849 objects misplaced (6.632%); 7.7 MiB/s, 118 objects/s recovering 2024-03-26T04:03:20.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:20 smithi006 ceph-mon[37501]: osdmap e341: 8 total, 7 up, 5 in 2024-03-26T04:03:20.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:21.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:20 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:03:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:20 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:03:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:20 smithi086 ceph-mon[32965]: pgmap v683: 158 pgs: 1 remapped+peering, 1 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 441 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 10 KiB/s rd, 74 KiB/s wr, 1.30k op/s; 2707/21849 objects degraded (12.390%); 1449/21849 objects misplaced (6.632%); 7.7 MiB/s, 118 objects/s recovering 2024-03-26T04:03:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:20 smithi086 ceph-mon[32965]: osdmap e341: 8 total, 7 up, 5 in 2024-03-26T04:03:21.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:21.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:21.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:21.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:21.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:21.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:21.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:21.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:21.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:21.570 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:21.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:21.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:21.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:21.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:21.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:21 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1144/12126 objects degraded (9.434%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:21.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:21 smithi006 ceph-mon[37501]: osdmap e342: 8 total, 7 up, 5 in 2024-03-26T04:03:21.890 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:21 smithi006 podman[154112]: 2024-03-26 04:03:21.550189364 +0000 UTC m=+0.045094732 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:03:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:21 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1144/12126 objects degraded (9.434%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:21 smithi006 ceph-mon[29446]: osdmap e342: 8 total, 7 up, 5 in 2024-03-26T04:03:21.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:21.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:22.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:21 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1144/12126 objects degraded (9.434%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:22.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:21 smithi086 ceph-mon[32965]: osdmap e342: 8 total, 7 up, 5 in 2024-03-26T04:03:22.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:22.231 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:22.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.271 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:22.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.417 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:22.518 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:22.556 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:22.639 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Removed 132454 objects 2024-03-26T04:03:22.639 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Clean up completed and total clean up time :140.537 2024-03-26T04:03:22.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:22.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:22.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:22.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.809 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:22.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:22.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:22.935 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:22.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:22.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:03:22.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:03:22.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[37501]: pgmap v685: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 B/s wr, 1.48k op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 9.2 MiB/s, 142 objects/s recovering 2024-03-26T04:03:22.956 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:03:22.956 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:03:22.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:22 smithi006 ceph-mon[29446]: pgmap v685: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 B/s wr, 1.48k op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 9.2 MiB/s, 142 objects/s recovering 2024-03-26T04:03:23.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:23.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:23.177 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:03:23.177 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:03:23.178 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:22 smithi086 ceph-mon[32965]: pgmap v685: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 682 B/s rd, 0 B/s wr, 1.48k op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 9.2 MiB/s, 142 objects/s recovering 2024-03-26T04:03:23.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:23.302 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:23.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:23.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:23.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:23.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:23.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:23.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:23.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:23.681 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:23.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:03:23.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:23.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.809 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:23.818 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-26T04:03:23.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-26T04:03:23.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:23 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1927614083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:23.843 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:23 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:23.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:23 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1927614083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:23.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:23 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:23.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:23.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:23.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:24.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:24.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:23 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1927614083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:23 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:03:24.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:24 smithi006 podman[154112]: 2024-03-26 04:03:24.029215599 +0000 UTC m=+2.524120997 container create 3502e48dc652783c8d6d78ddbd2e8c7949cb8fe5eb204354e5da76a53cd09f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:03:24.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.197 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:24.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.324 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:24.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:24.534 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:24.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:03:24.559 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:03:24.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:24.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:24.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: pgmap v687: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 872 B/s rd, 0 B/s wr, 894 op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: osdmap e343: 8 total, 7 up, 5 in 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: osdmap e344: 8 total, 7 up, 5 in 2024-03-26T04:03:24.795 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: pgmap v687: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 872 B/s rd, 0 B/s wr, 894 op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: osdmap e343: 8 total, 7 up, 5 in 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: osdmap e344: 8 total, 7 up, 5 in 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:24.796 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:24.812 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:24.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:24.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:24.963 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:24.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:25.050 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: pgmap v687: 158 pgs: 2 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 1 remapped+peering, 27 active+undersized+degraded, 51 active+undersized, 76 active+clean; 239 MiB data, 3.6 GiB used, 443 GiB / 447 GiB avail; 872 B/s rd, 0 B/s wr, 894 op/s; 1144/12126 objects degraded (9.434%); 506/12126 objects misplaced (4.173%); 6.5 MiB/s, 104 objects/s recovering 2024-03-26T04:03:25.050 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: osdmap e343: 8 total, 7 up, 5 in 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: osdmap e344: 8 total, 7 up, 5 in 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:25.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:25.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:25.105 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:24 smithi006 podman[154112]: 2024-03-26 04:03:24.938148419 +0000 UTC m=+3.433053775 container init 3502e48dc652783c8d6d78ddbd2e8c7949cb8fe5eb204354e5da76a53cd09f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:03:25.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:24 smithi006 podman[154112]: 2024-03-26 04:03:24.94209068 +0000 UTC m=+3.436996040 container start 3502e48dc652783c8d6d78ddbd2e8c7949cb8fe5eb204354e5da76a53cd09f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T04:03:25.106 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:25 smithi006 podman[154112]: 2024-03-26 04:03:25.004837258 +0000 UTC m=+3.499742630 container attach 3502e48dc652783c8d6d78ddbd2e8c7949cb8fe5eb204354e5da76a53cd09f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T04:03:25.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:25.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:25.234 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:03:25.234 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:03:25.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:25.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:25.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:25.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:25.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.481 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:25.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:25.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:25.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:25.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:25.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.738 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:25.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:25.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.869 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:25.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:25.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:25.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:25.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:26.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:26.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.125 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:26.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:26.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:03:25 smithi006 podman[154112]: 2024-03-26 04:03:25.845794828 +0000 UTC m=+4.340700179 container died 3502e48dc652783c8d6d78ddbd2e8c7949cb8fe5eb204354e5da76a53cd09f05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-03-26T04:03:26.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:26.252 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:26.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:26.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:26.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:26.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:26.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:26.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:03:26.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:26.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.638 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:26.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:26.739 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:26.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:26.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:26.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T04:03:26.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:26.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:26.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:26.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:27.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:27.027 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:27.027 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:26 smithi006 ceph-mon[37501]: pgmap v690: 158 pgs: 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 76 active+clean; 118 MiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 874 B/s rd, 0 B/s wr, 883 op/s; 810/6342 objects degraded (12.772%); 506/6342 objects misplaced (7.979%); 2.5 MiB/s, 39 objects/s recovering 2024-03-26T04:03:27.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:27.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:26 smithi006 ceph-mon[29446]: pgmap v690: 158 pgs: 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 76 active+clean; 118 MiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 874 B/s rd, 0 B/s wr, 883 op/s; 810/6342 objects degraded (12.772%); 506/6342 objects misplaced (7.979%); 2.5 MiB/s, 39 objects/s recovering 2024-03-26T04:03:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:26 smithi086 ceph-mon[32965]: pgmap v690: 158 pgs: 3 active+clean+remapped, 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+undersized+degraded, 52 active+undersized, 76 active+clean; 118 MiB data, 3.3 GiB used, 444 GiB / 447 GiB avail; 874 B/s rd, 0 B/s wr, 883 op/s; 810/6342 objects degraded (12.772%); 506/6342 objects misplaced (7.979%); 2.5 MiB/s, 39 objects/s recovering 2024-03-26T04:03:27.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:27.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:27.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:27.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.280 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:27.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:27.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:03:27.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:27.398 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:03:27.398 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:03:27.399 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2024-03-26T04:03:27.400 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi006.front.sepia.ceph.com 2024-03-26T04:03:27.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.2 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2024-03-26T04:03:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:27.422 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:27.436 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:27.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:27.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:27.622 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.2/config 2024-03-26T04:03:27.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:27.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:27.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:27.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:27.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:03:27.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:27.916 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:27 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 109/687 objects degraded (15.866%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:27.916 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:27 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 109/687 objects degraded (15.866%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:27.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:27.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:27.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:28.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:28.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:27 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 109/687 objects degraded (15.866%), 23 pgs degraded (PG_DEGRADED) 2024-03-26T04:03:28.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:28.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:28.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:28.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:28.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:28.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:28.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:28.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:28.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:28.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:28.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:28.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:28.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:28.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:28.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:29.006 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:28 smithi086 ceph-mon[32965]: pgmap v691: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 636 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:29.027 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:28 smithi006 ceph-mon[37501]: pgmap v691: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 636 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:29.027 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:28 smithi006 ceph-mon[29446]: pgmap v691: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 636 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:29.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.095 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:29.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:29.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:29.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:29.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.350 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:29.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:29.452 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:29.478 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:29.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:29.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:29.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.605 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:29.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:29.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:29.731 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:29.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:29.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:29.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.858 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:29.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:29.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:29.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:29.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:30.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:30.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.108 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:30.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:30.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:30.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:30.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.359 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:30.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:30.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.486 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:30.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:30.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:30.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:30 smithi086 ceph-mon[32965]: pgmap v692: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 487 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:30.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:30.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:30.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:30 smithi006 ceph-mon[37501]: pgmap v692: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 487 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:30.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:30 smithi006 ceph-mon[29446]: pgmap v692: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 487 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:30.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:30.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.739 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:30.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:30.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:30.865 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:30.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:30.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:30.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:30.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:30.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:31.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:31.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.118 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:31.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:31.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.245 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:31.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:31.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:31.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:31.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:31.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:31.597 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:31.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:31.711 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:03:31.711 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:03:31.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:31.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:31.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:31.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:31.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:31.882 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:31.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:32.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:32.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:32.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:32.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:32.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:32.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T04:03:32.266 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:32.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:32.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:32.392 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:32.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:32.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:32.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:32.625 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:32.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:32.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:32.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:32.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:32 smithi086 ceph-mon[32965]: pgmap v693: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:32.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:32.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:32 smithi006 ceph-mon[37501]: pgmap v693: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:32.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:32 smithi006 ceph-mon[29446]: pgmap v693: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 477 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:32.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:32.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:32.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:33.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:33.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:33.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:33.171 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:33.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:33.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:33.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:33.426 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:33.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:33.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:33.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:33.638 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:03:33.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:33.657 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:03:33.683 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:03:33.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.685 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:33.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:33.705 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:03:33.725 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:03:33.745 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:03:33.766 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:03:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:03:33.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:33.809 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:33.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:33.940 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:33.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:33.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:34.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.066 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:34.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:34.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T04:03:34.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:34.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:34.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:34.447 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:34.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:34.459 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:34.459 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:34.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:34.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:34.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: pgmap v694: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 390 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3661693983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:03:34.749 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: pgmap v694: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 390 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3661693983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:03:34.750 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:34 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:34.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:34.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:34.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: pgmap v694: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 390 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3661693983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:03:34.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:34 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:34.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:34.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:34.966 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:34.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:35.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:35.091 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:35.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.092 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:35.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:35.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.220 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:35.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:35.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:35.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:35.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:35.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:35.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:35.465 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:35 smithi006 ceph-mon[29446]: pgmap v695: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 386 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:35.465 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:35 smithi006 ceph-mon[29446]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:03:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.477 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:35.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:35.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:35.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:35.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:35.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:35.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:35.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:35.758 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:35 smithi006 ceph-mon[37501]: pgmap v695: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 386 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:35.758 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:35 smithi006 ceph-mon[37501]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:03:35.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:35 smithi086 ceph-mon[32965]: pgmap v695: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 386 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:35.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:35 smithi086 ceph-mon[32965]: Health check failed: 1 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-26T04:03:35.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:35.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:35.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:35.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:36.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:36.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:36.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:36.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:36.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:36.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:36.352 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:36.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:36.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:36.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:36.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:36.508 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_2' removed 2024-03-26T04:03:36.594 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:36.594 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3406699278' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.595 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:36.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3406699278' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.595 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:36 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:36.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:36.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:36.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:36.759 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:36.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3406699278' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:36 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:03:36.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:36.888 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:36.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:36.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:36.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:37.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.015 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:37.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:37.024 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:03:37.054 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:03:37.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:37.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:37.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:37.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:37.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:37.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.394 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:37.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:37.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:37.519 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:37.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:37.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:37.645 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:37.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:37.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:37.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.772 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:37.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:37 smithi086 ceph-mon[32965]: pgmap v696: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 943 B/s rd, 0 B/s wr, 174 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:37 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:03:37.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:37 smithi086 ceph-mon[32965]: osdmap e345: 8 total, 7 up, 5 in 2024-03-26T04:03:37.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:37.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[37501]: pgmap v696: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 943 B/s rd, 0 B/s wr, 174 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:37.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:03:37.884 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[37501]: osdmap e345: 8 total, 7 up, 5 in 2024-03-26T04:03:37.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[29446]: pgmap v696: 158 pgs: 1 active+undersized+remapped+backfill_toofull, 3 active+clean+remapped, 23 active+undersized+degraded, 55 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 943 B/s rd, 0 B/s wr, 174 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:37.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:03:37.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:37 smithi006 ceph-mon[29446]: osdmap e345: 8 total, 7 up, 5 in 2024-03-26T04:03:37.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:37.898 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:37.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:37.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:38.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:38.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:38.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:38.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.153 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:38.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:38.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:38.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:38.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:38.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:38.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:38.413 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T04:03:38.414 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T04:03:38.414 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-26T04:03:38.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-26T04:03:38.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:38.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:38.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:38.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:38 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:03:38.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:38 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:03:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:38.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:38.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.783 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:38.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:38.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:38 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:03:38.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:38.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:38.909 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:38.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:39.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:39.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:03:39.057 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:03:39.058 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:03:39.058 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T04:03:39.058 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:03:39.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:39.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:39.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:39.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:39.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:39.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:39.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:39.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:39.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:39.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:39.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:39.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:39.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:39 smithi086 ceph-mon[32965]: pgmap v698: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 462 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:39.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:39 smithi086 ceph-mon[32965]: osdmap e346: 8 total, 7 up, 5 in 2024-03-26T04:03:39.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:39 smithi006 ceph-mon[37501]: pgmap v698: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 462 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:39 smithi006 ceph-mon[37501]: osdmap e346: 8 total, 7 up, 5 in 2024-03-26T04:03:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:39 smithi006 ceph-mon[29446]: pgmap v698: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 462 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:39 smithi006 ceph-mon[29446]: osdmap e346: 8 total, 7 up, 5 in 2024-03-26T04:03:39.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:39.921 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:39.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:40.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:9.f 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:9.b 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:9.a 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:9.7 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:4.4 2024-03-26T04:03:40.024 INFO:teuthology.orchestra.run.smithi006.stdout:4.1f 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.1c 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.18 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:5.4 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:6.1c 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.17 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.14 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:6.14 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:3.14 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:5.d 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:6.1d 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:3.c 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:1.8 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.11 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.1a 2024-03-26T04:03:40.025 INFO:teuthology.orchestra.run.smithi006.stdout:4.10 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:6.9 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:6.13 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:1.5 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:4.13 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:1.a 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.1f 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:6.8 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:1.9 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.16 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.5 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.9 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.4 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:6.1f 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:3.1a 2024-03-26T04:03:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:5.1c 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.16 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:6.1 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:6.5 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.d 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:2.1 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.19 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:5.8 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.1d 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:3.10 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.1b 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:3.17 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.12 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:4.6 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:3.18 2024-03-26T04:03:40.027 INFO:teuthology.orchestra.run.smithi006.stdout:5.14 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:3.0 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:1.7 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:5.b 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:1.1 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:5.18 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:6.3 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:9.6 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:5.19 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:5.1e 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:6.17 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:3.1e 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:5.2 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:3.13 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:6.2 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:6.a 2024-03-26T04:03:40.028 INFO:teuthology.orchestra.run.smithi006.stdout:6.f 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.19 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:5.6 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:5.a 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:5.e 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:5.f 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.0 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.11 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:5.1f 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.12 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.d 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.4 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:6.e 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:9.0 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:9.1 2024-03-26T04:03:40.029 INFO:teuthology.orchestra.run.smithi006.stdout:9.2 2024-03-26T04:03:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.055 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:40.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:40.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:40.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:40.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:40.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:40.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.435 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:40.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:40.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:40.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:40.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T04:03:40.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:40.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:40.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:40.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:40.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:40.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:40.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:40.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:40.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:40.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:40.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:41.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:41.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:41.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:41.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:41.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.2 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.13 --file /var/log/ceph/exp.4.13.2 2024-03-26T04:03:41.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:41.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:41.224 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:41.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:41.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:41.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:41.426 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:41.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:41.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:41.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:41.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:41.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:41.705 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:41.756 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:41 smithi006 ceph-mon[37501]: pgmap v700: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:41.756 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:41 smithi006 ceph-mon[29446]: pgmap v700: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:41.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.832 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:41.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:41.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:41.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:41.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:41.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:42.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:42.086 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.2/config 2024-03-26T04:03:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:41 smithi086 ceph-mon[32965]: pgmap v700: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:42.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:42.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:42.216 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:42.318 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:42.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:42.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:42.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:42.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:42.602 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:42.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:42.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:42.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.859 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:42.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:42.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:42.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:42.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:43.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:43.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:43.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:43.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:43.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:43.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:43.384 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:43.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:43.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:43.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:43.627 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:03:43.633 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:43.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:43.735 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:43.760 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:43.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:43.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:43.863 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:43 smithi006 ceph-mon[37501]: pgmap v701: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:43.864 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:03:43.864 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:43 smithi006 ceph-mon[29446]: pgmap v701: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:43.864 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:03:43.890 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:43.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:43.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:43.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:44.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:43 smithi086 ceph-mon[32965]: pgmap v701: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:03:44.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:44.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:44.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:44.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:44.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:44.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:44.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:44.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:44.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.531 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:44.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:44.615 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:44.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:44.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:44.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:44.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.783 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:44.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:44.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:44.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:44.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[37501]: osdmap e347: 8 total, 7 up, 5 in 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[37501]: osdmap e348: 8 total, 7 up, 5 in 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[29446]: osdmap e347: 8 total, 7 up, 5 in 2024-03-26T04:03:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:44 smithi006 ceph-mon[29446]: osdmap e348: 8 total, 7 up, 5 in 2024-03-26T04:03:44.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:44.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:45.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:45.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:45.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:44 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:03:45.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1736175490' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:03:45.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:44 smithi086 ceph-mon[32965]: osdmap e347: 8 total, 7 up, 5 in 2024-03-26T04:03:45.072 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:44 smithi086 ceph-mon[32965]: osdmap e348: 8 total, 7 up, 5 in 2024-03-26T04:03:45.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:45.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:45.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:45.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.297 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:45.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:45.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:45.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:45.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:45.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:45.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:45.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:45.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:45.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:45.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:45.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:45 smithi006 ceph-mon[37501]: pgmap v703: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 650 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:45.872 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:45 smithi006 ceph-mon[29446]: pgmap v703: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 650 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:45.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:45.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:45.947 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:45.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:46.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:46.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:45 smithi086 ceph-mon[32965]: pgmap v703: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 650 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:46.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:46.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:46.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:46.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.334 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:46.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:46.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:46.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:46.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:46.566 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:46.571 INFO:teuthology.orchestra.run.smithi006.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T04:03:46.592 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:46.632 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_3' created 2024-03-26T04:03:46.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:46.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:46.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:46.806 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:46.806 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T04:03:46.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:46.806 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T04:03:46.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:03:46.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:46.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:46.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:46.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:46.978 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:46.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:47.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-26T04:03:47.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.103 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:47.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:47.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:47.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:47.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:47.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:47.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:47.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:47.488 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:47.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:47.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:47.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:47.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:47.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.744 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:47.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:47.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.870 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:47.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:47.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[37501]: pgmap v705: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T04:03:47.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[37501]: osdmap e349: 8 total, 7 up, 5 in 2024-03-26T04:03:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[29446]: pgmap v705: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T04:03:47.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:47 smithi006 ceph-mon[29446]: osdmap e349: 8 total, 7 up, 5 in 2024-03-26T04:03:47.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:47.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:47.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:47 smithi086 ceph-mon[32965]: pgmap v705: 142 pgs: 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.5 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2027677331' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-26T04:03:48.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:47 smithi086 ceph-mon[32965]: osdmap e349: 8 total, 7 up, 5 in 2024-03-26T04:03:48.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:48.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:48.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:48.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:48.254 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:48.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:48.361 INFO:teuthology.orchestra.run.smithi006.stderr:Exporting 4.13 info 4.13( v 241'19 (0'0,241'19] local-lis/les=293/294 n=5 ec=67/67 lis/c=293/293 les/c/f=294/294/0 sis=293) 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Read #4:c8333aab:gc::gc.11:head# 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Read #4:cba150de:::obj_delete_at_hint.0000000051:head# 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Read #4:cc909b6b:::obj_delete_at_hint.0000000106:head# 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Read #4:cd178ae0:reshard::reshard.0000000014:head# 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Read #4:cedbb205:::obj_delete_at_hint.0000000017:head# 2024-03-26T04:03:48.364 INFO:teuthology.orchestra.run.smithi006.stderr:Export successful 2024-03-26T04:03:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.385 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:48.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:48.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:48.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:48.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:48.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:48.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:48.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.775 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:48.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:48.876 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:48.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:48 smithi006 ceph-mon[37501]: osdmap e350: 8 total, 7 up, 5 in 2024-03-26T04:03:48.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:48 smithi006 ceph-mon[29446]: osdmap e350: 8 total, 7 up, 5 in 2024-03-26T04:03:48.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:48.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:49.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:49.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:49.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:48 smithi086 ceph-mon[32965]: osdmap e350: 8 total, 7 up, 5 in 2024-03-26T04:03:49.098 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-26T04:03:49.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.169 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:49.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:49.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:49.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.295 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:49.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:49.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:49.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:49.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:49.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:49.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:49.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:49.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: pgmap v708: 158 pgs: 16 unknown, 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: osdmap e351: 8 total, 7 up, 5 in 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[29446]: osdmap e352: 8 total, 7 up, 5 in 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: pgmap v708: 158 pgs: 16 unknown, 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:03:49.678 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: osdmap e351: 8 total, 7 up, 5 in 2024-03-26T04:03:49.679 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:03:49.679 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:03:49.679 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:49 smithi006 ceph-mon[37501]: osdmap e352: 8 total, 7 up, 5 in 2024-03-26T04:03:49.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:49.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:49.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.2 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.13 2024-03-26T04:03:49.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:49.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:49.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:49.909 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.2/config 2024-03-26T04:03:49.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:49.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:49.950 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:03:49.970 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:03:49.990 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:03:50.011 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:03:50.031 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:03:50.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:50.051 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:03:50.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:50.071 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:03:50.071 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:03:50.072 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:03:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: pgmap v708: 158 pgs: 16 unknown, 23 active+undersized+degraded, 51 active+undersized, 68 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:03:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: osdmap e351: 8 total, 7 up, 5 in 2024-03-26T04:03:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:03:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:03:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:49 smithi086 ceph-mon[32965]: osdmap e352: 8 total, 7 up, 5 in 2024-03-26T04:03:50.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.195 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:50.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:50.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:50.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:50.451 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:50.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:50.563 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:50.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:50.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:50.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:50.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:50.896 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2248200252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:50.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2248200252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:50.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:50.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:50.967 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:50.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:51.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2248200252' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:03:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:51.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:51.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:51.356 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:51.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:51.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:51.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:51.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:51.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:51.747 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:03:51.788 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:03:51.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:51.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:51.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[37501]: pgmap v711: 158 pgs: 8 unknown, 23 active+undersized+degraded, 56 active+undersized, 71 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[37501]: 3.8 scrub starts 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[37501]: 3.8 scrub ok 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[29446]: pgmap v711: 158 pgs: 8 unknown, 23 active+undersized+degraded, 56 active+undersized, 71 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[29446]: 3.8 scrub starts 2024-03-26T04:03:51.898 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:51 smithi006 ceph-mon[29446]: 3.8 scrub ok 2024-03-26T04:03:51.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:52.001 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:52.002 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:52.033 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:03:52.033 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:03:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:51 smithi086 ceph-mon[32965]: pgmap v711: 158 pgs: 8 unknown, 23 active+undersized+degraded, 56 active+undersized, 71 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:51 smithi086 ceph-mon[32965]: 3.8 scrub starts 2024-03-26T04:03:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:51 smithi086 ceph-mon[32965]: 3.8 scrub ok 2024-03-26T04:03:52.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:52.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.128 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:52.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:52.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:52.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:52.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:52.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:52.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:52.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.639 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:52.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:52.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:52.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:52.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:52.969 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:52 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T04:03:52.969 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:52 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T04:03:52.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:52 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T04:03:52.970 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:52 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T04:03:52.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:53.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:53.023 INFO:teuthology.orchestra.run.smithi006.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T04:03:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:52 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T04:03:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:52 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T04:03:53.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:53.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:53.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:53.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:53.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:53.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:53.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:53.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:53.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:53.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:53.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:53.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:53.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:53.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:53.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:53.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:03:53.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:53.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:53.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:53.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:53.955 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:53.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:54.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:53 smithi006 ceph-mon[37501]: pgmap v712: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 895 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:54.045 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:53 smithi006 ceph-mon[29446]: pgmap v712: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 895 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:54.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.080 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:54.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:53 smithi086 ceph-mon[32965]: pgmap v712: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 895 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:54.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:54.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:54.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:54.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:54.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:54.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T04:03:54.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:54.591 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:03:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:03:54.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:54.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:54.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:54.827 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:54.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:54.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T04:03:54.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T04:03:54.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T04:03:54.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T04:03:54.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:54.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:54.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:54.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:54.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:54.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:55.074 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T04:03:55.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T04:03:55.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T04:03:55.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T04:03:55.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:03:55.075 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:03:55.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:55.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:55.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:55.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:55.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:55.315 INFO:teuthology.orchestra.run.smithi006.stdout: marking collection for removal 2024-03-26T04:03:55.315 INFO:teuthology.orchestra.run.smithi006.stdout:setting '_remove' omap key 2024-03-26T04:03:55.315 INFO:teuthology.orchestra.run.smithi006.stdout:finish_remove_pgs 4.13_head removing 4.13 2024-03-26T04:03:55.317 INFO:teuthology.orchestra.run.smithi006.stdout:Remove successful 2024-03-26T04:03:55.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:03:55.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:55.367 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:55.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:55.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:55.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.624 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:55.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:55.667 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-26T04:03:55.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:55.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.754 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:55.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: pgmap v713: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 762 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/312909977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:55.782 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: pgmap v713: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 762 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/312909977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:55.783 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:55 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:55.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:55.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:55.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:55.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:03:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:56.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: pgmap v713: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 762 B/s rd, 0 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/312909977' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:55 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:03:56.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:56.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:56.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:56.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:56.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:56.371 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:56.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.396 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:56.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:56.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:56.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.523 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:56.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:56.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:56.647 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:56.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:56.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell -n osd.2 --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.13.2 2024-03-26T04:03:56.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:56.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:56.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:56.908 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:56.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:56.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:57.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:57.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.034 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:57.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:57.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:57.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:56 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T04:03:57.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:56 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T04:03:57.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:56 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:03:57.091 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:56 smithi086 ceph-mon[32965]: osdmap e353: 8 total, 7 up, 5 in 2024-03-26T04:03:57.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:57.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:57.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T04:03:57.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[29446]: osdmap e353: 8 total, 7 up, 5 in 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:03:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:56 smithi006 ceph-mon[37501]: osdmap e353: 8 total, 7 up, 5 in 2024-03-26T04:03:57.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:57.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:57.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:57.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:57.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:57.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:57.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:57.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:57.540 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/osd.2/config 2024-03-26T04:03:57.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:57.568 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:57.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:57.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[29446]: pgmap v715: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T04:03:57.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[37501]: pgmap v715: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:57.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:57 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:03:57.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:57.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:57.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:57.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:57.935 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:57.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:58.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:03:58.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:58.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:03:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:57 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T04:03:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:57 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T04:03:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:57 smithi086 ceph-mon[32965]: pgmap v715: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.3 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:57 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:03:58.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:58.185 DEBUG:teuthology.orchestra.run.smithi006:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-26T04:03:58.188 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T04:03:58.188 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T04:03:58.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:58.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:58.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:58.319 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:58.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:58.371 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:hints = 1 2024-03-26T04:03:58.371 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T04:03:58.371 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object prefix: benchmark_data_smithi006_159841 2024-03-26T04:03:58.371 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:03:58.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 0 0 0 0 0 0 - 0 2024-03-26T04:03:58.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:58.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:58.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:58.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:58.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:58.737 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:58.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:58.878 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:58.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:58.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:03:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:58 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T04:03:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:58 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T04:03:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:58 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:03:58.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:58 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:03:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:59.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:03:59.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T04:03:59.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T04:03:59.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:03:59.017 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:03:59.017 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T04:03:59.018 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T04:03:59.018 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:03:59.018 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:58 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:03:59.117 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:03:59.168 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:59.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:03:59.251 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:03:59.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:59.316 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:59.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:59.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 1 16 4347 4331 270.673 270.688 0.00618131 0.00367602 2024-03-26T04:03:59.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:03:59.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:59.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:03:59.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:03:59.604 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:03:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:03:59.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:59.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:59.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:03:59.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:03:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:03:59.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:03:59.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[37501]: pgmap v716: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[37501]: osdmap e354: 8 total, 7 up, 5 in 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[29446]: pgmap v716: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:03:59.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:03:59 smithi006 ceph-mon[29446]: osdmap e354: 8 total, 7 up, 5 in 2024-03-26T04:03:59.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:00.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:00.068 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:04:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:59 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T04:04:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:59 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T04:04:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:59 smithi086 ceph-mon[32965]: pgmap v716: 158 pgs: 23 active+undersized+degraded, 59 active+undersized, 76 active+clean; 431 KiB data, 3.0 GiB used, 444 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s; 109/687 objects degraded (15.866%) 2024-03-26T04:04:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:03:59 smithi086 ceph-mon[32965]: osdmap e354: 8 total, 7 up, 5 in 2024-03-26T04:04:00.106 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:04:00.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:00.153 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:04:00.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:00.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:00.193 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:04:00.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:04:00.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:00.274 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:04:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:00.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:00.320 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:04:00.320 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:04:00.321 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:04:00.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 2 16 7955 7939 248.076 225.5 0.00355458 0.00402386 2024-03-26T04:04:00.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:00.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:00.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:00.577 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:00.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:00.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:04:00.722 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:00.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:00.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:00.862 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:00.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:00.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:01.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:01.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T04:04:01.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T04:04:01.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3840583331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4228136836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:01.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T04:04:01.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T04:04:01.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3840583331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4228136836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:00 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T04:04:01.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:00 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T04:04:01.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3840583331' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:00 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:01.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4228136836' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:01.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:01.148 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:01.250 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:01.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:01.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 3 16 11631 11615 241.961 229.75 0.00257435 0.00412576 2024-03-26T04:04:01.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:01.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:01.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:01.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:01.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:01.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:01.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:01.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:01.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:01.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:01.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:01.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:01.980 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:01.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:02.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:01 smithi086 ceph-mon[32965]: pgmap v718: 158 pgs: 28 active+undersized+degraded, 54 active+undersized, 76 active+clean; 149 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 298 op/s; 1609/7818 objects degraded (20.581%) 2024-03-26T04:04:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:01 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:01 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1609/7818 objects degraded (20.581%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:01 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:01 smithi086 ceph-mon[32965]: osdmap e355: 8 total, 7 up, 5 in 2024-03-26T04:04:02.120 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:02.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[29446]: pgmap v718: 158 pgs: 28 active+undersized+degraded, 54 active+undersized, 76 active+clean; 149 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 298 op/s; 1609/7818 objects degraded (20.581%) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1609/7818 objects degraded (20.581%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[29446]: osdmap e355: 8 total, 7 up, 5 in 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[37501]: pgmap v718: 158 pgs: 28 active+undersized+degraded, 54 active+undersized, 76 active+clean; 149 MiB data, 3.2 GiB used, 444 GiB / 447 GiB avail; 1.2 KiB/s rd, 19 MiB/s wr, 298 op/s; 1609/7818 objects degraded (20.581%) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1609/7818 objects degraded (20.581%), 28 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:01 smithi006 ceph-mon[37501]: osdmap e355: 8 total, 7 up, 5 in 2024-03-26T04:04:02.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:02.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:02.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:02.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:02.371 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 4 16 15123 15107 236.029 218.25 0.00406677 0.00421783 2024-03-26T04:04:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:02.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:02.495 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:02.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:02.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:02.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:02.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:02.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:02.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:02.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:02.812 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:02 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T04:04:02.813 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:02 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T04:04:02.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:02 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T04:04:02.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:02 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T04:04:02.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:02.943 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:02.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:03.044 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:03.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.088 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:03.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:02 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T04:04:03.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:02 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T04:04:03.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:03.225 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:03.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:03.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:03.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:03.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 5 16 18220 18204 227.532 193.562 0.00540474 0.00438919 2024-03-26T04:04:03.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:03.500 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:03.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:03.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:03.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:03.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:03.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:03.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:03.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:03.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:04.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:04.050 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:04.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:04.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:04.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:04.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: 6.7 deep-scrub starts 2024-03-26T04:04:04.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: pgmap v720: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 765 B/s rd, 66 MiB/s wr, 1.06k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: 6.7 deep-scrub ok 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[37501]: osdmap e356: 8 total, 7 up, 5 in 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: 6.7 deep-scrub starts 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: pgmap v720: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 765 B/s rd, 66 MiB/s wr, 1.06k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:04.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: 6.7 deep-scrub ok 2024-03-26T04:04:04.076 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:04:04.076 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:04:04.076 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:03 smithi006 ceph-mon[29446]: osdmap e356: 8 total, 7 up, 5 in 2024-03-26T04:04:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: 6.7 deep-scrub starts 2024-03-26T04:04:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: pgmap v720: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 765 B/s rd, 66 MiB/s wr, 1.06k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: 6.7 deep-scrub ok 2024-03-26T04:04:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:04:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:04:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:03 smithi086 ceph-mon[32965]: osdmap e356: 8 total, 7 up, 5 in 2024-03-26T04:04:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:04:04.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:04:04.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:04.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:04.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:04.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:04.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:04.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:04.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 6 16 21427 21411 223.013 200.438 0.00449614 0.00447837 2024-03-26T04:04:04.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:04.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:04.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215919 8.7 8:e03686cb:::benchmark_data_smithi006_103169_object85988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 374.55592293000001, 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9954844170000001, 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.512 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215919 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.513 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478961+0000", 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478962+0000", 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.252e-06 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478968+0000", 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6489999999999996e-06 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.514 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478978+0000", 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0757999999999999e-05 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.478987+0000", 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6619999999999994e-06 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.515 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479605+0000", 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00061837100000000005 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479713+0000", 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107643 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479818+0000", 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000105418 2024-03-26T04:04:04.516 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484843+0000", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050241180000000002 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484884+0000", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1573000000000003e-05 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531169+0000", 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.046285129000000001 2024-03-26T04:04:04.517 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531182+0000", 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2838e-05 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.350540+0000", 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.81935782999999995 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.518 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.350553+0000", 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3091000000000001e-05 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474362+0000", 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.123808623 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.519 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474445+0000", 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3461999999999995e-05 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215922 8.7 8:e91e8581:::benchmark_data_smithi006_103169_object85991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 374.55559650200001, 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9959090370000001, 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.520 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215922 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.521 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479287+0000", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479288+0000", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3799999999999996e-07 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479293+0000", 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1699999999999996e-06 2024-03-26T04:04:04.522 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479304+0000", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0829000000000001e-05 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.479311+0000", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3499999999999999e-06 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482220+0000", 2024-03-26T04:04:04.523 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029082969999999998 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482335+0000", 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114765 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.482446+0000", 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000111179 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.524 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.484848+0000", 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024021229999999999 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.489800+0000", 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049523019999999996 2024-03-26T04:04:04.525 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531214+0000", 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041413932000000001 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.531223+0000", 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2799999999999992e-06 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.526 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351560+0000", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82033632300000003 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351570+0000", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0655999999999999e-05 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475126+0000", 2024-03-26T04:04:04.527 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.123555978 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475196+0000", 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9914999999999997e-05 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215939 8.0 8:06bae83a:::benchmark_data_smithi006_103169_object86008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 374.49493593900002, 2024-03-26T04:04:04.528 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.999205474, 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215939 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.529 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539948+0000", 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.530 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539949+0000", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2720000000000001e-06 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539957+0000", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7510000000000008e-06 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539970+0000", 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3461999999999999e-05 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.539989+0000", 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8306000000000001e-05 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.540071+0000", 2024-03-26T04:04:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2101999999999994e-05 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.540082+0000", 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1216000000000001e-05 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.348869+0000", 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80878707500000002 2024-03-26T04:04:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.348882+0000", 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3156000000000001e-05 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.524064+0000", 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.17518202 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for degraded object", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.524582+0000", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00051825700000000005 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528374+0000", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037915850000000001 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528484+0000", 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010967999999999999 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,4,5", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528634+0000", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015087499999999999 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537225+0000", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0085907970000000007 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537255+0000", 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9855000000000001e-05 2024-03-26T04:04:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537305+0000", 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0235000000000003e-05 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537421+0000", 2024-03-26T04:04:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011579 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537485+0000", 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3402e-05 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539153+0000", 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016686380000000001 2024-03-26T04:04:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215944 8.7 8:edad7f0e:::benchmark_data_smithi006_103169_object86013:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 374.48658695400002, 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0056385689999998, 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215944 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548297+0000", 2024-03-26T04:04:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548298+0000", 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5319999999999999e-06 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548305+0000", 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3049999999999998e-06 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548318+0000", 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3959e-05 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548327+0000", 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5760000000000006e-06 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548399+0000", 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1643000000000005e-05 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.548409+0000", 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0488e-05 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351605+0000", 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.80319530400000005 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351613+0000", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8189999999999998e-06 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539126+0000", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1875127670000001 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539245+0000", 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118879 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.539341+0000", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5586000000000003e-05 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552494+0000", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013153583 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552617+0000", 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012313 2024-03-26T04:04:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553170+0000", 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055303500000000001 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553426+0000", 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025522899999999999 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553808+0000", 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00038286700000000001 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553935+0000", 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012686699999999999 2024-03-26T04:04:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215910 8.b 8:da361f87:::benchmark_data_smithi006_103169_object85979:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69626597400003, 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.13719901, 2024-03-26T04:04:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215910 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338618+0000", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338619+0000", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.505e-06 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338625+0000", 2024-03-26T04:04:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.327e-06 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338635+0000", 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0754000000000001e-05 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338646+0000", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1031e-05 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338887+0000", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024031500000000001 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338894+0000", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3730000000000004e-06 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.356045+0000", 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.017151473 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.356056+0000", 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0589e-05 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.474260+0000", 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.118203982 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.475817+0000", 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001556661 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215912 8.b 8:d870adcc:::benchmark_data_smithi006_103169_object85981:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69621338899998, 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1374827700000001, 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215912 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338670+0000", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338671+0000", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3000000000000001e-07 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338674+0000", 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6459999999999999e-06 2024-03-26T04:04:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338681+0000", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8959999999999997e-06 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338688+0000", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4049999999999997e-06 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339027+0000", 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033922800000000001 2024-03-26T04:04:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339032+0000", 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5269999999999998e-06 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358921+0000", 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019888565 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358931+0000", 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0400999999999999e-05 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476084+0000", 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1171520420000001 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476153+0000", 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9529999999999993e-05 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215918 8.b 8:d5bbe624:::benchmark_data_smithi006_103169_object85987:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69618725300001, 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1375509960000001, 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215918 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338696+0000", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338697+0000", 2024-03-26T04:04:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.65e-07 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338699+0000", 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2280000000000001e-06 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338705+0000", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7270000000000004e-06 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338710+0000", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3689999999999998e-06 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339061+0000", 2024-03-26T04:04:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035118299999999997 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339066+0000", 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8579999999999999e-06 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358952+0000", 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019886052000000001 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358958+0000", 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.178e-06 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476183+0000", 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.117224792 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476247+0000", 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4244000000000004e-05 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215923 8.b 8:d3d0d2cc:::benchmark_data_smithi006_103169_object85992:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69616598900001, 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1376206259999999, 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215923 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:04:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338718+0000", 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4999999999999998e-07 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338721+0000", 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4310000000000001e-06 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338726+0000", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3249999999999998e-06 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338731+0000", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6030000000000004e-06 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339093+0000", 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00036193599999999999 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339098+0000", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8559999999999998e-06 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358978+0000", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019879903000000001 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.358984+0000", 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8200000000000002e-06 2024-03-26T04:04:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476277+0000", 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1172925170000001 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.476338+0000", 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1785000000000006e-05 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215928 8.f 8:f5663f39:::benchmark_data_smithi006_103169_object85997:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69614441200002, 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.199381638, 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215928 2024-03-26T04:04:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338739+0000", 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338740+0000", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4099999999999999e-07 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338742+0000", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8030000000000001e-06 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338748+0000", 2024-03-26T04:04:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7740000000000001e-06 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338754+0000", 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8340000000000004e-06 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339328+0000", 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00057439999999999998 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339335+0000", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6170000000000003e-06 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351418+0000", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012082881 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351426+0000", 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8380000000000002e-06 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.528886+0000", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1774600500000001 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.530740+0000", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001854154 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.530984+0000", 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024428999999999998 2024-03-26T04:04:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537856+0000", 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0068720939999999996 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537872+0000", 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5116000000000001e-05 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537908+0000", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.6279999999999998e-05 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.537948+0000", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0487e-05 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538014+0000", 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6076999999999998e-05 2024-03-26T04:04:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538121+0000", 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106502 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215941 8.b 8:d0ea7a54:::benchmark_data_smithi006_103169_object86010:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69611377299998, 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.152127299, 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215941 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338770+0000", 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9900000000000001e-07 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338773+0000", 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.773e-06 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338779+0000", 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3409999999999999e-06 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338786+0000", 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7170000000000002e-06 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339125+0000", 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033892500000000001 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339130+0000", 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6600000000000003e-06 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.359012+0000", 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.019882115999999998 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.359018+0000", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1380000000000004e-06 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479021+0000", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1200031269999999 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479501+0000", 2024-03-26T04:04:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00048008000000000002 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.479629+0000", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012821600000000001 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.489981+0000", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010351816999999999 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490089+0000", 2024-03-26T04:04:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010812 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490173+0000", 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3881999999999994e-05 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490507+0000", 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00033375599999999999 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490569+0000", 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2279999999999993e-05 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.490897+0000", 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000327852 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215946 8.f 8:f2c0634b:::benchmark_data_smithi006_103169_object86015:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 373.69608928500003, 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.214591854, 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215946 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338794+0000", 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338795+0000", 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.75e-07 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338797+0000", 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.65e-06 2024-03-26T04:04:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338804+0000", 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1210000000000003e-06 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338810+0000", 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0739999999999998e-06 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339366+0000", 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055637999999999998 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting_for_map not empty", 2024-03-26T04:04:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.339371+0000", 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0309999999999998e-06 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351451+0000", 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.012079546 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.351457+0000", 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.764e-06 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538151+0000", 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1866938380000001 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538279+0000", 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127262 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,4", 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.538419+0000", 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014042899999999999 2024-03-26T04:04:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552469+0000", 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014049682000000001 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.552483+0000", 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4778e-05 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553042+0000", 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00055900500000000001 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553209+0000", 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016694799999999999 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553270+0000", 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0551999999999999e-05 2024-03-26T04:04:04.600 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.553386+0000", 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116319 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235177 8.7 8:eff9d2da:::benchmark_data_smithi006_103169_object105246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 326.51351355000003, 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99837947699999996, 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235177 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521370+0000", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521371+0000", 2024-03-26T04:04:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.19e-06 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521376+0000", 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4730000000000002e-06 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521383+0000", 2024-03-26T04:04:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1409999999999999e-06 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521411+0000", 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.7846000000000001e-05 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521523+0000", 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0001121 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.521566+0000", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2948000000000003e-05 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514095+0000", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99252881900000001 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514211+0000", 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011636700000000001 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.514295+0000", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3829e-05 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519561+0000", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0052661920000000003 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519653+0000", 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2083999999999995e-05 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519686+0000", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2561000000000002e-05 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519720+0000", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4196999999999998e-05 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519750+0000", 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9730000000000002e-05 2024-03-26T04:04:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:235179 8.7 8:e08ecef2:::benchmark_data_smithi006_103169_object105248:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 326.512335732, 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99733449600000001, 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 235179 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522548+0000", 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522549+0000", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3200000000000004e-07 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522553+0000", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5210000000000004e-06 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522564+0000", 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0729e-05 2024-03-26T04:04:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522571+0000", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.742e-06 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522619+0000", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8296e-05 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:36.522627+0000", 2024-03-26T04:04:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8420000000000005e-06 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516438+0000", 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99381165900000001 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516533+0000", 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4327999999999995e-05 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3", 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.516644+0000", 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011116999999999999 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519620+0000", 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029761089999999998 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519810+0000", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000190106 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519834+0000", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4304e-05 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519861+0000", 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6877e-05 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:37.519882+0000", 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1081e-05 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253203 9.a 9:52465b47:::benchmark_data_smithi006_132277_object120746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.538790+0000", 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.496093428000002, 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1205381089999999, 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253203 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538790+0000", 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538790+0000", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538791+0000", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.049e-06 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538795+0000", 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3289999999999998e-06 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538802+0000", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4189999999999996e-06 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.538808+0000", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1419999999999998e-06 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.546991+0000", 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0081826680000000006 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.547141+0000", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015032899999999999 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,7", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.547281+0000", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000140202 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.562149+0000", 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.014867239000000001 2024-03-26T04:04:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.587728+0000", 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.025579373999999998 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.587742+0000", 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3638999999999999e-05 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.653217+0000", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.065474966 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.653355+0000", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000138386 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.659248+0000", 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00589341 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.659328+0000", 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9956999999999994e-05 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253205 9.a 9:5d2c924a:::benchmark_data_smithi006_132277_object120748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.541472+0000", 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.493411946999998, 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1291899439999999, 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253205 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541472+0000", 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541472+0000", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541473+0000", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.08e-06 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541476+0000", 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0790000000000001e-06 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541487+0000", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0984e-05 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.541493+0000", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.63e-06 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.551105+0000", 2024-03-26T04:04:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096119310000000006 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.551227+0000", 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121811 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,7", 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.551355+0000", 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000127401 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.562157+0000", 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010802223999999999 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594781+0000", 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.032624137999999997 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594792+0000", 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.076e-05 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654381+0000", 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0595889540000001 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654504+0000", 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000123266 2024-03-26T04:04:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.670627+0000", 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016122748999999999 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.670662+0000", 2024-03-26T04:04:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4937000000000001e-05 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253215 9.0 9:0c224fac:::benchmark_data_smithi006_132277_object120758:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.555498+0000", 2024-03-26T04:04:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.479385450999999, 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0762868139999999, 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253215 2024-03-26T04:04:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555498+0000", 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555498+0000", 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555499+0000", 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.004e-06 2024-03-26T04:04:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555504+0000", 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0440000000000003e-06 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555515+0000", 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0696000000000001e-05 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555523+0000", 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9859999999999994e-06 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555639+0000", 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116513 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555757+0000", 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117815 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555871+0000", 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113511 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.561493+0000", 2024-03-26T04:04:04.641 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056225320000000004 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.595431+0000", 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.033937562999999997 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.595438+0000", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6599999999999998e-06 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.622853+0000", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0274150259999999 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.622962+0000", 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010994500000000001 2024-03-26T04:04:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.631702+0000", 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.008739686 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.631785+0000", 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2832999999999998e-05 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253216 9.0 9:094d2a35:::benchmark_data_smithi006_132277_object120759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.555543+0000", 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.479340641999997, 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0765685030000001, 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253216 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555543+0000", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555543+0000", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555544+0000", 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0299999999999999e-07 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555547+0000", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3610000000000001e-06 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555554+0000", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3950000000000003e-06 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555561+0000", 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3350000000000003e-06 2024-03-26T04:04:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.557857+0000", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022960659999999998 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.557977+0000", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012021 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.558103+0000", 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000125678 2024-03-26T04:04:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.561500+0000", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0033972669999999998 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.595458+0000", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.033958212000000002 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.595463+0000", 2024-03-26T04:04:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6160000000000004e-06 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.623262+0000", 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0277989890000001 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.623354+0000", 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2213999999999995e-05 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.632042+0000", 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0086878700000000003 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.632112+0000", 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9787000000000001e-05 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253218 9.a 9:5e9f8932:::benchmark_data_smithi006_132277_object120761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.555574+0000", 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.479309997999998, 2024-03-26T04:04:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.115037965, 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253218 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555574+0000", 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555574+0000", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555574+0000", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8999999999999997e-07 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555577+0000", 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8700000000000001e-06 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555583+0000", 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2190000000000001e-06 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.555589+0000", 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7479999999999999e-06 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.560018+0000", 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0044285180000000002 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.560162+0000", 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000144339 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,3,7", 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.560309+0000", 2024-03-26T04:04:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000146864 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.562364+0000", 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0020549090000000002 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594821+0000", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.032457601000000003 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594829+0000", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8609999999999998e-06 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654543+0000", 2024-03-26T04:04:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.059713635 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654650+0000", 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000107706 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.670568+0000", 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015917996 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.670612+0000", 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3208999999999999e-05 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253220 9.f 9:f1fdf39b:::benchmark_data_smithi006_132277_object120763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.584282+0000", 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.450601378999998, 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.04658586, 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253220 2024-03-26T04:04:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584282+0000", 2024-03-26T04:04:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584282+0000", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584283+0000", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1209999999999999e-06 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584286+0000", 2024-03-26T04:04:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.864e-06 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584291+0000", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5990000000000004e-06 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584298+0000", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7710000000000001e-06 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594280+0000", 2024-03-26T04:04:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0099818990000000007 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594290+0000", 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0614999999999999e-05 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.623204+0000", 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0289137230000001 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.623328+0000", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012449100000000001 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,4", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.623464+0000", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000135945 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.627729+0000", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0042648039999999996 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.627925+0000", 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000196274 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.630716+0000", 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0027911020000000002 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.630768+0000", 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1239999999999997e-05 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.670 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.630868+0000", 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100412 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253221 9.a 9:558b54a8:::benchmark_data_smithi006_132277_object120764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.584306+0000", 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 50.450577809000002, 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0855957519999999, 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:04:04.671 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253221 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584306+0000", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:04:04.672 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584306+0000", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584306+0000", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8999999999999998e-07 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584308+0000", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8619999999999999e-06 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:04:04.673 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584311+0000", 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.977e-06 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.584314+0000", 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0350000000000002e-06 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594858+0000", 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010543534 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.674 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.594864+0000", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7379999999999997e-06 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654679+0000", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.059815204 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654782+0000", 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102858 2024-03-26T04:04:04.675 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.654912+0000", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013006199999999999 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.658857+0000", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039442169999999999 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.664064+0000", 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0052070179999999999 2024-03-26T04:04:04.676 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.669709+0000", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0056449380000000004 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.669772+0000", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3853e-05 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.669902+0000", 2024-03-26T04:04:04.677 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000129166 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:04:04.678 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:04:04.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:04.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:04.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:04.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:04.849 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[37501]: 6.6 scrub starts 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[37501]: 6.6 scrub ok 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[37501]: 4.a deep-scrub starts 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[37501]: 4.a deep-scrub ok 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[29446]: 6.6 scrub starts 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[29446]: 6.6 scrub ok 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[29446]: 4.a deep-scrub starts 2024-03-26T04:04:04.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:04 smithi006 ceph-mon[29446]: 4.a deep-scrub ok 2024-03-26T04:04:04.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:04.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:04.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:04.989 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:04.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:05.085 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:04 smithi086 ceph-mon[32965]: 6.6 scrub starts 2024-03-26T04:04:05.085 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:04 smithi086 ceph-mon[32965]: 6.6 scrub ok 2024-03-26T04:04:05.085 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:04 smithi086 ceph-mon[32965]: 4.a deep-scrub starts 2024-03-26T04:04:05.085 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:04 smithi086 ceph-mon[32965]: 4.a deep-scrub ok 2024-03-26T04:04:05.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:05.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:05.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:05.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:05.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T04:04:05.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:05.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:05.372 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 7 16 24471 24455 218.321 190.25 0.00581149 0.00457615 2024-03-26T04:04:05.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:05.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:05.500 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:05.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:05.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:05.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:05.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:05.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:05.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:05.853 INFO:teuthology.orchestra.run.smithi006.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-26T04:04:05.879 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:05.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:05.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:05.997 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:05 smithi006 ceph-mon[37501]: pgmap v722: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:05.998 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:05 smithi006 ceph-mon[29446]: pgmap v722: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:06.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:05 smithi086 ceph-mon[32965]: pgmap v722: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 443 MiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 852 B/s rd, 74 MiB/s wr, 1.18k op/s; 3286/21909 objects degraded (14.998%) 2024-03-26T04:04:06.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:06.131 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:04:06.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:06.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:06.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:04:06.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:06.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.371 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:06.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 8 16 27116 27100 211.676 165.312 0.00880299 0.0047166 2024-03-26T04:04:06.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:06.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:06.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:06.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:06.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:06.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:06.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:06.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:06.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[37501]: 4.1e deep-scrub starts 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[37501]: 4.1e deep-scrub ok 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[29446]: 4.1e deep-scrub starts 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[29446]: 4.1e deep-scrub ok 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T04:04:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:06 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T04:04:06.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:06.922 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:06.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:07.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:07.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:07.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:06 smithi086 ceph-mon[32965]: 4.1e deep-scrub starts 2024-03-26T04:04:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:06 smithi086 ceph-mon[32965]: 4.1e deep-scrub ok 2024-03-26T04:04:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:06 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T04:04:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:06 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T04:04:07.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:07.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:07.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:07.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:07.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:07.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 9 16 29717 29701 206.218 162.562 0.00174547 0.00484108 2024-03-26T04:04:07.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:07.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:07.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:07.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:07.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:07.748 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:07.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:07.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:07.887 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:07.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:07.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 5.1 deep-scrub starts 2024-03-26T04:04:07.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 5.1 deep-scrub ok 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: pgmap v723: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 743 B/s rd, 209 MiB/s wr, 3.35k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 11669/71667 objects degraded (16.282%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 1.3 scrub starts 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[37501]: 1.3 scrub ok 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 5.1 deep-scrub starts 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 5.1 deep-scrub ok 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: pgmap v723: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 743 B/s rd, 209 MiB/s wr, 3.35k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T04:04:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T04:04:07.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 11669/71667 objects degraded (16.282%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:07.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 1.3 scrub starts 2024-03-26T04:04:07.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:07 smithi006 ceph-mon[29446]: 1.3 scrub ok 2024-03-26T04:04:07.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:08.035 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:08.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 5.1 deep-scrub starts 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 5.1 deep-scrub ok 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: pgmap v723: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 743 B/s rd, 209 MiB/s wr, 3.35k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 11669/71667 objects degraded (16.282%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 1.3 scrub starts 2024-03-26T04:04:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:07 smithi086 ceph-mon[32965]: 1.3 scrub ok 2024-03-26T04:04:08.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:08.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:08.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.312 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:08.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:08.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 10 15 31971 31956 199.689 140.938 0.00419401 0.00500292 2024-03-26T04:04:08.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:08.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:08.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:08.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:08.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:08.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:08.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:08.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:08.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:08.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:08.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:08.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:08.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:08.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:08.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:09.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:08 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T04:04:09.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:08 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T04:04:09.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:09.120 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:09.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:09.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:08 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T04:04:09.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:08 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T04:04:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:08 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T04:04:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:08 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T04:04:09.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:09.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:09.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:09.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:09.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 11 16 34004 33988 193.081 127 0.00636996 0.00517432 2024-03-26T04:04:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:09.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:09.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:09.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:09.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:09.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:09.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:09.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:09.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:09.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:09.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:09.942 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:09.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:09.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:10.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:10.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:10.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:09 smithi086 ceph-mon[32965]: pgmap v724: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 639 B/s rd, 166 MiB/s wr, 2.66k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:09 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T04:04:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:09 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T04:04:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:09 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:09 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[37501]: pgmap v724: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 639 B/s rd, 166 MiB/s wr, 2.66k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[29446]: pgmap v724: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 1.4 GiB data, 6.6 GiB used, 440 GiB / 447 GiB avail; 639 B/s rd, 166 MiB/s wr, 2.66k op/s; 11669/71667 objects degraded (16.282%) 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T04:04:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T04:04:10.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:10.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:09 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:10.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:10.206 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:10.308 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:10.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.345 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:10.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:10.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 12 16 36122 36106 188.021 132.375 0.00543673 0.00531372 2024-03-26T04:04:10.446 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:10.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:10.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:10.620 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:10.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:10.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:10.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:10.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:10.887 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:10.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:10.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:10.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.013 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:10 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T04:04:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:10 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T04:04:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:10 smithi086 ceph-mon[32965]: 5.16 deep-scrub starts 2024-03-26T04:04:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:10 smithi086 ceph-mon[32965]: 5.16 deep-scrub ok 2024-03-26T04:04:11.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[37501]: 5.16 deep-scrub starts 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[37501]: 5.16 deep-scrub ok 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[29446]: 5.16 deep-scrub starts 2024-03-26T04:04:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:10 smithi006 ceph-mon[29446]: 5.16 deep-scrub ok 2024-03-26T04:04:11.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:11.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:11.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:11.373 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 13 16 38135 38119 183.235 125.812 0.00348289 0.00544628 2024-03-26T04:04:11.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:11.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:11.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:11.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:11.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:11.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:11.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:11.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:11.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:11.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:12.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:12.081 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[37501]: pgmap v725: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 1.8 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.55k op/s; 15559/90267 objects degraded (17.237%) 2024-03-26T04:04:12.081 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:04:12.081 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:04:12.081 INFO:teuthology.orchestra.run.smithi006.stdout:get_pg_num_history pg_num_history pg_num_history(e335 pg_nums {1={19=1,271=11},2={64=3},3={65=32},4={67=32},5={69=32},6={71=32},8={155=16},9={253=16}} deleted_pools 251,8) 2024-03-26T04:04:12.081 INFO:teuthology.orchestra.run.smithi006.stdout:Importing pgid 4.13 2024-03-26T04:04:12.081 INFO:teuthology.orchestra.run.smithi006.stdout:Write #4:c8333aab:gc::gc.11:head# 2024-03-26T04:04:12.081 INFO:teuthology.orchestra.run.smithi006.stdout:snapset 0=[]:{} 2024-03-26T04:04:12.081 INFO:teuthology.orchestra.run.smithi006.stdout:Write #4:cba150de:::obj_delete_at_hint.0000000051:head# 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:snapset 0=[]:{} 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:Write #4:cc909b6b:::obj_delete_at_hint.0000000106:head# 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:snapset 0=[]:{} 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:Write #4:cd178ae0:reshard::reshard.0000000014:head# 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:snapset 0=[]:{} 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:Write #4:cedbb205:::obj_delete_at_hint.0000000017:head# 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:snapset 0=[]:{} 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:write_pg epoch 323 info 4.13( v 241'19 (0'0,241'19] local-lis/les=293/294 n=5 ec=67/67 lis/c=293/293 les/c/f=294/294/0 sis=293) 2024-03-26T04:04:12.082 INFO:teuthology.orchestra.run.smithi006.stdout:Import successful 2024-03-26T04:04:12.082 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[29446]: pgmap v725: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 1.8 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.55k op/s; 15559/90267 objects degraded (17.237%) 2024-03-26T04:04:12.082 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:04:12.082 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:11 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:04:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:11 smithi086 ceph-mon[32965]: pgmap v725: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 1.8 GiB data, 7.8 GiB used, 439 GiB / 447 GiB avail; 1.0 KiB/s rd, 159 MiB/s wr, 2.55k op/s; 15559/90267 objects degraded (17.237%) 2024-03-26T04:04:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:11 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:04:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:11 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:04:12.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:12.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:12.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:12.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:12.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:12.331 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:12.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.354 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:12.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:12.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 14 16 39878 39862 177.928 108.938 0.00196552 0.00558051 2024-03-26T04:04:12.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:12.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:12.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:12.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:12.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:12.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.758 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:12.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:12.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:12.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:12.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:12.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:12.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:13.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: 1.2 scrub starts 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: 1.2 scrub ok 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: osdmap e357: 8 total, 7 up, 5 in 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T04:04:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:12 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T04:04:13.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:13.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: 1.2 scrub starts 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: 1.2 scrub ok 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: osdmap e357: 8 total, 7 up, 5 in 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T04:04:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: 1.2 scrub starts 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: 1.2 scrub ok 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: osdmap e357: 8 total, 7 up, 5 in 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T04:04:13.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:12 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T04:04:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:13.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:13.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:13.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:13.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 15 16 41546 41530 173.016 104.25 0.0073566 0.00577558 2024-03-26T04:04:13.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:13.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:13.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:13.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:13.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:13.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:13.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:13.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.832 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:13.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:13.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:13.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:13.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:14.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:13 smithi086 ceph-mon[32965]: pgmap v727: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 185 MiB/s wr, 2.96k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:13 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 17763/107571 objects degraded (16.513%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:14.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:13 smithi006 ceph-mon[37501]: pgmap v727: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 185 MiB/s wr, 2.96k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:13 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 17763/107571 objects degraded (16.513%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:13 smithi006 ceph-mon[29446]: pgmap v727: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 185 MiB/s wr, 2.96k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:13 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 17763/107571 objects degraded (16.513%), 31 pgs degraded (PG_DEGRADED) 2024-03-26T04:04:14.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:14.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:14.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:14.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:14.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 16 16 43352 43336 169.256 112.875 0.00488805 0.00590287 2024-03-26T04:04:14.403 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:14.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:14.541 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:14.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:14.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:14.673 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.674 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:14.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:14.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:14.901 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:14.929 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:14.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[37501]: osdmap e358: 8 total, 7 up, 5 in 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:04:15.031 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:14 smithi006 ceph-mon[29446]: osdmap e358: 8 total, 7 up, 5 in 2024-03-26T04:04:15.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:15.067 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:15.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:14 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:04:15.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:14 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:04:15.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:14 smithi086 ceph-mon[32965]: osdmap e358: 8 total, 7 up, 5 in 2024-03-26T04:04:15.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:15.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:15.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:15.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 17 16 44804 44788 164.638 90.75 0.0041109 0.00606916 2024-03-26T04:04:15.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:15.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:15.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:15.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:15.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.626 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:15.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:15.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:15.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/exp.4.13.2 2024-03-26T04:04:15.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:15.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:15.904 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:15.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:15.951 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:15.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:16.000 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:16.003 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:16.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:16.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:15 smithi086 ceph-mon[32965]: pgmap v728: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1023 B/s rd, 178 MiB/s wr, 2.85k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:15 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T04:04:16.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:15 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T04:04:16.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:16.131 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[37501]: pgmap v728: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1023 B/s rd, 178 MiB/s wr, 2.85k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:16.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T04:04:16.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T04:04:16.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[29446]: pgmap v728: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 2.2 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1023 B/s rd, 178 MiB/s wr, 2.85k op/s; 17763/107571 objects degraded (16.513%) 2024-03-26T04:04:16.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T04:04:16.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:15 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T04:04:16.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:16.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:16.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:16.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 18 16 46960 46944 162.977 134.75 0.00288068 0.00612997 2024-03-26T04:04:16.386 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:16.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:16.522 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:16.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:16.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:16.704 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:04:16.704 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:04:16.704 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T04:04:16.704 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:04:16.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:16.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.794 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:16.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:16.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:16.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:16.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:17.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:17.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:16 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:04:17.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:16 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:04:17.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:16 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T04:04:17.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:16 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T04:04:17.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T04:04:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:16 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T04:04:17.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.208 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:17.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:17.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:17.346 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:17.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:17.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 19 16 49035 49019 161.225 129.688 0.00440329 0.00619827 2024-03-26T04:04:17.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:17.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:17.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:17.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:17.624 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:17.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:17.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:17.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:17.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:17.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:17.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:18.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:18.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:18.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:17 smithi086 ceph-mon[32965]: pgmap v730: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:17 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T04:04:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:17 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T04:04:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:17 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T04:04:18.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:17 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[37501]: pgmap v730: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[29446]: pgmap v730: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T04:04:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:17 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T04:04:18.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:18.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:18.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:18.323 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:18.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:04:18.375012+0000 min lat: 0.00114192 max lat: 0.159537 avg lat: 0.00625086 2024-03-26T04:04:18.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:04:18.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 20 16 51182 51166 159.872 134.188 0.00333439 0.00625086 2024-03-26T04:04:18.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:18.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:18.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:18.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:18.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:18.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:18.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:18.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:18.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:18.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:18.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:18.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:18.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 3.12 deep-scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 3.12 deep-scrub ok 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 3.12 deep-scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 3.12 deep-scrub ok 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T04:04:18.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:18 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T04:04:18.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:19.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:19.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 3.12 deep-scrub starts 2024-03-26T04:04:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 3.12 deep-scrub ok 2024-03-26T04:04:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T04:04:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T04:04:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T04:04:19.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:18 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T04:04:19.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:19.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:19.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:19.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:19.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 21 16 53312 53296 158.598 133.125 0.0160982 0.00629997 2024-03-26T04:04:19.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:19.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T04:04:19.422 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:19.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:19.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:19.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:19.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.720 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:19.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:19.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:19.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:19.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:19.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:20.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:20.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:20.040 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[37501]: pgmap v731: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:20.040 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T04:04:20.040 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T04:04:20.040 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 22425/134583 objects degraded (16.663%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:20.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[29446]: pgmap v731: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:20.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T04:04:20.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T04:04:20.041 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:19 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 22425/134583 objects degraded (16.663%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:19 smithi086 ceph-mon[32965]: pgmap v731: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 2.7 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 115 MiB/s wr, 1.85k op/s; 22425/134583 objects degraded (16.663%) 2024-03-26T04:04:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:19 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T04:04:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:19 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T04:04:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:19 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 22425/134583 objects degraded (16.663%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:20.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:20.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:20.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:20.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:20.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:20.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 22 16 55202 55186 156.758 118.125 0.00666073 0.00637511 2024-03-26T04:04:20.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:20.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:20.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:20.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:20.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:20.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:20.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:20.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:20.804 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:04:20.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:20.826 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7] out_osds: [6, 5, 3] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2024-03-26T04:04:20.826 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:04:20.826 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-26T04:04:20.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-26T04:04:20.844 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:20.869 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:04:20.884 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:20.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:20.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:20.916 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:04:20.964 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:04:20.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:21.007 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:04:21.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:21.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[37501]: 4.3 deep-scrub starts 2024-03-26T04:04:21.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[37501]: 4.3 deep-scrub ok 2024-03-26T04:04:21.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T04:04:21.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T04:04:21.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/435007974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:21.054 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:04:21.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[29446]: 4.3 deep-scrub starts 2024-03-26T04:04:21.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[29446]: 4.3 deep-scrub ok 2024-03-26T04:04:21.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T04:04:21.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T04:04:21.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/435007974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:20 smithi086 ceph-mon[32965]: 4.3 deep-scrub starts 2024-03-26T04:04:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:20 smithi086 ceph-mon[32965]: 4.3 deep-scrub ok 2024-03-26T04:04:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:20 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T04:04:21.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:20 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T04:04:21.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/435007974' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:04:21.094 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:04:21.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:21.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:21.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:21.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:21.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 23 16 57016 57000 154.871 113.375 0.0201999 0.00645081 2024-03-26T04:04:21.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:21.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:21.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:21.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:21.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:21.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:21.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:21.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:21.827 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:04:21.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:21.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:21.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:21.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:21.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:21.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:22.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: pgmap v732: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 25534/149604 objects degraded (17.068%) 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: osdmap e359: 8 total, 7 up, 5 in 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T04:04:22.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T04:04:22.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:21 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T04:04:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.120 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:22.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: pgmap v732: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 25534/149604 objects degraded (17.068%) 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: osdmap e359: 8 total, 7 up, 5 in 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T04:04:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: pgmap v732: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.0 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s; 25534/149604 objects degraded (17.068%) 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: osdmap e359: 8 total, 7 up, 5 in 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T04:04:22.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:21 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T04:04:22.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:22.259 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:22.361 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:22.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 24 16 59088 59072 153.813 129.5 0.00858795 0.00649711 2024-03-26T04:04:22.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.401 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:22.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:22.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:22.539 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:22.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:22.677 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:22.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:22.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:22.818 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:22.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:22.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:22.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:23.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:22 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:04:23.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:22 smithi086 ceph-mon[32965]: osdmap e360: 8 total, 7 up, 5 in 2024-03-26T04:04:23.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:23.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:23.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:04:23.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[37501]: osdmap e360: 8 total, 7 up, 5 in 2024-03-26T04:04:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2412875455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:04:23.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:22 smithi006 ceph-mon[29446]: osdmap e360: 8 total, 7 up, 5 in 2024-03-26T04:04:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:04:23.196 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:04:23.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:23.230 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:23.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:23.374 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 25 16 61213 61197 152.973 132.812 0.0078926 0.0065327 2024-03-26T04:04:23.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:23.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:23.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:23.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:23.778 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:23.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:23.921 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:23.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:24.023 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:24.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:23 smithi086 ceph-mon[32965]: pgmap v735: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:24.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:23 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:23 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:24.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:23 smithi086 ceph-mon[32965]: osdmap e361: 8 total, 7 up, 5 in 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[37501]: pgmap v735: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[37501]: osdmap e361: 8 total, 7 up, 5 in 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[29446]: pgmap v735: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:23 smithi006 ceph-mon[29446]: osdmap e361: 8 total, 7 up, 5 in 2024-03-26T04:04:24.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:24.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:24.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:24.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:24.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:24.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 26 16 63353 63337 152.233 133.75 0.00788646 0.00656399 2024-03-26T04:04:24.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:24.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:24.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:24.612 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:24.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:24.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:24.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:24.802 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:24.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:24.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:24.885 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:24.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:24.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:25.025 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:25.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:25.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 27451/164931 objects degraded (16.644%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:25.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:24 smithi086 ceph-mon[32965]: osdmap e362: 8 total, 7 up, 5 in 2024-03-26T04:04:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:25.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 27451/164931 objects degraded (16.644%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[29446]: osdmap e362: 8 total, 7 up, 5 in 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 27451/164931 objects degraded (16.644%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[37501]: osdmap e362: 8 total, 7 up, 5 in 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:25.132 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.172 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:25.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:25.197 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:04:25.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:25.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:25.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:25.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 27 16 65407 65391 151.349 128.375 0.00513642 0.0066027 2024-03-26T04:04:25.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:25.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:25.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:25.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:25.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:25.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:25.685 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:25.723 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:25.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:25.825 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:25.869 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:25.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:25.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:25.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:26.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:26.011 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:04:26.011 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:04:26.011 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:25 smithi006 ceph-mon[37501]: pgmap v737: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.69k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:26.012 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:25 smithi006 ceph-mon[29446]: pgmap v737: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.69k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:26.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.144 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:26.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:04:26.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:26.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:25 smithi086 ceph-mon[32965]: pgmap v737: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 3.3 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.69k op/s; 27451/164931 objects degraded (16.644%) 2024-03-26T04:04:26.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 28 16 67324 67308 150.222 119.812 0.00771637 0.00665223 2024-03-26T04:04:26.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:26.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:26.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:26.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:26.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:26.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:26.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:26.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T04:04:26.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:26.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:26.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:26.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:26.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:27.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:27.101 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:27.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:27.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:27.203 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.249 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:27.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:27.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:27.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:27.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 29 16 69264 69248 149.222 121.25 0.010311 0.00669619 2024-03-26T04:04:27.398 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:27.416 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:27.428 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:27.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:27.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:27.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:27.674 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:27.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:27.804 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:27.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:27.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:27.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:27.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:28.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:28.078 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:28.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:27 smithi006 ceph-mon[29446]: pgmap v739: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 921 B/s rd, 136 MiB/s wr, 2.17k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:27 smithi006 ceph-mon[37501]: pgmap v739: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 921 B/s rd, 136 MiB/s wr, 2.17k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:28.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:28.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:28.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:28.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:27 smithi086 ceph-mon[32965]: pgmap v739: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 921 B/s rd, 136 MiB/s wr, 2.17k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:28.317 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:28.352 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:28.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 30 16 71224 71208 148.331 122.5 0.00327973 0.00673635 2024-03-26T04:04:28.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:28.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:28.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:28.632 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:28.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:28.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:28.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:28.866 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:28.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:28.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:28.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:28.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:28.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:29.037 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:29.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:29.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:29.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:29.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:29.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 31 15 73007 72992 147.143 111.5 0.0148742 0.00679173 2024-03-26T04:04:29.417 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:29.453 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:29.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:29.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:29.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:29.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:29.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:29.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:29.874 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[29446]: pgmap v740: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 782 B/s rd, 97 MiB/s wr, 1.54k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:29.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:29.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:30.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:30.127 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:30.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[37501]: pgmap v740: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 782 B/s rd, 97 MiB/s wr, 1.54k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 32502/195243 objects degraded (16.647%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 32502/195243 objects degraded (16.647%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:30.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:29 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:30.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:30.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:30.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.310 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:30.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:29 smithi086 ceph-mon[32965]: pgmap v740: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 782 B/s rd, 97 MiB/s wr, 1.54k op/s; 32502/195243 objects degraded (16.647%) 2024-03-26T04:04:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 32502/195243 objects degraded (16.647%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:29 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:29 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:30.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 32 16 75015 74999 146.464 125.438 0.00773737 0.00682341 2024-03-26T04:04:30.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.448 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:30.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:30.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:30.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:30.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:30.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:30.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:30.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:30.857 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:30.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:30.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:30.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:30.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:30.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:31.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:31.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:31.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:31.275 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:31.376 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 33 16 77123 77107 146.018 131.75 0.00334336 0.00684445 2024-03-26T04:04:31.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:31.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:31.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:31.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:31.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:31.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:31.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:31 smithi006 ceph-mon[37501]: pgmap v741: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 35545/209832 objects degraded (16.940%) 2024-03-26T04:04:31.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:31 smithi006 ceph-mon[37501]: osdmap e363: 8 total, 7 up, 5 in 2024-03-26T04:04:31.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:31 smithi006 ceph-mon[29446]: pgmap v741: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 35545/209832 objects degraded (16.940%) 2024-03-26T04:04:31.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:31 smithi006 ceph-mon[29446]: osdmap e363: 8 total, 7 up, 5 in 2024-03-26T04:04:31.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:31.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:31.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:31.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:31.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.831 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:31.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:31 smithi086 ceph-mon[32965]: pgmap v741: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.3 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s; 35545/209832 objects degraded (16.940%) 2024-03-26T04:04:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:31 smithi086 ceph-mon[32965]: osdmap e363: 8 total, 7 up, 5 in 2024-03-26T04:04:31.933 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:31.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:31.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:32.073 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:32.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:32.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:32.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:32.254 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:32.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:32.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:32.375 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 34 16 79223 79207 145.584 131.25 0.0106923 0.00686484 2024-03-26T04:04:32.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:32.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:32.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:32.632 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:32.668 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:32.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:32.744 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:32.745 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[37501]: osdmap e364: 8 total, 7 up, 5 in 2024-03-26T04:04:32.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:32.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:32.745 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:32 smithi006 ceph-mon[29446]: osdmap e364: 8 total, 7 up, 5 in 2024-03-26T04:04:32.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.804 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:32.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:32 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:32 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:32.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:32 smithi086 ceph-mon[32965]: osdmap e364: 8 total, 7 up, 5 in 2024-03-26T04:04:32.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:32.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:32.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:32.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:33.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:33.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:33.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:33.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.209 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:33.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:33.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:33.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:33.376 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 35 16 81273 81257 145.084 128.125 0.00382247 0.00688794 2024-03-26T04:04:33.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:33.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:33.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:33.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:33.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:33.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:33.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:33.759 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:33.804 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:04:33.804 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:04:33.860 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:33.898 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:33.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:34.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:34.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:34.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[37501]: pgmap v744: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.52k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:34.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:04:34.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:04:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[29446]: pgmap v744: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.52k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:04:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:33 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:04:34.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:34.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:34.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:34.315 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:33 smithi086 ceph-mon[32965]: pgmap v744: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.3 KiB/s rd, 157 MiB/s wr, 2.52k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:33 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:04:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:33 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:04:34.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-26T04:04:34.380 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 36 16 83341 83325 144.644 129.25 0.00442233 0.00690914 2024-03-26T04:04:34.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:34.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:34.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:34.591 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:34.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:34.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:34.693 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:34.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:34.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:34.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:34.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:34.929 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[37501]: osdmap e365: 8 total, 7 up, 5 in 2024-03-26T04:04:34.929 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:04:34.929 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 37352/224478 objects degraded (16.639%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:34.929 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[37501]: osdmap e366: 8 total, 7 up, 5 in 2024-03-26T04:04:34.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[29446]: osdmap e365: 8 total, 7 up, 5 in 2024-03-26T04:04:34.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:04:34.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 37352/224478 objects degraded (16.639%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:34.930 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:34 smithi006 ceph-mon[29446]: osdmap e366: 8 total, 7 up, 5 in 2024-03-26T04:04:34.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:35.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:35.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:35.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:35.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:35.302 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:35.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:34 smithi086 ceph-mon[32965]: osdmap e365: 8 total, 7 up, 5 in 2024-03-26T04:04:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:34 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:04:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 37352/224478 objects degraded (16.639%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:35.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:34 smithi086 ceph-mon[32965]: osdmap e366: 8 total, 7 up, 5 in 2024-03-26T04:04:35.376 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 37 16 84992 84976 143.521 103.188 0.00611308 0.0069621 2024-03-26T04:04:35.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:35.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:35.481 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:04:35.525 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:04:35.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:35.582 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:04:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:35.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:35.624 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:04:35.664 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:04:35.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:35.707 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:04:35.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:35.748 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:04:35.748 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:04:35.748 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:04:35.749 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:04:35.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:35.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:35.934 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:04:35.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:35.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T04:04:35.981 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:35.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:35.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:36.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:36.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:36.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:36.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:36.269 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:35 smithi086 ceph-mon[32965]: pgmap v746: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.62k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:35 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2750925155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:35 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3946047254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:35 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:36.376 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 38 16 86777 86761 142.68 111.562 0.00652949 0.00700461 2024-03-26T04:04:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[37501]: pgmap v746: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.62k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2750925155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3946047254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[29446]: pgmap v746: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 4.6 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.62k op/s; 37352/224478 objects degraded (16.639%) 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2750925155' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3946047254' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:35 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:36.391 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:36.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:36.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:36.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:36.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:36.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:36.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:36.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:36.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.819 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:36.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:36.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:36.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:37.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:37.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.095 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:37.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:37.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[37501]: osdmap e367: 8 total, 7 up, 5 in 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:37.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:36 smithi006 ceph-mon[29446]: osdmap e367: 8 total, 7 up, 5 in 2024-03-26T04:04:37.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.229 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:37.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:37.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:37.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:36 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:37.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:36 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:37.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:36 smithi086 ceph-mon[32965]: osdmap e367: 8 total, 7 up, 5 in 2024-03-26T04:04:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:37.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 39 16 88380 88364 141.59 100.188 0.00324479 0.00705844 2024-03-26T04:04:37.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.507 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:37.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:37.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.649 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:37.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:37.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.779 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:37.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:37.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:37.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:37.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:38.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:38.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:38.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:38.175 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:38.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:38.224 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:37 smithi006 ceph-mon[37501]: pgmap v749: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:38.224 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:37 smithi006 ceph-mon[29446]: pgmap v749: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:38.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:38.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:37 smithi086 ceph-mon[32965]: pgmap v749: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 140 MiB/s wr, 2.25k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:38.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:04:38.377638+0000 min lat: 0.00114192 max lat: 0.159537 avg lat: 0.0071151 2024-03-26T04:04:38.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:04:38.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 40 16 89922 89906 140.459 96.375 0.00867224 0.0071151 2024-03-26T04:04:38.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:38.481 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:38.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:38.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:38.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:38.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:38.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:38.991 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:38.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:38.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:39.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:39.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:39.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:04:39.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:04:39.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:04:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:04:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:38 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:39.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:39.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:04:39.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:04:39.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:04:39.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:39.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 41 16 91576 91560 139.555 103.375 0.00894898 0.00715985 2024-03-26T04:04:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:39.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:39.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:39.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:39.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:39.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:39.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:39.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:39.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:39.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:40.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:40.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:40.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:40.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:40.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.337 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:40.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: pgmap v750: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: pgmap v751: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: pgmap v752: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 144 MiB/s wr, 2.30k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 42367/254712 objects degraded (16.633%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: osdmap e368: 8 total, 7 up, 5 in 2024-03-26T04:04:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:40.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 42 16 93015 92999 138.373 89.9375 0.00306263 0.00721745 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: pgmap v750: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: pgmap v751: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: pgmap v752: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 144 MiB/s wr, 2.30k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 42367/254712 objects degraded (16.633%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: osdmap e368: 8 total, 7 up, 5 in 2024-03-26T04:04:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: pgmap v750: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 852 B/s rd, 105 MiB/s wr, 1.68k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: pgmap v751: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 130 MiB/s wr, 2.08k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: pgmap v752: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.1 KiB/s rd, 144 MiB/s wr, 2.30k op/s; 42367/254712 objects degraded (16.633%) 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 42367/254712 objects degraded (16.633%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: osdmap e368: 8 total, 7 up, 5 in 2024-03-26T04:04:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:40.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:40.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:40.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:40.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:40.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:40.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:40.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:40.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:40.854 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:40.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:40.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:40.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:40.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:41.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:41.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:41.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:04:41.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:04:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:41.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:41.314 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:41.378 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 43 15 94918 94903 137.922 119 0.00871692 0.00724622 2024-03-26T04:04:41.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:41.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:41.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:41.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:41.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:41.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:41.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:41.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:04:41.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:41.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:41.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:41.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:96041 10.f 10:f45c696b:::benchmark_data_smithi006_159841_object96040:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:04:41.991132+0000", 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0070944429999999998, 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0071202770000000004, 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:42.000 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 96041 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991132+0000", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:42.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991132+0000", 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991133+0000", 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7000000000000004e-07 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991180+0000", 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7410999999999997e-05 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991190+0000", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0417999999999999e-05 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991196+0000", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6319999999999996e-06 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991265+0000", 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9257000000000005e-05 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991392+0000", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012677200000000001 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991486+0000", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4433999999999997e-05 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995920+0000", 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044332299999999998 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.004 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:96046 10.7 10:ec2d00f4:::benchmark_data_smithi006_159841_object96045:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:04:41.991396+0000", 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0068301730000000001, 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0069119890000000003, 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:04:42.005 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 96046 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991396+0000", 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991396+0000", 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991397+0000", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0400000000000005e-07 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991471+0000", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3628999999999996e-05 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991482+0000", 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1242e-05 2024-03-26T04:04:42.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991491+0000", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7709999999999995e-06 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.993427+0000", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00193625 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.993563+0000", 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013609900000000001 2024-03-26T04:04:42.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.993703+0000", 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000140168 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995926+0000", 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022230710000000001 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:96047 10.d 10:b26588a5:::benchmark_data_smithi006_159841_object96046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:04:41.991516+0000", 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0067102059999999998, 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0068422600000000002, 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 96047 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:42.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991516+0000", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991516+0000", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991517+0000", 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2799999999999996e-07 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991587+0000", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0685000000000005e-05 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991596+0000", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3860000000000007e-06 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991607+0000", 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0988999999999999e-05 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991669+0000", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1836999999999998e-05 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991831+0000", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016224999999999999 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.991996+0000", 2024-03-26T04:04:42.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016494400000000001 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995789+0000", 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0037937230000000001 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995805+0000", 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5367999999999999e-05 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:96051 10.3 10:c301e278:::benchmark_data_smithi006_159841_object96050:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e368)", 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:04:41.995416+0000", 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0028100590000000002, 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029936469999999999, 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 96051 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.015 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995416+0000", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995416+0000", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995418+0000", 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.46e-06 2024-03-26T04:04:42.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995456+0000", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8389999999999997e-05 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995468+0000", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1731e-05 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995478+0000", 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0332e-05 2024-03-26T04:04:42.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.995839+0000", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00036108800000000002 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.996001+0000", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016174700000000001 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.996184+0000", 2024-03-26T04:04:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018291999999999999 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15540.0:96052 10.b e368/349 10:dd1f7e74:::benchmark_data_smithi006_159841_object96051:head v 368'5974, mlcod=368'5971)", 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:04:41.997453+0000", 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.00077288699999999999, 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00099863899999999995, 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:42.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997453+0000", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997453+0000", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997454+0000", 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1900000000000003e-07 2024-03-26T04:04:42.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997517+0000", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3082000000000004e-05 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997525+0000", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4449999999999996e-06 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997532+0000", 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5899999999999996e-06 2024-03-26T04:04:42.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997566+0000", 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3864e-05 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:04:41.997578+0000", 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2354999999999999e-05 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T04:04:42.022 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 5 2024-03-26T04:04:42.023 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:04:42.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.051 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:42.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:42.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:41 smithi086 ceph-mon[32965]: pgmap v754: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 44982/267078 objects degraded (16.842%) 2024-03-26T04:04:42.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:42.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:42.258 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:41 smithi006 ceph-mon[37501]: pgmap v754: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 44982/267078 objects degraded (16.842%) 2024-03-26T04:04:42.259 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:41 smithi006 ceph-mon[29446]: pgmap v754: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.4 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 854 op/s; 44982/267078 objects degraded (16.842%) 2024-03-26T04:04:42.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:42.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.326 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:42.338 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:42.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 44 16 96644 96628 137.238 107.812 0.00729012 0.00728212 2024-03-26T04:04:42.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:42.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:42.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:42.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:42.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:42.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:42.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:42.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:42.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.858 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:42.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:42.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:42.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:43.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:43.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.133 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:43.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:43.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:04:43.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:43.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.275 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:43.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:43.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:43.377 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 45 15 98212 98197 136.367 98.0625 0.00950215 0.00732875 2024-03-26T04:04:43.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T04:04:43.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:43.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:43.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:43.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:43.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:43.699 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:43.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-26T04:04:43.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:43.830 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:43.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:43.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:43.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:44.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:44.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:44.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:44.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:43 smithi006 ceph-mon[37501]: pgmap v755: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 802 B/s rd, 79 MiB/s wr, 1.27k op/s; 46473/278988 objects degraded (16.658%) 2024-03-26T04:04:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:43 smithi006 ceph-mon[29446]: pgmap v755: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 802 B/s rd, 79 MiB/s wr, 1.27k op/s; 46473/278988 objects degraded (16.658%) 2024-03-26T04:04:44.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.162 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:44.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:44.234 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:43 smithi086 ceph-mon[32965]: pgmap v755: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.7 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 802 B/s rd, 79 MiB/s wr, 1.27k op/s; 46473/278988 objects degraded (16.658%) 2024-03-26T04:04:44.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:44.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:44.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:44.318 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:44.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 46 16 99987 99971 135.81 110.875 0.00733267 0.0073591 2024-03-26T04:04:44.402 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.445 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:44.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:44.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:44.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:44.622 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:44.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:44.728 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:44.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:44.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:44.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:44.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:45.231 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 46473/278988 objects degraded (16.658%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:45.232 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:45.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:45.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:45.378 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 47 16 101898 101882 135.461 119.438 0.00604504 0.00737823 2024-03-26T04:04:45.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 46473/278988 objects degraded (16.658%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:45.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 46473/278988 objects degraded (16.658%), 31 pgs degraded, 74 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:45.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:45.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:45.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:45.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:45.701 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:45.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:45.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:45.966 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:45.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:45.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:46.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:46.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:46.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:46.243 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:46.289 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:45 smithi086 ceph-mon[32965]: pgmap v756: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 49200/291837 objects degraded (16.859%) 2024-03-26T04:04:46.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:46.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:46.378 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 48 16 103448 103432 134.657 96.875 0.00265288 0.0074155 2024-03-26T04:04:46.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:45 smithi006 ceph-mon[29446]: pgmap v756: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 49200/291837 objects degraded (16.859%) 2024-03-26T04:04:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:45 smithi006 ceph-mon[37501]: pgmap v756: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 5.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 49200/291837 objects degraded (16.859%) 2024-03-26T04:04:46.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:46.510 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:46.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:46.652 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:46.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:46.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.789 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:46.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:46.794 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:04:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:04:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:04:46.794 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:04:46.794 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:04:46.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:46.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:46.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:47.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:47.060 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:47.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:47.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:47.200 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:47.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:47.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:47.366 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:47.366 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:47.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 49 16 105171 105155 134.106 107.688 0.00472813 0.00745284 2024-03-26T04:04:47.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:47.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:47.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:47.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:47.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T04:04:47.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:47.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:47.878 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:47.913 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:47.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:47.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:47 smithi086 ceph-mon[32965]: pgmap v757: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:47.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4135385922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:47.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:48.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:48.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:48.087 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:48.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:48.207 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:48.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:48.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:48.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:48.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 50 16 106970 106954 133.673 112.438 0.00705155 0.00747713 2024-03-26T04:04:48.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[37501]: pgmap v757: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4135385922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[29446]: pgmap v757: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.10k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4135385922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:04:48.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:48.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:48.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:48.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:48.623 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:48.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:48.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:48.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:48.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:48.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:48.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:48.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:49.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:49.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:49.121 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:04:49.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:49.176 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:49.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:49.301 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:49.322 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:49 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:49 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:04:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:49 smithi086 ceph-mon[32965]: osdmap e369: 8 total, 7 up, 5 in 2024-03-26T04:04:49.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 51 15 109035 109020 133.583 129.125 0.00459554 0.00748192 2024-03-26T04:04:49.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:49.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:04:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[37501]: osdmap e369: 8 total, 7 up, 5 in 2024-03-26T04:04:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:04:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:04:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:49 smithi006 ceph-mon[29446]: osdmap e369: 8 total, 7 up, 5 in 2024-03-26T04:04:49.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:49.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:49.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:49.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:49.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:49.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:49.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:49.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:49.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:50.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148363 8.2 8:4631d34b:::benchmark_data_smithi006_103169_object18432:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e195)", 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.003362+0000", 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.13286549700001, 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0286756239999999, 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.141 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148363 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003362+0000", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003362+0000", 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003363+0000", 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0130000000000001e-06 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003369+0000", 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7370000000000003e-06 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003377+0000", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1969999999999996e-06 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003384+0000", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.153e-06 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003437+0000", 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3012e-05 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003545+0000", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000107771 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.003655+0000", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010990899999999999 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.008136+0000", 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044816379999999996 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.008149+0000", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2625e-05 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.031951+0000", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0238021239999999 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.032037+0000", 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6445000000000003e-05 2024-03-26T04:04:50.146 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148376 8.2 8:4e896291:::benchmark_data_smithi006_103169_object18445:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.022619+0000", 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.11360875499997, 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0164433390000001, 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.147 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148376 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022619+0000", 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022619+0000", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022619+0000", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6700000000000003e-07 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022622+0000", 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8009999999999999e-06 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022628+0000", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1210000000000003e-06 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022635+0000", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4790000000000001e-06 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022692+0000", 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7658999999999998e-05 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.022703+0000", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0326000000000001e-05 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.032430+0000", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0097278750000001 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.032546+0000", 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011597800000000001 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.032629+0000", 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2426e-05 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.036931+0000", 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043020790000000003 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.036978+0000", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7571000000000002e-05 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.038985+0000", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020065819999999998 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.039017+0000", 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2373999999999999e-05 2024-03-26T04:04:50.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.039062+0000", 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4501e-05 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.154 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148421 8.2 8:4e7d6719:::benchmark_data_smithi006_103169_object18490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.072784+0000", 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.06344288899999, 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96881163199999998, 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.158 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148421 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072784+0000", 2024-03-26T04:04:50.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072784+0000", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072785+0000", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.44e-07 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072789+0000", 2024-03-26T04:04:50.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8449999999999996e-06 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072798+0000", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8950000000000004e-06 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072806+0000", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8290000000000008e-06 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072871+0000", 2024-03-26T04:04:50.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4966000000000001e-05 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.072881+0000", 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0427e-05 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.034687+0000", 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96180558500000002 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.034820+0000", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013307799999999999 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.034957+0000", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013709799999999999 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041342+0000", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0063849220000000003 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041362+0000", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0131000000000001e-05 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041481+0000", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011863500000000001 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041543+0000", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2255999999999995e-05 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041596+0000", 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3321e-05 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148455 8.2 8:47d19f63:::benchmark_data_smithi006_103169_object18524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.110190+0000", 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.02603785899998, 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93553252799999997, 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.165 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148455 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110190+0000", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110190+0000", 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110191+0000", 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.088e-06 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110196+0000", 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1590000000000001e-06 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110208+0000", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1864e-05 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110214+0000", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4019999999999999e-06 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110263+0000", 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8925999999999999e-05 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.110272+0000", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.653e-06 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.037009+0000", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92673743200000003 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.037123+0000", 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011361 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.037260+0000", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013726600000000001 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041368+0000", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041078260000000002 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041423+0000", 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5183999999999999e-05 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045643+0000", 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042197600000000004 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045685+0000", 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2750999999999998e-05 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045722+0000", 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6606999999999997e-05 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148460 8.2 8:4b3cc241:::benchmark_data_smithi006_103169_object18529:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.114724+0000", 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.02150320200002, 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93798837999999995, 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.174 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148460 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114724+0000", 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114724+0000", 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114725+0000", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3200000000000004e-07 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114729+0000", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0099999999999997e-06 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114738+0000", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8489999999999995e-06 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114744+0000", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7570000000000001e-06 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114800+0000", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6419999999999999e-05 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.114812+0000", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2368000000000001e-05 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.039114+0000", 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92430181199999994 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.039244+0000", 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012981000000000001 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.039382+0000", 2024-03-26T04:04:50.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013788599999999999 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041662+0000", 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00228057 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.043785+0000", 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021223449999999999 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052585+0000", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0087998339999999994 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052654+0000", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9491000000000003e-05 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052713+0000", 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8396000000000001e-05 2024-03-26T04:04:50.180 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148466 8.2 8:4dd6fc49:::benchmark_data_smithi006_103169_object18535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.120404+0000", 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.01582356400002, 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.937513704, 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.181 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148466 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120404+0000", 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120404+0000", 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120405+0000", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.249e-06 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120411+0000", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2389999999999999e-06 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120425+0000", 2024-03-26T04:04:50.183 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.411e-05 2024-03-26T04:04:50.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120435+0000", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5109999999999992e-06 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120506+0000", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0975000000000004e-05 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.120519+0000", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3294e-05 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041717+0000", 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92119741899999996 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041802+0000", 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5315000000000006e-05 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.041889+0000", 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6737999999999995e-05 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045550+0000", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003661798 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.047599+0000", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020484769999999999 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057813+0000", 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010214226 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.187 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057859+0000", 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5416e-05 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057918+0000", 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8937000000000002e-05 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.189 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148479 8.2 8:4dfadd78:::benchmark_data_smithi006_103169_object18548:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.189 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.134421+0000", 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.001806262, 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92791684799999996, 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148479 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134421+0000", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134421+0000", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134422+0000", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1930000000000001e-06 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.191 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134428+0000", 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4040000000000003e-06 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134439+0000", 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0797e-05 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134445+0000", 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8349999999999998e-06 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.192 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134502+0000", 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6489000000000001e-05 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.134510+0000", 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2309999999999996e-06 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.193 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.043831+0000", 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.90932129100000003 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.043957+0000", 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000126095 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.044052+0000", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4635000000000004e-05 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.047665+0000", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00361276 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.049984+0000", 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023194140000000001 2024-03-26T04:04:50.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062258+0000", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012273256 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062301+0000", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3582999999999999e-05 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062338+0000", 2024-03-26T04:04:50.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6865e-05 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148545 8.2 8:4fc1dab6:::benchmark_data_smithi006_103169_object18614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.212549+0000", 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.92367846000002, 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.85004018599999998, 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.197 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148545 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212549+0000", 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212549+0000", 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212550+0000", 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.327e-06 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212559+0000", 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4600000000000003e-06 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212570+0000", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0832999999999999e-05 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212577+0000", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3769999999999998e-06 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212628+0000", 2024-03-26T04:04:50.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0800000000000002e-05 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.212636+0000", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2120000000000003e-06 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045753+0000", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.833117208 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045838+0000", 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5180999999999996e-05 2024-03-26T04:04:50.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.045920+0000", 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1998999999999999e-05 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052446+0000", 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0065256089999999999 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052505+0000", 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9017000000000001e-05 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062490+0000", 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0099850700000000004 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062541+0000", 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1258999999999998e-05 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062589+0000", 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7834000000000003e-05 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148561 8.2 8:40880508:::benchmark_data_smithi006_103169_object18630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.234146+0000", 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.90208154800001, 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.82859356200000001, 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.207 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148561 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234146+0000", 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234146+0000", 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234147+0000", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5400000000000001e-06 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234153+0000", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9100000000000002e-06 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234166+0000", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2897e-05 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234174+0000", 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6580000000000002e-06 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234245+0000", 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0988999999999994e-05 2024-03-26T04:04:50.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.234257+0000", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2466000000000001e-05 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.047715+0000", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.81345754999999997 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.047815+0000", 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100536 2024-03-26T04:04:50.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.047912+0000", 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6685e-05 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052725+0000", 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048131349999999996 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052781+0000", 2024-03-26T04:04:50.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5816000000000002e-05 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062648+0000", 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0098669710000000004 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062700+0000", 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2361000000000002e-05 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062739+0000", 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9048000000000003e-05 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148571 8.2 8:4eb821f0:::benchmark_data_smithi006_103169_object18640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.246983+0000", 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.88924442000001, 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.81600833900000003, 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.215 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148571 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.246983+0000", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.246983+0000", 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.246984+0000", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.238e-06 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.246990+0000", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3730000000000001e-06 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.247001+0000", 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1207e-05 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.217 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.247010+0000", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7530000000000007e-06 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.247066+0000", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6876999999999998e-05 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.247075+0000", 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9020000000000005e-06 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.050017+0000", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.80294128600000003 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.050140+0000", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000123872 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.050265+0000", 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012448099999999999 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052849+0000", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025838269999999999 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052863+0000", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4448e-05 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062871+0000", 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010007798 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062949+0000", 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8166999999999999e-05 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.222 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:50 smithi086 ceph-mon[32965]: pgmap v759: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:50.222 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:50 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T04:04:50.222 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:50 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T04:04:50.222 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:50 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 50922/305055 objects degraded (16.693%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:50.222 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:50 smithi086 ceph-mon[32965]: osdmap e370: 8 total, 7 up, 5 in 2024-03-26T04:04:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062991+0000", 2024-03-26T04:04:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2110000000000002e-05 2024-03-26T04:04:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.223 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148590 8.2 8:41360ee6:::benchmark_data_smithi006_103169_object18659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.268110+0000", 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86811760900002, 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.80249077700000004, 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148590 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.224 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268110+0000", 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268110+0000", 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268111+0000", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2979999999999999e-06 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268117+0000", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4720000000000003e-06 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268128+0000", 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1165999999999999e-05 2024-03-26T04:04:50.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268136+0000", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8000000000000004e-06 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268196+0000", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9998999999999999e-05 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.268206+0000", 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999993e-06 2024-03-26T04:04:50.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.052899+0000", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.78469303700000004 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.053027+0000", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012854199999999999 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.053155+0000", 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012795800000000001 2024-03-26T04:04:50.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057772+0000", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004616963 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057929+0000", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015647600000000001 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070486+0000", 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.012557634 2024-03-26T04:04:50.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070551+0000", 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4843000000000002e-05 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070601+0000", 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9489e-05 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.230 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148592 8.2 8:4fc4c97f:::benchmark_data_smithi006_103169_object18661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.271560+0000", 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86466782700001, 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.79918719199999999, 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.231 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148592 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271560+0000", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271560+0000", 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271561+0000", 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3680000000000001e-06 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271567+0000", 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1090000000000003e-06 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271581+0000", 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3743999999999999e-05 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271589+0000", 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8129999999999996e-06 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271657+0000", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7954999999999996e-05 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.271668+0000", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1756999999999999e-05 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.055323+0000", 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.783654305 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.055463+0000", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000139975 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.055616+0000", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015330899999999999 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062179+0000", 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006563251 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.236 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062348+0000", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016843000000000001 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070657+0000", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0083093860000000002 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070708+0000", 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0840000000000001e-05 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.070747+0000", 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8949999999999998e-05 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:148608 8.2 8:4b50b3dd:::benchmark_data_smithi006_103169_object18677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e196)", 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:56:39.298036+0000", 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.83819129900002, 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.78169135499999998, 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.238 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 148608 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298036+0000", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298036+0000", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298037+0000", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1489999999999999e-06 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298042+0000", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3529999999999997e-06 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298051+0000", 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.628e-06 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298060+0000", 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9919999999999996e-06 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298110+0000", 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9830000000000002e-05 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:39.298119+0000", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3379999999999994e-06 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.057965+0000", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.75984580199999996 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.058118+0000", 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015250499999999999 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,5", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.058265+0000", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014738800000000001 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062394+0000", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041289020000000003 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.062602+0000", 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208259 2024-03-26T04:04:50.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.079631+0000", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017028557 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.079673+0000", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2209000000000002e-05 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:56:40.079727+0000", 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4443e-05 2024-03-26T04:04:50.244 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246577 9.c 9:3c22429d:::benchmark_data_smithi006_132277_object114120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.662223578, 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0190602820000001, 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.245 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246577 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474005+0000", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.392e-06 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474011+0000", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6559999999999997e-06 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474028+0000", 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6803000000000002e-05 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474036+0000", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2830000000000001e-06 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474180+0000", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143886 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474200+0000", 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0151000000000001e-05 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487746+0000", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0135455710000001 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487828+0000", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2149000000000002e-05 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487907+0000", 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9178999999999995e-05 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492885+0000", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049778419999999997 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492929+0000", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4484000000000003e-05 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492960+0000", 2024-03-26T04:04:50.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1207999999999997e-05 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493012+0000", 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1739e-05 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493064+0000", 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1938999999999999e-05 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246589 9.c 9:39119fb8:::benchmark_data_smithi006_132277_object114132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.64747199200001, 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0130038640000001, 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246589 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.252 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488756+0000", 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0669999999999999e-06 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488763+0000", 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9990000000000002e-06 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488773+0000", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9250000000000007e-06 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488779+0000", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7579999999999998e-06 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488826+0000", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7361999999999998e-05 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488834+0000", 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2060000000000001e-06 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.489766+0000", 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.000932545 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.489892+0000", 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000126031 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.490007+0000", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011425399999999999 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493161+0000", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031543069999999999 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493214+0000", 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3563000000000002e-05 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501677+0000", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084626419999999994 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501713+0000", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5618999999999999e-05 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501759+0000", 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6585999999999997e-05 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.258 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246591 9.c 9:3623bec9:::benchmark_data_smithi006_132277_object114134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.635078604, 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0007254729999999, 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246591 2024-03-26T04:04:50.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501151+0000", 2024-03-26T04:04:50.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8729999999999999e-06 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501160+0000", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2569999999999996e-06 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501186+0000", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5873000000000001e-05 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501201+0000", 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5702999999999999e-05 2024-03-26T04:04:50.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501323+0000", 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121037 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501350+0000", 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7759999999999998e-05 2024-03-26T04:04:50.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491627+0000", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99027648999999995 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491716+0000", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9291999999999998e-05 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491794+0000", 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8319999999999996e-05 2024-03-26T04:04:50.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493276+0000", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001481254 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493292+0000", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6690000000000001e-05 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501788+0000", 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084954339999999996 2024-03-26T04:04:50.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501830+0000", 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2565000000000003e-05 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501874+0000", 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3924999999999997e-05 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.265 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246599 9.c 9:3b75a9f3:::benchmark_data_smithi006_132277_object114142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.626861019, 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99391891300000001, 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246599 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509368+0000", 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2049999999999999e-06 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509372+0000", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9150000000000004e-06 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509382+0000", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0016e-05 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509390+0000", 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9869999999999999e-06 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509442+0000", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1116999999999999e-05 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509451+0000", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0990000000000005e-06 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493330+0000", 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98387935100000001 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493444+0000", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114162 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493519+0000", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5093999999999996e-05 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498353+0000", 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048332870000000003 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498420+0000", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7359000000000003e-05 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503230+0000", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048096390000000001 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503261+0000", 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1149999999999998e-05 2024-03-26T04:04:50.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503285+0000", 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246608 9.c 9:3b4033d5:::benchmark_data_smithi006_132277_object114151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.61790282299999, 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98504782099999999, 2024-03-26T04:04:50.272 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246608 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518326+0000", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.578e-06 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518331+0000", 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0690000000000001e-06 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518341+0000", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5929999999999995e-06 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518348+0000", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6550000000000004e-06 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518403+0000", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5022000000000003e-05 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518413+0000", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6120000000000005e-06 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.494934+0000", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97652113500000004 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.495065+0000", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013094800000000001 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.495183+0000", 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118294 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498409+0000", 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003225983 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498423+0000", 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3837e-05 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503319+0000", 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004895885 2024-03-26T04:04:50.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503349+0000", 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9784000000000001e-05 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503372+0000", 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3425999999999999e-05 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246642 9.c 9:342d0ba9:::benchmark_data_smithi006_132277_object114185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.279 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.580671751, 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94969048599999994, 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246642 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555557+0000", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.057e-06 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.281 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555561+0000", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1509999999999997e-06 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555571+0000", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.893e-06 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555578+0000", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0040000000000002e-06 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.282 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555623+0000", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5130000000000003e-05 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555629+0000", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2659999999999998e-06 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496611+0000", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94098225300000005 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.283 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496742+0000", 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013053199999999999 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496862+0000", 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120195 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501516+0000", 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046534380000000002 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.284 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501570+0000", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4661999999999999e-05 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505186+0000", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036161320000000002 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505220+0000", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3342e-05 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.285 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505246+0000", 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6431000000000002e-05 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246653 9.c 9:309ed127:::benchmark_data_smithi006_132277_object114196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 104.56693893000001, 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93844572599999998, 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:04:50.286 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246653 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.287 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569289+0000", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9099999999999991e-07 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569294+0000", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.16e-06 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569304+0000", 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0180000000000001e-05 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.288 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569310+0000", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1689999999999997e-06 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569354+0000", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4168999999999997e-05 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569361+0000", 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.99e-06 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.289 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498462+0000", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92910126800000004 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498588+0000", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125926 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498728+0000", 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013995300000000001 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.290 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501622+0000", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028934070000000002 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501633+0000", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1182000000000001e-05 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507630+0000", 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059974529999999998 2024-03-26T04:04:50.291 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507690+0000", 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9312999999999999e-05 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507734+0000", 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4564999999999998e-05 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:04:50.292 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:04:50.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:50.343 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:50.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 52 16 110817 110801 133.155 111.312 0.00832757 0.00750611 2024-03-26T04:04:50.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[37501]: pgmap v759: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 50922/305055 objects degraded (16.693%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[37501]: osdmap e370: 8 total, 7 up, 5 in 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[29446]: pgmap v759: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.2 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 50922/305055 objects degraded (16.693%) 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 50922/305055 objects degraded (16.693%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:50.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:50 smithi006 ceph-mon[29446]: osdmap e370: 8 total, 7 up, 5 in 2024-03-26T04:04:50.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:50.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:50.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:50.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:50.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:50.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:50.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:50.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:50.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:50.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:51.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:51.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:51.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:51.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:51.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T04:04:51.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:51.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:51.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:51.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 53 15 112825 112810 133.011 125.562 0.00885738 0.00751411 2024-03-26T04:04:51.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:51.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:51.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:51.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:51.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.609 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:51.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:51.650 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:51.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:51.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:51.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:51.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:51.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:51.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:51.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:52.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: pgmap v761: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 53801/318753 objects degraded (16.879%) 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T04:04:52.059 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:52 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: pgmap v761: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 53801/318753 objects degraded (16.879%) 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T04:04:52.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: pgmap v761: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.5 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.66k op/s; 53801/318753 objects degraded (16.879%) 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T04:04:52.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:52 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T04:04:52.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:52.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:52.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:52.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:52.338 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:04:52.379 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:04:52.379 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 54 16 114734 114718 132.756 119.25 0.00399328 0.00752861 2024-03-26T04:04:52.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:52.419 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:04:52.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.438 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:52.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:52.462 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:04:52.498 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:04:52.537 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:04:52.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:52.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:04:52.577 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:04:52.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:52.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:52.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:52.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:52.869 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:52.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:52.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:52.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:04:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:53.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:53.060 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1711449572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:53.060 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T04:04:53.060 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T04:04:53.062 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1711449572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:53.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T04:04:53.063 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:53 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T04:04:53.073 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1711449572' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:04:53.073 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:53 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T04:04:53.073 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:53 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T04:04:53.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:53.148 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:53.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:53.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:53.280 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:53.382 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 55 16 116625 116609 132.49 118.188 0.0176914 0.00754324 2024-03-26T04:04:53.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:53.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:53.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:53.545 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:53.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:53.546 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:04:53.546 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:04:53.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:53.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:53.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:53.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:53.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:53.869 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:04:53.870 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:04:53.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:53.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:53.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:54.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:54.125 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:54.133 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:04:54.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[37501]: pgmap v762: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s; 55535/332382 objects degraded (16.708%) 2024-03-26T04:04:54.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:04:54.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:04:54.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[37501]: 1.6 scrub starts 2024-03-26T04:04:54.134 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[37501]: 1.6 scrub ok 2024-03-26T04:04:54.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[29446]: pgmap v762: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s; 55535/332382 objects degraded (16.708%) 2024-03-26T04:04:54.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:04:54.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:04:54.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[29446]: 1.6 scrub starts 2024-03-26T04:04:54.135 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-mon[29446]: 1.6 scrub ok 2024-03-26T04:04:54.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:54.230 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:54 smithi086 ceph-mon[32965]: pgmap v762: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 6.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 639 B/s rd, 106 MiB/s wr, 1.69k op/s; 55535/332382 objects degraded (16.708%) 2024-03-26T04:04:54.230 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:54 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:04:54.231 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:54 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:04:54.231 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:54 smithi086 ceph-mon[32965]: 1.6 scrub starts 2024-03-26T04:04:54.231 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:54 smithi086 ceph-mon[32965]: 1.6 scrub ok 2024-03-26T04:04:54.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:54.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T04:04:54.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:54.380 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 56 16 118493 118477 132.209 116.75 0.0052491 0.00755968 2024-03-26T04:04:54.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:04:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:04:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:04:54.415 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:54.425 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:54.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:54.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:54.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:54.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:54.731 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:54.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:54.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:54.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:54.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:54.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:54.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:55.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.134 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:55.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:55 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 55535/332382 objects degraded (16.708%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:55.138 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:55 smithi086 ceph-mon[32965]: pgmap v763: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 58446/346332 objects degraded (16.876%) 2024-03-26T04:04:55.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 55535/332382 objects degraded (16.708%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:55.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:55.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[37501]: pgmap v763: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 58446/346332 objects degraded (16.876%) 2024-03-26T04:04:55.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 55535/332382 objects degraded (16.708%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:55.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:04:55.228 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:55 smithi006 ceph-mon[29446]: pgmap v763: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.0 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 107 MiB/s wr, 1.72k op/s; 58446/346332 objects degraded (16.876%) 2024-03-26T04:04:55.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:55.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:55.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:55.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:55.380 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 57 16 120447 120431 132.031 122.125 0.00707278 0.00756976 2024-03-26T04:04:55.412 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:55.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:55.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:55.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:55.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:04:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:04:55.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:55.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:55.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:55.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:55.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:04:55.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:55.956 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:04:55.957 INFO:tasks.thrashosds.thrasher:Failed to reset_purged_snaps_last, ignoring 2024-03-26T04:04:55.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.4 reset_purged_snaps_last 2024-03-26T04:04:55.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:55.994 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:55.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:56.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:56.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.145 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:56.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:56.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:56.284 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:56.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:56.380 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 58 15 122346 122331 131.802 118.75 0.00323763 0.00758271 2024-03-26T04:04:56.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:56.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:56.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.562 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:56.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:56.652 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:04:56.652 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:04:56.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:56.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:56.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:56.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T04:04:56.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T04:04:56.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T04:04:56.797 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T04:04:56.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:56.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T04:04:56.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T04:04:56.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T04:04:56.798 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:56 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T04:04:56.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:04:56.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:56 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T04:04:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:56 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T04:04:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:56 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T04:04:56.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:56 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T04:04:56.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:56.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:56.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:56.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:56.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:56.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:57.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:57.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:57.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:57.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:57.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:57.335 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:57.338 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:04:57.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:57.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 59 16 124086 124070 131.402 108.688 0.0102584 0.00760625 2024-03-26T04:04:57.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T04:04:57.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:57.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:57.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:57.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:57.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:57.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.695 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:57.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:57.779 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:04:57.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:57.809 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T04:04:57.810 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T04:04:57.810 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[37501]: pgmap v764: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:57.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T04:04:57.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T04:04:57.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:57 smithi006 ceph-mon[29446]: pgmap v764: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:57.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:57 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T04:04:57.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:57 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T04:04:57.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:57 smithi086 ceph-mon[32965]: pgmap v764: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 133 MiB/s wr, 2.13k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:57.843 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:57.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:57.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:57.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:57.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:58.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:58.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:58.223 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:58.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:58.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:04:58.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:04:58.385101+0000 min lat: 0.00114192 max lat: 0.159537 avg lat: 0.00762179 2024-03-26T04:04:58.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:04:58.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 60 15 125934 125919 131.138 115.562 0.00791432 0.00762179 2024-03-26T04:04:58.399 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:04:58.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:58.527 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:04:58.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:58.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:04:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:58.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:04:58.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:58.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:58.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:04:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:58 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T04:04:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:58 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T04:04:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:58 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T04:04:58.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:58 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T04:04:58.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T04:04:58.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T04:04:58.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:04:58.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:58.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:58.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:59.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:59.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:59.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:59.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:04:59.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:04:59.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:59.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 61 16 127750 127734 130.847 113.438 0.00883347 0.00763829 2024-03-26T04:04:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:59.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:59.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:04:59.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:59.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: osdmap e371: 8 total, 7 up, 5 in 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 6.16 deep-scrub starts 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 6.16 deep-scrub ok 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: pgmap v766: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 4.7 deep-scrub starts 2024-03-26T04:04:59.559 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: 4.7 deep-scrub ok 2024-03-26T04:04:59.560 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 60290/361110 objects degraded (16.696%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:59.560 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:04:59 smithi086 ceph-mon[32965]: osdmap e372: 8 total, 7 up, 5 in 2024-03-26T04:04:59.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:04:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.689 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:59.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:04:59.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:04:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:04:59.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: osdmap e371: 8 total, 7 up, 5 in 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 6.16 deep-scrub starts 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 6.16 deep-scrub ok 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: pgmap v766: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 4.7 deep-scrub starts 2024-03-26T04:04:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: 4.7 deep-scrub ok 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 60290/361110 objects degraded (16.696%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[37501]: osdmap e372: 8 total, 7 up, 5 in 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1870497522' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: osdmap e371: 8 total, 7 up, 5 in 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 6.16 deep-scrub starts 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 6.16 deep-scrub ok 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: pgmap v766: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 7.3 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.1 KiB/s rd, 126 MiB/s wr, 2.02k op/s; 60290/361110 objects degraded (16.696%) 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 4.7 deep-scrub starts 2024-03-26T04:04:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: 4.7 deep-scrub ok 2024-03-26T04:04:59.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 60290/361110 objects degraded (16.696%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:04:59.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:04:59 smithi006 ceph-mon[29446]: osdmap e372: 8 total, 7 up, 5 in 2024-03-26T04:04:59.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:04:59.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:04:59.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:04:59.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:00.075 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:00.093 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:05:00.094 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:05:00.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:00.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:00.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:00.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:00.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:00.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 62 15 129459 129444 130.461 106.875 0.00737536 0.00766099 2024-03-26T04:05:00.385 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:00.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:00.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:00.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:00.669 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:00.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:00.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:00.778 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:00 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:05:00.778 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:00 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:05:00.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:00.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:00.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T04:05:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:00 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:05:00.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:00 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:05:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:00 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:05:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:00 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:05:00.907 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:00.944 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:00.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:01.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:01.080 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:01.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:01.248 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:01.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:01.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:01.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:01.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 63 16 131253 131237 130.169 112.062 0.00530587 0.00767874 2024-03-26T04:05:01.386 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:01.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:01.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:01.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:01.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:01.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:01.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:01.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:01.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:01 smithi086 ceph-mon[32965]: 10.c scrub starts 2024-03-26T04:05:01.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:01 smithi086 ceph-mon[32965]: pgmap v768: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 63173/375006 objects degraded (16.846%) 2024-03-26T04:05:01.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:01 smithi086 ceph-mon[32965]: 5.7 deep-scrub starts 2024-03-26T04:05:01.688 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:01 smithi086 ceph-mon[32965]: 5.7 deep-scrub ok 2024-03-26T04:05:01.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:01.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[37501]: 10.c scrub starts 2024-03-26T04:05:01.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[37501]: pgmap v768: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 63173/375006 objects degraded (16.846%) 2024-03-26T04:05:01.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[37501]: 5.7 deep-scrub starts 2024-03-26T04:05:01.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[37501]: 5.7 deep-scrub ok 2024-03-26T04:05:01.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[29446]: 10.c scrub starts 2024-03-26T04:05:01.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[29446]: pgmap v768: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.6 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 63173/375006 objects degraded (16.846%) 2024-03-26T04:05:01.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[29446]: 5.7 deep-scrub starts 2024-03-26T04:05:01.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:01 smithi006 ceph-mon[29446]: 5.7 deep-scrub ok 2024-03-26T04:05:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.810 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:01.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:01.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:01.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:01.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:01.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:01.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:05:02.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:02.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:02.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:02.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-26T04:05:02.227 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:02.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:02.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:02.377 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:02.384 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 64 16 132992 132976 129.833 108.688 0.00934879 0.00769812 2024-03-26T04:05:02.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:02.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:02.566 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:05:02.610 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:02 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3107700918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:02.611 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:05:02.611 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:02 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3107700918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:02.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:02.631 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:02.652 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:05:02.672 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:02.686 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:02.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:02.701 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:05:02.738 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:05:02.777 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:05:02.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:02.800 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:02 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3107700918' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:02.819 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:05:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:05:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:05:02.819 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:05:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:05:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:05:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:05:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:05:02.820 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:05:02.828 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:02.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:02.977 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:02.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:03.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:03.119 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:03.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:03.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:03.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:03.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 65 15 134765 134750 129.541 110.875 0.00752831 0.00771569 2024-03-26T04:05:03.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:03.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:03.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:03.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:135058 10.c 10:39146c91:::benchmark_data_smithi006_159841_object135057:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:03.545423+0000", 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 0.0077666840000000003, 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0078054539999999999, 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:03.554 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 135058 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545423+0000", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:03.555 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545423+0000", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545424+0000", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.114e-06 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545460+0000", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5976999999999998e-05 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:03.556 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545472+0000", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1401e-05 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545479+0000", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4189999999999996e-06 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545530+0000", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0936999999999997e-05 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:03.557 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545660+0000", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013004800000000001 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.545812+0000", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015208800000000001 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.548635+0000", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028228160000000001 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:03.558 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.549821+0000", 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011863119999999999 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.15540.0:135066 10.5 e372/349)", 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:03.553234+0000", 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 4294967295.9999561, 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012981000000000001, 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "queued for pg", 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:03.559 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553234+0000", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553234+0000", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553235+0000", 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3600000000000003e-07 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.560 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553290+0000", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5455999999999999e-05 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553305+0000", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4647000000000001e-05 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:03.553313+0000", 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8949999999999998e-06 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:03.561 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:03.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:03.562 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:03.562 INFO:teuthology.orchestra.run.smithi086.stdout: ], 2024-03-26T04:05:03.562 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 2 2024-03-26T04:05:03.562 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:05:03.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:03.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:03.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:03.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:03.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:03.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:03.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:03 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T04:05:03.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:03 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T04:05:03.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:03 smithi086 ceph-mon[32965]: pgmap v769: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s; 64916/388392 objects degraded (16.714%) 2024-03-26T04:05:03.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:03 smithi086 ceph-mon[32965]: 10.c scrub ok 2024-03-26T04:05:03.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[37501]: pgmap v769: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s; 64916/388392 objects degraded (16.714%) 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[37501]: 10.c scrub ok 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[29446]: pgmap v769: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 7.9 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 639 B/s rd, 109 MiB/s wr, 1.75k op/s; 64916/388392 objects degraded (16.714%) 2024-03-26T04:05:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:03 smithi006 ceph-mon[29446]: 10.c scrub ok 2024-03-26T04:05:03.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:03.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:04.048 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:04.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.076 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:04.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:04.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:04.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:04.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:04.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.339 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:04.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:04.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 66 15 136553 136538 129.271 111.75 0.00656394 0.00773193 2024-03-26T04:05:04.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:04.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:04.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:04.554 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:04.567 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:04.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:04.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:04.739 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:04 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T04:05:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:04 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T04:05:04.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:04 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 64916/388392 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:04.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:04.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:04.876 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:04.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:04.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T04:05:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T04:05:04.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 64916/388392 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T04:05:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T04:05:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:04 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 64916/388392 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:04.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.014 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:05.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:05.115 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:05.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.139 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:05.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:05.240 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.264 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:05.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:05.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:05.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 67 16 137950 137934 128.644 87.25 0.00317936 0.0077629 2024-03-26T04:05:05.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.394 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:05.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:05.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:05.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:05.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:05.562 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:05.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:05.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:05.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:05.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:05.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T04:05:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T04:05:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 10.d deep-scrub starts 2024-03-26T04:05:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: pgmap v770: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 67596/401361 objects degraded (16.842%) 2024-03-26T04:05:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T04:05:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T04:05:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:05 smithi086 ceph-mon[32965]: 10.0 scrub starts 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 10.d deep-scrub starts 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: pgmap v770: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 67596/401361 objects degraded (16.842%) 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T04:05:05.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[37501]: 10.0 scrub starts 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 10.d deep-scrub starts 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: pgmap v770: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s; 67596/401361 objects degraded (16.842%) 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T04:05:05.851 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:05 smithi006 ceph-mon[29446]: 10.0 scrub starts 2024-03-26T04:05:05.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:05.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:05.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:05.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:06.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.067 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:06.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:06.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:06.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:06.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:06.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:06.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.341 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:06.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:06.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 68 16 139382 139366 128.068 89.5 0.0607816 0.00780089 2024-03-26T04:05:06.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:06.479 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:06.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:06.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T04:05:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:06.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:06.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:06.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:06.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:06.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:06.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:06.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:06.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:06.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:06.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:06.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:06.997 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:07.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:07.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:07.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.179 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:07.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:07.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:07.320 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:07.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:07.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 69 16 140774 140758 127.473 87 0.00523966 0.00784062 2024-03-26T04:05:07.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:07.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:07.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:07.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:07.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:07.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:07.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:07.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:07.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:07 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T04:05:07.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:07 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T04:05:07.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:07 smithi086 ceph-mon[32965]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.14k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:07.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:07.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:07.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:07.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T04:05:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T04:05:07.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[37501]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.14k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T04:05:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T04:05:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:07 smithi006 ceph-mon[29446]: pgmap v771: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1.2 KiB/s rd, 133 MiB/s wr, 2.14k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:07.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:08.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:08.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:08.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:08.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.299 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:08.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 70 16 142255 142239 126.974 92.5625 0.0047643 0.00787122 2024-03-26T04:05:08.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:08.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:08.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:08.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:08.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:08.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:08 smithi086 ceph-mon[32965]: 10.0 scrub ok 2024-03-26T04:05:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:08 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T04:05:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:08 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T04:05:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:08 smithi086 ceph-mon[32965]: 10.2 scrub starts 2024-03-26T04:05:08.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:08.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:08.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[37501]: 10.0 scrub ok 2024-03-26T04:05:08.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T04:05:08.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T04:05:08.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[37501]: 10.2 scrub starts 2024-03-26T04:05:08.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[29446]: 10.0 scrub ok 2024-03-26T04:05:08.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T04:05:08.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T04:05:08.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:08 smithi006 ceph-mon[29446]: 10.2 scrub starts 2024-03-26T04:05:08.957 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:08.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:09.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.139 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:09.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:09.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:09.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:09.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:09.377 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:09.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 71 16 143595 143579 126.366 83.75 0.00494561 0.00790845 2024-03-26T04:05:09.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:09.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:09.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:09.540 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:09.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:09.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:09.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:09.770 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:09 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T04:05:09.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:09 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T04:05:09.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:09 smithi086 ceph-mon[32965]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.76k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:09.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:09 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 69141/413868 objects degraded (16.706%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:09.787 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:09.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:09.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T04:05:09.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T04:05:09.871 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[37501]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.76k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:09.872 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 69141/413868 objects degraded (16.706%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:09.872 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T04:05:09.872 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T04:05:09.872 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[29446]: pgmap v772: 158 pgs: 1 active+clean+scrubbing, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.4 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 110 MiB/s wr, 1.76k op/s; 69141/413868 objects degraded (16.706%) 2024-03-26T04:05:09.872 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:09 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 69141/413868 objects degraded (16.706%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:05:09.921 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:05:09.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:09.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:09.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:10.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:10.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:10.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:10.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:10.245 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:10.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:10.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:10.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 72 16 144971 144955 125.805 86 0.00429907 0.00794342 2024-03-26T04:05:10.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:10.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:10.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:10.550 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:10.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:10.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:10.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:10.773 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:05:10.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:10.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:10.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:10.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:10.957 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:10.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:11.058 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.084 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:11.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:10 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:10 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/109465357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:10 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.130 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:11.130 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:11.130 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/109465357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.130 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:11.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:11.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/109465357' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:10 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:05:11.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:11.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.222 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:11.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:11.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:11.361 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:11.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:11.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 73 16 146313 146297 125.23 83.875 0.00619385 0.00798062 2024-03-26T04:05:11.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:11.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:11.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:11.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:11.635 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:11.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:11.740 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:11.777 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:11.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:11.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:11.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:11.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:11.923 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:05:12.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:12.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:12.069 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:05:12.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:05:12.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:12.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:05:12.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 74 active+clean; 8.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s; 71351/424254 objects degraded (16.818%) 2024-03-26T04:05:12.070 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:11 smithi086 ceph-mon[32965]: osdmap e373: 8 total, 7 up, 5 in 2024-03-26T04:05:12.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 74 active+clean; 8.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s; 71351/424254 objects degraded (16.818%) 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[37501]: osdmap e373: 8 total, 7 up, 5 in 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: pgmap v774: 158 pgs: 1 active+clean+scrubbing, 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 74 active+clean; 8.6 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 1023 B/s rd, 103 MiB/s wr, 1.64k op/s; 71351/424254 objects degraded (16.818%) 2024-03-26T04:05:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:11 smithi006 ceph-mon[29446]: osdmap e373: 8 total, 7 up, 5 in 2024-03-26T04:05:12.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:12.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:12.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:12.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:12.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 74 16 147829 147813 124.818 94.75 0.00415738 0.00800714 2024-03-26T04:05:12.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:12.466 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:12.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:12.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:12.599 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:12.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:12.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:12.738 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:12.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:12.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:12.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:12.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:12.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:12.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:13.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:13.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.158 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:13.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:13.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:13.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:13.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:13.385 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 75 16 149148 149132 124.253 82.4375 0.00577424 0.00804392 2024-03-26T04:05:13.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:13.411 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:05:13.411 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:05:13.411 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:05:13.411 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:05:13.411 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:05:13.418 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:13.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:13.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:13.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:13.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:13.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:13.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:13.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:13.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:13.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:13.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:14.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:14.085 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:13 smithi086 ceph-mon[32965]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s; 72666/434889 objects degraded (16.709%) 2024-03-26T04:05:14.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:14.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:14.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T04:05:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:13 smithi006 ceph-mon[37501]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s; 72666/434889 objects degraded (16.709%) 2024-03-26T04:05:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:13 smithi006 ceph-mon[29446]: pgmap v775: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 8.8 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s; 72666/434889 objects degraded (16.709%) 2024-03-26T04:05:14.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:14.219 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:14.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:14.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:14.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:14.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 76 16 150471 150455 123.706 82.6875 0.005439 0.00807796 2024-03-26T04:05:14.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:14.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:14.502 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:14.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:14.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:14.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:14.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:14.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:14.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:14.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:14.903 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:14 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 72666/434889 objects degraded (16.709%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:14.903 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:14 smithi086 ceph-mon[32965]: osdmap e374: 8 total, 7 up, 5 in 2024-03-26T04:05:14.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:14.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:14.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:15.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:15.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:15.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:15.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:14 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 72666/434889 objects degraded (16.709%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:14 smithi006 ceph-mon[37501]: osdmap e374: 8 total, 7 up, 5 in 2024-03-26T04:05:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:14 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 72666/434889 objects degraded (16.709%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:14 smithi006 ceph-mon[29446]: osdmap e374: 8 total, 7 up, 5 in 2024-03-26T04:05:15.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:15.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:15.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:15.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:15.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:15.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:15.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 77 16 151809 151793 123.186 83.625 0.00633291 0.00811304 2024-03-26T04:05:15.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:15.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:15.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:15.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:15.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:15.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:15.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:15.820 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:15.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:15.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:15.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:15 smithi086 ceph-mon[32965]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 74896/445251 objects degraded (16.821%) 2024-03-26T04:05:15.956 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:15 smithi006 ceph-mon[37501]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 74896/445251 objects degraded (16.821%) 2024-03-26T04:05:15.957 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:15 smithi006 ceph-mon[29446]: pgmap v777: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 74896/445251 objects degraded (16.821%) 2024-03-26T04:05:15.971 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:16.010 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:16.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:16.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:16.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:16.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:16.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:16.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:16.384 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7] out_osds: [6, 5, 3] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2024-03-26T04:05:16.384 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2024-03-26T04:05:16.384 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2024-03-26T04:05:16.384 DEBUG:teuthology.orchestra.run.smithi006:osd.2> sudo journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:05:16.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 78 16 153226 153210 122.742 88.5625 0.00362918 0.00813715 2024-03-26T04:05:16.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl start ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:05:16.387 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:184189 9.d 9:bd97202a:::benchmark_data_smithi006_132277_object51732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e302)", 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:01:50.043406+0000", 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 206.34151451899999, 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1806825299999999, 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.388 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 184189 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043406+0000", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043406+0000", 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043407+0000", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9599999999999998e-07 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043412+0000", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7360000000000001e-06 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043422+0000", 2024-03-26T04:05:16.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7850000000000003e-06 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043428+0000", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.19e-06 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043469+0000", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1309e-05 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043562+0000", 2024-03-26T04:05:16.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3224000000000001e-05 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4,5", 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.043635+0000", 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3076000000000003e-05 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.045753+0000", 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021172669999999999 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.045795+0000", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2574999999999998e-05 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.046247+0000", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00045143199999999998 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.188653+0000", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.142405895 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.188661+0000", 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3259999999999996e-06 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.224012+0000", 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0353512519999999 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.224089+0000", 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6567000000000001e-05 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:184209 9.d 9:b07a148d:::benchmark_data_smithi006_132277_object51752:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:01:50.196237+0000", 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 206.18868412099999, 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0353725709999999, 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 184209 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.395 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196237+0000", 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196237+0000", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196238+0000", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0410000000000001e-06 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196243+0000", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.916e-06 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196253+0000", 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0360000000000001e-05 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196259+0000", 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8370000000000002e-06 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196345+0000", 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6089999999999997e-05 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.398 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.196352+0000", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0049999999999998e-06 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.224495+0000", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.028142841 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.224603+0000", 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108352 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.399 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.224733+0000", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130254 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228552+0000", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0038189980000000001 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228635+0000", 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2225000000000006e-05 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.400 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231535+0000", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029006309999999999 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231572+0000", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6350000000000003e-05 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231609+0000", 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7670999999999999e-05 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.401 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:184225 9.d 9:b73f49c4:::benchmark_data_smithi006_132277_object51768:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:01:50.211246+0000", 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 206.17367501499999, 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0220803220000001, 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.402 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 184225 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211246+0000", 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211246+0000", 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.403 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211247+0000", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4099999999999997e-07 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211251+0000", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.211e-06 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211260+0000", 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4839999999999993e-06 2024-03-26T04:05:16.404 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211266+0000", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7969999999999997e-06 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211319+0000", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2744000000000002e-05 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.211326+0000", 2024-03-26T04:05:16.405 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7780000000000002e-06 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.226779+0000", 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0154529990000001 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.406 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.226924+0000", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145585 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.227008+0000", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3554999999999995e-05 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228578+0000", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015706100000000001 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.407 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231445+0000", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028666849999999999 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233239+0000", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001793949 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233287+0000", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8245999999999999e-05 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.408 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233326+0000", 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8838000000000003e-05 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:184250 9.d 9:b70e0da6:::benchmark_data_smithi006_132277_object51793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:01:50.246867+0000", 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 206.13805421999999, 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98661122099999998, 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.409 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 184250 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246867+0000", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246867+0000", 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.410 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246867+0000", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9499999999999998e-07 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246869+0000", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3329999999999999e-06 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246875+0000", 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7479999999999999e-06 2024-03-26T04:05:16.411 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246880+0000", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8280000000000002e-06 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246932+0000", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1963999999999999e-05 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.246941+0000", 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1970000000000002e-06 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.412 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228654+0000", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98171289699999997 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228732+0000", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8474999999999997e-05 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.228800+0000", 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7181000000000006e-05 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.413 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231486+0000", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026866920000000001 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.231498+0000", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1874e-05 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233379+0000", 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0018809429999999999 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.414 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.415 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233435+0000", 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5788000000000002e-05 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233478+0000", 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2905999999999999e-05 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:184278 9.d 9:b62c752a:::benchmark_data_smithi006_132277_object51821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e303)", 2024-03-26T04:05:16.416 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:01:50.270815+0000", 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 206.11410536299999, 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96690084600000004, 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 184278 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.417 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270815+0000", 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270815+0000", 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270816+0000", 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1999999999999998e-07 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.418 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270821+0000", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6090000000000003e-06 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270830+0000", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2070000000000001e-06 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270836+0000", 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8619999999999999e-06 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.419 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270891+0000", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4632999999999997e-05 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:50.270900+0000", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4269999999999997e-06 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.230125+0000", 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95922504200000003 2024-03-26T04:05:16.420 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.230196+0000", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0587999999999996e-05 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 1,4", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.230264+0000", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8182000000000006e-05 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233202+0000", 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029380309999999998 2024-03-26T04:05:16.421 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.233489+0000", 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028753399999999999 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.237621+0000", 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004131254 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.422 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.237672+0000", 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1248999999999997e-05 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:01:51.237716+0000", 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4308000000000003e-05 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:244979 9.1 9:89c38e11:::benchmark_data_smithi006_132277_object112522:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.581915+0000", 2024-03-26T04:05:16.423 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.80300557699999, 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.027667324, 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 244979 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581915+0000", 2024-03-26T04:05:16.424 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581915+0000", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581916+0000", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8700000000000004e-07 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581923+0000", 2024-03-26T04:05:16.425 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6390000000000003e-06 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581936+0000", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3523000000000001e-05 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581946+0000", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6490000000000003e-06 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.581996+0000", 2024-03-26T04:05:16.426 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0348e-05 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.582006+0000", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6420000000000002e-06 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.604384+0000", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0223785830000001 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.604514+0000", 2024-03-26T04:05:16.427 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012959899999999999 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.604658+0000", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014370600000000001 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.606657+0000", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019990350000000001 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609437+0000", 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027805740000000001 2024-03-26T04:05:16.428 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609487+0000", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9691000000000002e-05 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609540+0000", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2806000000000002e-05 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609583+0000", 2024-03-26T04:05:16.429 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2642000000000003e-05 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245036 9.1 9:8f621ec8:::benchmark_data_smithi006_132277_object112579:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.636381+0000", 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.74853991500001, 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97340816299999999, 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.430 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245036 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636381+0000", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636381+0000", 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.431 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636381+0000", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1500000000000004e-07 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636385+0000", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3589999999999999e-06 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636392+0000", 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.52e-06 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.432 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636397+0000", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9599999999999999e-06 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636443+0000", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6153999999999999e-05 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.636454+0000", 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0498e-05 2024-03-26T04:05:16.433 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.606688+0000", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97023448999999995 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.606805+0000", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116959 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.606939+0000", 2024-03-26T04:05:16.434 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000133814 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609595+0000", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002655733 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609648+0000", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2596999999999998e-05 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609697+0000", 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9166000000000003e-05 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609749+0000", 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2782999999999999e-05 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609789+0000", 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9514999999999997e-05 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.436 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245049 9.1 9:82f92af0:::benchmark_data_smithi006_132277_object112592:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.651422+0000", 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.733498618, 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96129786900000003, 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245049 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.437 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651422+0000", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651422+0000", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651422+0000", 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7099999999999998e-07 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.438 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651424+0000", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.717e-06 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651429+0000", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9230000000000001e-06 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651432+0000", 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2459999999999999e-06 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.439 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651472+0000", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0166000000000001e-05 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.651478+0000", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4149999999999998e-06 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.608164+0000", 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95668620900000001 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.440 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.608273+0000", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108928 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.608389+0000", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115998 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609800+0000", 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001410681 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.441 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609836+0000", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6040000000000001e-05 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612662+0000", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028264879999999998 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612693+0000", 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0805000000000001e-05 2024-03-26T04:05:16.442 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612720+0000", 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6982e-05 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245054 9.1 9:87052cc5:::benchmark_data_smithi006_132277_object112597:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.655431+0000", 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.729489371, 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95980177300000002, 2024-03-26T04:05:16.443 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245054 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655431+0000", 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.444 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655431+0000", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655432+0000", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6399999999999998e-07 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655434+0000", 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.43e-06 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.445 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655440+0000", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9179999999999999e-06 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655445+0000", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9880000000000004e-06 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655498+0000", 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3146999999999998e-05 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.446 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.655509+0000", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1095e-05 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609876+0000", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95436666999999997 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.609985+0000", 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000108721 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.447 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.610108+0000", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012359299999999999 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612784+0000", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026758139999999999 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.613795+0000", 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001011271 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.448 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615163+0000", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001367215 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615195+0000", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2687000000000001e-05 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615233+0000", 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.786e-05 2024-03-26T04:05:16.449 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245089 9.1 9:870184a2:::benchmark_data_smithi006_132277_object112632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.691102+0000", 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.69381829899999, 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.924295808, 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.450 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245089 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691102+0000", 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691102+0000", 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.451 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691103+0000", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8500000000000002e-07 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691105+0000", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8920000000000001e-06 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691109+0000", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1790000000000001e-06 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.452 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691112+0000", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6160000000000002e-06 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691152+0000", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9996999999999997e-05 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691160+0000", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0450000000000001e-06 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.453 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.611174+0000", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92001306299999996 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.611279+0000", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000105123 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.611376+0000", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7122999999999998e-05 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.454 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615067+0000", 2024-03-26T04:05:16.455 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036909989999999999 2024-03-26T04:05:16.455 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.455 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615103+0000", 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6495999999999998e-05 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.457 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615341+0000", 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00023739799999999999 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615371+0000", 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0073000000000002e-05 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615398+0000", 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7518999999999999e-05 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.458 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245090 9.1 9:8b625acd:::benchmark_data_smithi006_132277_object112633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.691120+0000", 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.693800939, 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93198656300000005, 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245090 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.459 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691120+0000", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691120+0000", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691120+0000", 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1e-07 2024-03-26T04:05:16.460 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691123+0000", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.52e-06 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691128+0000", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0220000000000004e-06 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691134+0000", 2024-03-26T04:05:16.461 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0730000000000001e-06 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691187+0000", 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3368000000000001e-05 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.691195+0000", 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9650000000000008e-06 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.462 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612824+0000", 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92162887000000004 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612908+0000", 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3790999999999998e-05 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.612974+0000", 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6581999999999994e-05 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.463 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615244+0000", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002269535 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615293+0000", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8899000000000001e-05 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623022+0000", 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0077294290000000003 2024-03-26T04:05:16.464 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623068+0000", 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5605999999999997e-05 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623106+0000", 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8593000000000001e-05 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:245095 9.1 9:843bd0f2:::benchmark_data_smithi006_132277_object112638:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e329)", 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:02.692441+0000", 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 133.692480026, 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93081251600000003, 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 245095 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.466 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692441+0000", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692441+0000", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692442+0000", 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999997e-07 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.467 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692446+0000", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7029999999999997e-06 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692457+0000", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0667e-05 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692466+0000", 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3349999999999996e-06 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.468 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692525+0000", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8279999999999998e-05 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:02.692547+0000", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2569e-05 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.613847+0000", 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92129989099999998 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.469 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.613964+0000", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011664599999999999 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.614094+0000", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012984800000000001 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.615406+0000", 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001312084 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.470 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.616564+0000", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00115796 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623201+0000", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0066374219999999996 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623228+0000", 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7290000000000001e-05 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.471 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:03.623253+0000", 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4910999999999999e-05 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246581 9.3 9:c377580e:::benchmark_data_smithi006_132277_object114124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.479046+0000", 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.90587465900001, 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0025585960000001, 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.472 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246581 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479046+0000", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479046+0000", 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479047+0000", 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0999999999999998e-07 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479053+0000", 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4200000000000004e-06 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479060+0000", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0119999999999999e-06 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479068+0000", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9659999999999996e-06 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479111+0000", 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2882e-05 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479117+0000", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0000000000000002e-06 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.477115+0000", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99799790399999999 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.477267+0000", 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000152466 2024-03-26T04:05:16.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.477420+0000", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000152406 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481499+0000", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0040786290000000003 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481511+0000", 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2183e-05 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481533+0000", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1818e-05 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481563+0000", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0260000000000002e-05 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481605+0000", 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1940000000000002e-05 2024-03-26T04:05:16.478 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246582 9.3 9:cd1589a7:::benchmark_data_smithi006_132277_object114125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.479225+0000", 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.905696001, 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.009383248, 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.479 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246582 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479225+0000", 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479225+0000", 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479226+0000", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.37e-07 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479229+0000", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9129999999999996e-06 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479239+0000", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1400000000000006e-06 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479245+0000", 2024-03-26T04:05:16.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.759e-06 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479301+0000", 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5414000000000002e-05 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.479311+0000", 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9939999999999995e-06 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.479214+0000", 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99990363999999998 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.479321+0000", 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000106684 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.479390+0000", 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9311000000000001e-05 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481717+0000", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023267140000000001 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.482646+0000", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00092898099999999995 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488505+0000", 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058591140000000003 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488559+0000", 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3844000000000002e-05 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488608+0000", 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9002999999999998e-05 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.485 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246586 9.3 9:cd01013b:::benchmark_data_smithi006_132277_object114129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.483917+0000", 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.90100367900001, 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0071684430000001, 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246586 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483917+0000", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483917+0000", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483918+0000", 2024-03-26T04:05:16.487 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.09e-07 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483926+0000", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8050000000000007e-06 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483935+0000", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8060000000000001e-06 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483942+0000", 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6510000000000001e-06 2024-03-26T04:05:16.488 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483989+0000", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6978999999999999e-05 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.483996+0000", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.652e-06 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.480546+0000", 2024-03-26T04:05:16.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99655059599999996 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.480615+0000", 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8089000000000005e-05 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.480696+0000", 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1602999999999998e-05 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.482652+0000", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019561040000000002 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488746+0000", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060939469999999997 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.490988+0000", 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002242197 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491043+0000", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4527000000000002e-05 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491086+0000", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2577999999999998e-05 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246596 9.3 9:c6a84725:::benchmark_data_smithi006_132277_object114139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.508878+0000", 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.87604316700001, 2024-03-26T04:05:16.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98237711699999997, 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246596 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508878+0000", 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508878+0000", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508878+0000", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1099999999999995e-07 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508882+0000", 2024-03-26T04:05:16.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6799999999999999e-06 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508893+0000", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0647e-05 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508900+0000", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508957+0000", 2024-03-26T04:05:16.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7031999999999999e-05 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.508967+0000", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6460000000000005e-06 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481744+0000", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97277709599999995 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481815+0000", 2024-03-26T04:05:16.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0903999999999997e-05 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.481915+0000", 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100072 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487373+0000", 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054574920000000004 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488782+0000", 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001409357 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491147+0000", 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002365369 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491205+0000", 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8128000000000003e-05 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.498 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491255+0000", 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9239000000000001e-05 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246616 9.3 9:cab25676:::benchmark_data_smithi006_132277_object114159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.525530+0000", 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.859391061, 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96582670900000001, 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.499 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246616 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525530+0000", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525530+0000", 2024-03-26T04:05:16.500 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525530+0000", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8599999999999998e-07 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525534+0000", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.63e-06 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525540+0000", 2024-03-26T04:05:16.501 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4520000000000002e-06 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525545+0000", 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4939999999999997e-06 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525589+0000", 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4447000000000003e-05 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.502 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.525599+0000", 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1880000000000008e-06 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.482682+0000", 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95708369800000004 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.482747+0000", 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4498000000000005e-05 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.503 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.482826+0000", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8740999999999997e-05 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487377+0000", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045511479999999997 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488812+0000", 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00143483 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.504 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491294+0000", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024822379999999999 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491323+0000", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9311000000000001e-05 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491356+0000", 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3348e-05 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.505 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246619 9.3 9:ccc694b3:::benchmark_data_smithi006_132277_object114162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.526135+0000", 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.85878579300001, 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96537013800000004, 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246619 2024-03-26T04:05:16.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526135+0000", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526135+0000", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526136+0000", 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5000000000000001e-07 2024-03-26T04:05:16.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526140+0000", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9990000000000002e-06 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526146+0000", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6080000000000001e-06 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526152+0000", 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8479999999999997e-06 2024-03-26T04:05:16.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526195+0000", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2673999999999998e-05 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.526202+0000", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3300000000000001e-06 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.483891+0000", 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95768915600000004 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.509 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.483992+0000", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100332 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.484100+0000", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010778200000000001 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488619+0000", 2024-03-26T04:05:16.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045195510000000001 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488832+0000", 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021337 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491405+0000", 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025727739999999999 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491462+0000", 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6835e-05 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491505+0000", 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3028999999999997e-05 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246637 9.3 9:c1455c5b:::benchmark_data_smithi006_132277_object114180:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.550322+0000", 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.83459901000001, 2024-03-26T04:05:16.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94291641299999995, 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246637 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550322+0000", 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550322+0000", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550323+0000", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1999999999999998e-07 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550328+0000", 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1209999999999998e-06 2024-03-26T04:05:16.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550341+0000", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3750999999999999e-05 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550350+0000", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3850000000000002e-06 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550406+0000", 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6200999999999999e-05 2024-03-26T04:05:16.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.550414+0000", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3580000000000003e-06 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.485110+0000", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93469511999999999 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.485232+0000", 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000122638 2024-03-26T04:05:16.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,1", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.485352+0000", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120073 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.488626+0000", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0032735300000000002 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491557+0000", 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029311469999999998 2024-03-26T04:05:16.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493162+0000", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016048460000000001 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493201+0000", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9322000000000001e-05 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493238+0000", 2024-03-26T04:05:16.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7101000000000003e-05 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:253223 9.1 9:86bc532c:::benchmark_data_smithi006_132277_object120766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:12.585351+0000", 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 123.79956996200001, 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99906476300000002, 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:05:16.519 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 253223 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585351+0000", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585351+0000", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:16.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585352+0000", 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.049e-06 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585359+0000", 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6260000000000003e-06 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585368+0000", 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0739999999999993e-06 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.521 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585375+0000", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9940000000000003e-06 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585497+0000", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012234299999999999 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:12.585507+0000", 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.045e-05 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.580888+0000", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99538042900000001 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.581014+0000", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012584400000000001 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 4", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.581146+0000", 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013246800000000001 2024-03-26T04:05:16.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.583270+0000", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002124328 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.584321+0000", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001050734 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.584355+0000", 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3985000000000001e-05 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:16.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:13.584416+0000", 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0439000000000002e-05 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:16.525 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:05:16.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:16.592 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:16.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:16.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:16.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:16.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.856 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:16.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:16.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:16.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:17.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:17.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:17.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:17.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:17.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:17.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:17.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:17.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:17.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:17.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 79 16 154551 154535 122.236 82.8125 0.00578869 0.00817624 2024-03-26T04:05:17.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:17.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:17.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:17.545 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:17.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:17.686 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:17.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:17.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:17.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:05:17.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:17.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:17.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:17.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:18.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:18.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:17 smithi086 ceph-mon[32965]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:18.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.115 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:18.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:17 smithi006 ceph-mon[37501]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:17 smithi006 ceph-mon[29446]: pgmap v778: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:18.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:18.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:18.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:18.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:18.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:05:18.386804+0000 min lat: 0.00114192 max lat: 0.196773 avg lat: 0.00819479 2024-03-26T04:05:18.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:05:18.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 80 16 156154 156138 121.961 100.188 0.00454985 0.00819479 2024-03-26T04:05:18.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:18.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:18.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:18.535 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:18.602 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:18 smithi006 podman[167532]: 2024-03-26 04:05:18.351914743 +0000 UTC m=+0.564598944 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:05:18.603 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:18 smithi006 podman[167532]: 2024-03-26 04:05:18.476510611 +0000 UTC m=+0.689194804 container create 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T04:05:18.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T04:05:18.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:18.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.678 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:18.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:18.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:18.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:18.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:18.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:18.946 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:18.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:18.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:18.969 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:19.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:19.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:19.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:18 smithi006 podman[167532]: 2024-03-26 04:05:18.966445847 +0000 UTC m=+1.179130051 container init 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-26T04:05:19.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:18 smithi006 podman[167532]: 2024-03-26 04:05:18.972660257 +0000 UTC m=+1.185344451 container start 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:05:19.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 podman[167532]: 2024-03-26 04:05:19.074582014 +0000 UTC m=+1.287266210 container attach 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T04:05:19.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:19.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:19.330 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:19.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:19.382 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:05:19.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 81 16 157477 157461 121.476 82.6875 0.024841 0.00822837 2024-03-26T04:05:19.423 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:05:19.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:19.456 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:05:19.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:19.496 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:05:19.526 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:05:19.549 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:05:19.579 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:05:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:05:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:05:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:05:19.579 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:05:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:05:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:05:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:05:19.580 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:05:19.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:19.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:19.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T04:05:19.835 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 bash[167532]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-26T04:05:19.836 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate[167680]: --> ceph-volume raw activate successful for osd ID: 2 2024-03-26T04:05:19.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:19.891 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:19.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:19.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:20.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:20.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:20.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:19 smithi086 ceph-mon[32965]: pgmap v779: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2874545780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:20.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:19 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 76194/455574 objects degraded (16.725%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:20.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:20.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[37501]: pgmap v779: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:20.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2874545780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:20.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 76194/455574 objects degraded (16.725%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[29446]: pgmap v779: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.3 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 82 MiB/s wr, 1.30k op/s; 76194/455574 objects degraded (16.725%) 2024-03-26T04:05:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2874545780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:19 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 76194/455574 objects degraded (16.725%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:20.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:20.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:20.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:20.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:20.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:20.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 82 16 158917 158901 121.092 90 0.00378618 0.00825428 2024-03-26T04:05:20.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.447 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:20.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:20.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:20.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:20.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:20.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:20.722 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:20.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:20.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:20.863 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:20.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:05:20.873 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:05:20.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:21.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:21.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:21.117 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:20 smithi006 podman[167532]: 2024-03-26 04:05:20.911553722 +0000 UTC m=+3.124237912 container died 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T04:05:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:21.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:21.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:21.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 83 16 160644 160628 120.933 107.938 0.0291631 0.008265 2024-03-26T04:05:21.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:21.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:21.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:21.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:21.594 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:05:21.595 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:05:21.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:21.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.697 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:21.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:21.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:21.838 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:21 smithi006 podman[167929]: 2024-03-26 04:05:21.509199792 +0000 UTC m=+1.820451538 container remove 263acbe0469857c35c7dcf998fc4a6feaf4eebadd2cabbbb3ceca211e0feeb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-activate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:05:21.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:21.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:21.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:22.008 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:21 smithi086 ceph-mon[32965]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s; 78358/466116 objects degraded (16.811%) 2024-03-26T04:05:22.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:22.105 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:22.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:22.112 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:21 smithi006 ceph-mon[37501]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s; 78358/466116 objects degraded (16.811%) 2024-03-26T04:05:22.112 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:21 smithi006 ceph-mon[29446]: pgmap v780: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.5 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s; 78358/466116 objects degraded (16.811%) 2024-03-26T04:05:22.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:22.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:22.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:22.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T04:05:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:22.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:22.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 84 16 162331 162315 120.749 105.438 0.00520047 0.00827587 2024-03-26T04:05:22.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:22.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:22.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:22.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:22.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:22.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:22.757 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:22.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:22.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:22.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:22.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:05:22.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:22.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:22.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:23.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:23.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:23.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:23.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:22 smithi006 podman[168065]: 2024-03-26 04:05:22.745435333 +0000 UTC m=+0.642555492 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:05:23.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:23.225 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:23.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:23.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:23.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:23.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 85 16 163805 163789 120.412 92.125 0.00255155 0.00830016 2024-03-26T04:05:23.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:23.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:23.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:23.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:23.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:23.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 podman[168065]: 2024-03-26 04:05:23.354862155 +0000 UTC m=+1.251982302 container create 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-26T04:05:23.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:23.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:23.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:23.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T04:05:23.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:23.881 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:05:23.924 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:23.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:24.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:24.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[37501]: pgmap v781: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.7 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 79716/476943 objects degraded (16.714%) 2024-03-26T04:05:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4161398321' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[29446]: pgmap v781: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.7 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 79716/476943 objects degraded (16.714%) 2024-03-26T04:05:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4161398321' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 podman[168065]: 2024-03-26 04:05:23.667674477 +0000 UTC m=+1.564794632 container init 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:05:24.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 podman[168065]: 2024-03-26 04:05:23.671382666 +0000 UTC m=+1.568502812 container start 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:05:24.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 ceph-osd[168207]: -- 172.21.15.6:0/3898062814 <== mon.0 v2:172.21.15.6:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f8c5eff680 con 0x55f8c5ec9000 2024-03-26T04:05:24.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 bash[168065]: 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f 2024-03-26T04:05:24.141 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:23 smithi006 systemd[1]: Started Ceph osd.2 for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:05:24.156 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:24.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:23 smithi086 ceph-mon[32965]: pgmap v781: 158 pgs: 1 active+clean+scrubbing+deep, 31 active+undersized+degraded, 51 active+undersized, 75 active+clean; 9.7 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 79716/476943 objects degraded (16.714%) 2024-03-26T04:05:24.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:23 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4161398321' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.158 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:23 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:05:24.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:24.202 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:24.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:24.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:24.386 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 86 16 165298 165282 120.097 93.3125 0.010127 0.00832283 2024-03-26T04:05:24.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:24.479 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:24.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:24.616 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:24.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:24.639 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:05:24.207+0000 7fd416d4a7c0 -1 Falling back to public interface 2024-03-26T04:05:24.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:24.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:24.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:24.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:24.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:24.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:24.890 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:24.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:24.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:25.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:25.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: osdmap e375: 8 total, 7 up, 5 in 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 79716/476943 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: osdmap e376: 8 total, 7 up, 5 in 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:25.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: osdmap e375: 8 total, 7 up, 5 in 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 79716/476943 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: osdmap e376: 8 total, 7 up, 5 in 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:05:25.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: osdmap e375: 8 total, 7 up, 5 in 2024-03-26T04:05:25.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 79716/476943 objects degraded (16.714%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:25.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: osdmap e376: 8 total, 7 up, 5 in 2024-03-26T04:05:25.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:25.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:25.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:25.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:25.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:25.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:25.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:25.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:25.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 87 16 167107 167091 120.016 113.062 0.00653921 0.00832652 2024-03-26T04:05:25.414 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:25.438 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:25.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:25.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:25.564 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:25.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:25.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:25.681 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:25.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:25.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:25.851 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:25.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:25.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:25.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:25.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:26.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:26.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:25 smithi006 ceph-mon[37501]: pgmap v784: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 9.9 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 82198/489033 objects degraded (16.808%) 2024-03-26T04:05:26.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:25 smithi006 ceph-mon[29446]: pgmap v784: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 9.9 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 82198/489033 objects degraded (16.808%) 2024-03-26T04:05:26.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:26.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:25 smithi086 ceph-mon[32965]: pgmap v784: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 9.9 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 82198/489033 objects degraded (16.808%) 2024-03-26T04:05:26.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:26.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:26.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 88 16 168754 168738 119.822 102.938 0.00741806 0.00834186 2024-03-26T04:05:26.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:26.421 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:05:26.421 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:05:26.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:26.540 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:26.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:26.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:26.792 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:26.833 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:26.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:26.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:05:26.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:26.995 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:26.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:27.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:27.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:27.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:27.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:27.168 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:27.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T04:05:27.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:27.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:27.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:27.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:27.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 89 16 170384 170368 119.62 101.875 0.0142264 0.00835577 2024-03-26T04:05:27.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.418 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:27.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:27.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:27.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:27.631 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:05:27.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:27.711 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:27.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:27.842 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:27 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:05:27.649+0000 7fd416d4a7c0 -1 osd.2 335 log_to_monitors true 2024-03-26T04:05:27.844 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:27.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:27.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:27.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:27.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:27.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:28.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:27 smithi086 ceph-mon[32965]: pgmap v785: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:27 smithi086 ceph-mon[32965]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:05:28.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:28.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:28.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:27 smithi006 ceph-mon[29446]: pgmap v785: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:27 smithi006 ceph-mon[29446]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:05:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:27 smithi006 ceph-mon[37501]: pgmap v785: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 119 MiB/s wr, 1.90k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:27 smithi006 ceph-mon[37501]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:05:28.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:05:28.014+0000 7fd4081a2700 -1 osd.2 377 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:05:28.208 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:05:28.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:28.249 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:05:28.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:28.289 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:05:28.329 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:05:28.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:28.364 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total time run: 90.0072 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total writes made: 172092 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Write size: 65536 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object size: 65536 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Bandwidth (MB/sec): 119.499 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Bandwidth: 34.4997 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max bandwidth (MB/sec): 270.688 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min bandwidth (MB/sec): 82.4375 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average IOPS: 1911 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev IOPS: 551.995 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max IOPS: 4331 2024-03-26T04:05:28.387 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min IOPS: 1319 2024-03-26T04:05:28.388 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average Latency(s): 0.00836461 2024-03-26T04:05:28.388 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Latency(s): 0.00862392 2024-03-26T04:05:28.388 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max latency(s): 0.196773 2024-03-26T04:05:28.388 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min latency(s): 0.00114192 2024-03-26T04:05:28.389 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:05:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:28.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:28.392 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T04:05:28.424 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:05:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:05:28.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:28.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:28.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:28.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:28.669 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:28.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:28.806 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:28.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:28.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:28.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:28.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:29.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:29.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: osdmap e377: 8 total, 7 up, 5 in 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T04:05:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:29.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/869086321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:29.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:29.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:29.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:29.322 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:29.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: osdmap e377: 8 total, 7 up, 5 in 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/869086321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:05:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: osdmap e377: 8 total, 7 up, 5 in 2024-03-26T04:05:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: from='osd.2 v2:172.21.15.6:6809/2571159281' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi006", "root=default"]}]: dispatch 2024-03-26T04:05:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:29.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/869086321' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:29.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:29.447 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:29.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:29.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:29.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.726 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:29.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:29.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:29.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:29.857 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:05:29.857 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:05:29.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:05:29.857 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215934 8.9 8:9b840472:::benchmark_data_smithi006_103169_object86003:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 460.51753787199999, 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9143611719999996, 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215934 2024-03-26T04:05:29.858 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337352+0000", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337353+0000", 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.313e-06 2024-03-26T04:05:29.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337361+0000", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6170000000000001e-06 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337370+0000", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9700000000000005e-06 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337379+0000", 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.7999999999999993e-06 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337438+0000", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8943000000000001e-05 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337451+0000", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2081e-05 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.758498+0000", 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4210475499999999 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.761407+0000", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029086649999999999 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.854995+0000", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.093588355999999998 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.857945+0000", 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002950275 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.192696+0000", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.33475033199999998 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.195884+0000", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003187835 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.201543+0000", 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056593379999999999 2024-03-26T04:05:29.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.223734+0000", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022191072999999999 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.240769+0000", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017035136999999999 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.241861+0000", 2024-03-26T04:05:29.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001091987 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.244275+0000", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002413512 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.245980+0000", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001704999 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.251713+0000", 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0057333890000000002 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.865 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215951 8.9 8:9b9ea6e8:::benchmark_data_smithi006_103169_object86020:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e215)", 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 460.51742927999999, 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9678481669999996, 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215951 2024-03-26T04:05:29.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337460+0000", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337461+0000", 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2099999999999996e-07 2024-03-26T04:05:29.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337464+0000", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0759999999999999e-06 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337470+0000", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9819999999999996e-06 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337476+0000", 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2060000000000004e-06 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337537+0000", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0900999999999998e-05 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:49.337548+0000", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0538e-05 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.768786+0000", 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4312377959999996 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.771602+0000", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028160630000000002 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.869578+0000", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.097976928000000005 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.872463+0000", 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028841299999999999 2024-03-26T04:05:29.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.262539+0000", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.39007594200000001 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.265078+0000", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025392700000000002 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.269554+0000", 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044757060000000003 2024-03-26T04:05:29.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.288933+0000", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019379251 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.289948+0000", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010155539999999999 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.290994+0000", 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010457400000000001 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.298118+0000", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0071244020000000002 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.299493+0000", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013744079999999999 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.305309+0000", 2024-03-26T04:05:29.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058156529999999996 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215967 8.9 8:95a535a3:::benchmark_data_smithi006_103169_object86036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.37344975899998, 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.625424701, 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.874 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215967 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481440+0000", 2024-03-26T04:05:29.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481442+0000", 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.747e-06 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481455+0000", 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3691e-05 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481468+0000", 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2838e-05 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.481478+0000", 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6439999999999995e-06 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.771896+0000", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.290418598 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.774772+0000", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002875739 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.791709+0000", 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016937124000000001 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.794579+0000", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028692420000000001 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.065543+0000", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.27096404400000002 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.067925+0000", 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023821580000000001 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.072181+0000", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042559770000000002 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.094623+0000", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022442397999999999 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.095642+0000", 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010186749999999999 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.097572+0000", 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001930576 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.100437+0000", 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028645430000000002 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.101719+0000", 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012818549999999999 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.106865+0000", 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051458520000000002 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:215972 8.9 8:96cf40d7:::benchmark_data_smithi006_103169_object86041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.36400848900001, 2024-03-26T04:05:29.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6251629169999999, 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 215972 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.884 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490881+0000", 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490882+0000", 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5900000000000005e-07 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490886+0000", 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1099999999999996e-06 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490896+0000", 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4099999999999997e-06 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.490903+0000", 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8150000000000006e-06 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.775083+0000", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.284179795 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.778005+0000", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029216049999999999 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.805286+0000", 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.027281401 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.808134+0000", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028481230000000001 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.083016+0000", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.27488142500000001 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.085364+0000", 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023482640000000001 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.889 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.089565+0000", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042006719999999999 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.099908+0000", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010342953 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.107116+0000", 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0072084599999999999 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.108024+0000", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090759 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.109164+0000", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00114007 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.110733+0000", 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015694420000000001 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.116044+0000", 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0053108230000000001 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216001 8.9 8:9303bcb3:::benchmark_data_smithi006_103169_object86070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.29430022499997, 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8087078779999999, 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.892 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216001 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560589+0000", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560590+0000", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0359999999999999e-06 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560595+0000", 2024-03-26T04:05:29.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4379999999999997e-06 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560604+0000", 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1959999999999997e-06 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.560609+0000", 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.31e-06 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.895 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.778310+0000", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2177009320000001 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.781165+0000", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028549529999999999 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.883967+0000", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.10280199399999999 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.886904+0000", 2024-03-26T04:05:29.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002937042 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.312272+0000", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.425368146 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.314685+0000", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024124049999999998 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.318932+0000", 2024-03-26T04:05:29.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042475569999999999 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.341790+0000", 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.022857747000000001 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.358289+0000", 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016499092 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.898 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.360596+0000", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023069330000000002 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.361690+0000", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010932889999999999 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.363266+0000", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015763660000000001 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.369297+0000", 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060314419999999997 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216007 8.9 8:9fef8228:::benchmark_data_smithi006_103169_object86076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.285420219, 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5957876789999998, 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.900 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216007 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569469+0000", 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569470+0000", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.02e-06 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569476+0000", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2979999999999999e-06 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569488+0000", 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1822e-05 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.569496+0000", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6859999999999995e-06 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.809721+0000", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2402245949999999 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.812495+0000", 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027742750000000001 2024-03-26T04:05:29.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.826749+0000", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.014254195000000001 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.829604+0000", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002855064 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.123487+0000", 2024-03-26T04:05:29.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.29388293300000001 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.125888+0000", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002401065 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.130205+0000", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043169109999999997 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.155137+0000", 2024-03-26T04:05:29.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024931531 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.155665+0000", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00052799200000000002 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.156456+0000", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00079165599999999996 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.157703+0000", 2024-03-26T04:05:29.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012468920000000001 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.159481+0000", 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001777947 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.165257+0000", 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.005775797 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216021 8.9 8:9cfec2e3:::benchmark_data_smithi006_103169_object86090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.26656591800003, 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5884961029999998, 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216021 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588324+0000", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588325+0000", 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0640000000000001e-06 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588329+0000", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4660000000000001e-06 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588340+0000", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0958000000000001e-05 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.588349+0000", 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6109999999999994e-06 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.813568+0000", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.225218693 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.816433+0000", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002865795 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.840953+0000", 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024520088999999998 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.843796+0000", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028421879999999998 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.142384+0000", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.298587936 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.144752+0000", 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023685759999999998 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.149021+0000", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042687619999999997 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.165580+0000", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.016558661999999998 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.166607+0000", 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010269770000000001 2024-03-26T04:05:29.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.167951+0000", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013445040000000001 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.169092+0000", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011412499999999999 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.170635+0000", 2024-03-26T04:05:29.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001542615 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.176820+0000", 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0061849569999999996 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216026 8.9 8:9d812808:::benchmark_data_smithi006_103169_object86095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.26018639799997, 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8086321540000001, 2024-03-26T04:05:29.915 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216026 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594703+0000", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594704+0000", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.085e-06 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594709+0000", 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3279999999999999e-06 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594716+0000", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7929999999999998e-06 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.594723+0000", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9759999999999998e-06 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.888541+0000", 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.293817561 2024-03-26T04:05:29.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.891566+0000", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030245459999999999 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.907254+0000", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015688698000000001 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.910169+0000", 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029149839999999998 2024-03-26T04:05:29.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.376477+0000", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.46630807899999999 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.379058+0000", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0025804579999999999 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.383452+0000", 2024-03-26T04:05:29.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043940469999999999 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.393537+0000", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.010085213000000001 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.394133+0000", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059573000000000004 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.394972+0000", 2024-03-26T04:05:29.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00083899799999999998 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.396250+0000", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001277893 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.397862+0000", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001612589 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.403335+0000", 2024-03-26T04:05:29.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054731759999999997 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216052 8.9 8:9883e2bd:::benchmark_data_smithi006_103169_object86121:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.17830860999999, 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7761615759999998, 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.923 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216052 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676581+0000", 2024-03-26T04:05:29.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676582+0000", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6299999999999993e-07 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676585+0000", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4410000000000002e-06 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676595+0000", 2024-03-26T04:05:29.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2390000000000008e-06 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.676602+0000", 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9020000000000002e-06 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.892626+0000", 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2160241689999998 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.895561+0000", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029351799999999999 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.921143+0000", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.025582229000000001 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.924014+0000", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028710910000000001 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.411625+0000", 2024-03-26T04:05:29.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.48761071099999997 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.414079+0000", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002453616 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.418292+0000", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042136939999999996 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.441672+0000", 2024-03-26T04:05:29.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0233796 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.442545+0000", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00087358399999999995 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.443631+0000", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001085997 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.444972+0000", 2024-03-26T04:05:29.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001340911 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.446539+0000", 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001566743 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.452743+0000", 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0062035060000000001 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.930 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216058 8.9 8:907609bc:::benchmark_data_smithi006_103169_object86127:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e216)", 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 459.16028998299998, 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7699544309999999, 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216058 2024-03-26T04:05:29.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694600+0000", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694601+0000", 2024-03-26T04:05:29.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2099999999999995e-07 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694605+0000", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.267e-06 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694615+0000", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0131e-05 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:50.694622+0000", 2024-03-26T04:05:29.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6200000000000001e-06 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.930307+0000", 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2356848830000002 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.933185+0000", 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002878219 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.951929+0000", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.018744690000000001 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.954809+0000", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002879716 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.430784+0000", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.47597526899999998 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.433190+0000", 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0024058730000000002 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.437392+0000", 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042020180000000001 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.444178+0000", 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006786177 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.453940+0000", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0097615389999999996 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.455080+0000", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011395349999999999 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.456458+0000", 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001378867 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.458538+0000", 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002080087 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.464554+0000", 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060156189999999998 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216060 8.9 8:911f1304:::benchmark_data_smithi006_103169_object86129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:05:29.938 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 458.47907238099998, 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.139370537, 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216060 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375817+0000", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375818+0000", 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0500000000000003e-07 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375821+0000", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1700000000000001e-06 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375827+0000", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2890000000000003e-06 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.375832+0000", 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4750000000000004e-06 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.934247+0000", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5584147580000001 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.937162+0000", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029153260000000002 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.967070+0000", 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029908481000000001 2024-03-26T04:05:29.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.970070+0000", 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00300008 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.472966+0000", 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.50289568900000003 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.475312+0000", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002345499 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.479421+0000", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00410955 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.504198+0000", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024776461 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.505405+0000", 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012075269999999999 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.506060+0000", 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00065503600000000001 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.507216+0000", 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011562530000000001 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.945 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.509194+0000", 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019774689999999999 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.515188+0000", 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059938689999999998 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216086 8.9 8:95431b70:::benchmark_data_smithi006_103169_object86155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:05:29.946 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 458.44584083699999, 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1168013650000002, 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216086 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:05:29.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409049+0000", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409050+0000", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4600000000000003e-07 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409054+0000", 2024-03-26T04:05:29.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5990000000000004e-06 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409065+0000", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0984e-05 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.409073+0000", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7829999999999999e-06 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.937475+0000", 2024-03-26T04:05:29.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5284018110000002 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.940431+0000", 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029556610000000001 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.982173+0000", 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.041742082 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.985166+0000", 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002992818 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.490278+0000", 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.50511281900000005 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.493071+0000", 2024-03-26T04:05:29.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027928800000000002 2024-03-26T04:05:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.498275+0000", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052038709999999997 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.506403+0000", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0081282049999999995 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.516183+0000", 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0097795950000000003 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.517490+0000", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013068489999999999 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.518574+0000", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010838829999999999 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.520234+0000", 2024-03-26T04:05:29.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016606259999999999 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.525850+0000", 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056159529999999999 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216096 8.9 8:96b54a59:::benchmark_data_smithi006_103169_object86165:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 458.40091676399999, 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1427821639999998, 2024-03-26T04:05:29.955 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216096 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453973+0000", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453974+0000", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1800000000000005e-07 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453977+0000", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.422e-06 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453985+0000", 2024-03-26T04:05:29.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1950000000000003e-06 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:51.453990+0000", 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7589999999999997e-06 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.985562+0000", 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5315718939999998 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.988447+0000", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002884583 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.004127+0000", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015680555999999998 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.006941+0000", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002814212 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.541439+0000", 2024-03-26T04:05:29.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.534497577 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.544323+0000", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028842519999999999 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.548578+0000", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042547619999999996 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.568435+0000", 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019856780000000001 2024-03-26T04:05:29.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.585897+0000", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017462485999999999 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.587101+0000", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012040950000000001 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.589232+0000", 2024-03-26T04:05:29.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002131029 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.590881+0000", 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001648341 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.596755+0000", 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0058744030000000003 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216125 8.9 8:9586e5b7:::benchmark_data_smithi006_103169_object86194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 457.42891061500001, 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2384351320000002, 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216125 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.963 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425979+0000", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425980+0000", 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2399999999999996e-07 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425984+0000", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8089999999999999e-06 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425992+0000", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3410000000000003e-06 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.425997+0000", 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2959999999999998e-06 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.989600+0000", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5636028149999999 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:53.992579+0000", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029790649999999999 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.019346+0000", 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026766503000000001 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.022354+0000", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0030078790000000002 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.611057+0000", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.58870329300000002 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.613926+0000", 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028689359999999999 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.618353+0000", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0044269069999999999 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.637880+0000", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.019527013999999999 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.653783+0000", 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015903098000000001 2024-03-26T04:05:29.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.654853+0000", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010697739999999999 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.656744+0000", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001891055 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.658330+0000", 2024-03-26T04:05:29.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001585927 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.664414+0000", 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.006084496 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216143 8.9 8:9157ba64:::benchmark_data_smithi006_103169_object86212:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 457.38489587399999, 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2590048, 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.970 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216143 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469994+0000", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469995+0000", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1209999999999999e-06 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.469999+0000", 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9949999999999999e-06 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.470011+0000", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1939999999999999e-05 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.470018+0000", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0940000000000002e-06 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.022753+0000", 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.55273555 2024-03-26T04:05:29.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.025701+0000", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00294751 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.041508+0000", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.015806875000000001 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.044487+0000", 2024-03-26T04:05:29.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029794130000000002 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.678561+0000", 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.63407396900000002 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.681234+0000", 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0026731569999999998 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.685340+0000", 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0041054120000000001 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.714580+0000", 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.029240051999999999 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.719298+0000", 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0047185259999999998 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.720597+0000", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001298565 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.722146+0000", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015487260000000001 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.723755+0000", 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0016092509999999999 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.728999+0000", 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0052436440000000004 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216152 8.9 8:9b1864ad:::benchmark_data_smithi006_103169_object86221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 457.30394380299998, 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2448713520000001, 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.978 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216152 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550946+0000", 2024-03-26T04:05:29.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550947+0000", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.031e-06 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550952+0000", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8289999999999998e-06 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550962+0000", 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0088000000000001e-05 2024-03-26T04:05:29.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:52.550968+0000", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5930000000000002e-06 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.026438+0000", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.475469897 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.029344+0000", 2024-03-26T04:05:29.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029057240000000002 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.055621+0000", 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.026276711000000001 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.058605+0000", 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029844210000000001 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.743804+0000", 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.68519932800000005 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.746158+0000", 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023533019999999998 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.750518+0000", 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0043598029999999998 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.781267+0000", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.030749739000000002 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.785314+0000", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0040463369999999997 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.786421+0000", 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001107547 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.788464+0000", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020427940000000001 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.790342+0000", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001877952 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.795817+0000", 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0054752560000000004 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216241 8.9 8:9d5583f9:::benchmark_data_smithi006_103169_object86310:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 455.44443227199997, 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1741758330000001, 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216241 2024-03-26T04:05:29.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410457+0000", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410458+0000", 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1300000000000001e-07 2024-03-26T04:05:29.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410460+0000", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.322e-06 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410465+0000", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9049999999999996e-06 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.410469+0000", 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8210000000000003e-06 2024-03-26T04:05:29.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.464870+0000", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.054400873000000002 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:54.466194+0000", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001324684 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.015018+0000", 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.54882358899999995 2024-03-26T04:05:29.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.017951+0000", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0029328179999999998 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.545176+0000", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.52722475800000002 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.547506+0000", 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023308679999999998 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.551679+0000", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004172327 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.574926+0000", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.023247431999999998 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.575425+0000", 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049900400000000005 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.576344+0000", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00091846699999999996 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.577460+0000", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001116314 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.578946+0000", 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0014856380000000001 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:55.584633+0000", 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056876000000000001 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:216821 8.9 8:9c89ad88:::benchmark_data_smithi006_103169_object86890:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e218)", 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 453.69417671100001, 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1976572089999999, 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:29.993 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 216821 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160713+0000", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160714+0000", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4200000000000005e-07 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160717+0000", 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3670000000000001e-06 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160725+0000", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8220000000000007e-06 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.160730+0000", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3730000000000001e-06 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.195041+0000", 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.034310933000000002 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.196339+0000", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00129762 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.745873+0000", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.54953434800000001 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:56.748690+0000", 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028172319999999998 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.316439+0000", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.56774893599999998 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.318912+0000", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002472798 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.323900+0000", 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049881480000000004 2024-03-26T04:05:29.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.348124+0000", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.024223176999999999 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.349422+0000", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012988159999999999 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.349961+0000", 2024-03-26T04:05:29.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000539008 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.350738+0000", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00077698199999999995 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.352285+0000", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001546541 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:57:57.358370+0000", 2024-03-26T04:05:30.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0060852659999999998 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:218800 8.9 8:9f4e53b9:::benchmark_data_smithi006_103169_object88869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e220)", 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 447.66235117100001, 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1631977849999999, 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:30.001 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 218800 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192538+0000", 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:30.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192540+0000", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.062e-06 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192544+0000", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5340000000000001e-06 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192553+0000", 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0529999999999996e-06 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.192559+0000", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1020000000000002e-06 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.237081+0000", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.044522220000000001 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.238406+0000", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013245279999999999 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.736273+0000", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.49786729099999999 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:02.739003+0000", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0027294670000000002 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.322123+0000", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.58312056099999998 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.324262+0000", 2024-03-26T04:05:30.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021390210000000001 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.328111+0000", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003849037 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.345681+0000", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017569544999999999 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.346173+0000", 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00049171700000000002 2024-03-26T04:05:30.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.347475+0000", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001302115 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.348740+0000", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012655310000000001 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.350131+0000", 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013907139999999999 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:03.355736+0000", 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0056052869999999996 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15039.0:219696 8.9 8:9e2d1e9d:::benchmark_data_smithi006_103169_object89765:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e223)", 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 444.43067563300002, 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.143167109, 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:05:30.008 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15039", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 219696 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424214+0000", 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424215+0000", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5899999999999995e-07 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424218+0000", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2030000000000001e-06 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424226+0000", 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7889999999999995e-06 2024-03-26T04:05:30.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.424231+0000", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3229999999999997e-06 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.475754+0000", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.051523173999999998 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for missing object", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.476935+0000", 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011807969999999999 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.992045+0000", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.51511001700000003 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for rw locks", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:05.994655+0000", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002610064 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.537108+0000", 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.542452657 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.539400+0000", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022919250000000002 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,3,4", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.543629+0000", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0042293779999999998 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.551304+0000", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0076750179999999996 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.559469+0000", 2024-03-26T04:05:30.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008164391 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.560475+0000", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001006175 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.561404+0000", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000928833 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.562774+0000", 2024-03-26T04:05:30.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001370263 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T03:58:06.567381+0000", 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046072429999999996 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:05:30.015 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:05:30.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:30.039 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:30.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:30.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:30.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:30.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:30.283 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: pgmap v787: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:30.283 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:05:30.283 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:05:30.284 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: osd.2 v2:172.21.15.6:6809/2571159281 boot 2024-03-26T04:05:30.284 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: osdmap e378: 8 total, 8 up, 5 in 2024-03-26T04:05:30.284 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 83685/501132 objects degraded (16.699%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:30.284 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:29 smithi086 ceph-mon[32965]: osdmap e379: 8 total, 8 up, 5 in 2024-03-26T04:05:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: pgmap v787: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:05:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:05:30.352 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: osd.2 v2:172.21.15.6:6809/2571159281 boot 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: osdmap e378: 8 total, 8 up, 5 in 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 83685/501132 objects degraded (16.699%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[37501]: osdmap e379: 8 total, 8 up, 5 in 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: pgmap v787: 158 pgs: 31 active+undersized+degraded, 51 active+undersized, 76 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.34k op/s; 83685/501132 objects degraded (16.699%) 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: osd.2 v2:172.21.15.6:6809/2571159281 boot 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: osdmap e378: 8 total, 8 up, 5 in 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 83685/501132 objects degraded (16.699%), 31 pgs degraded, 82 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:30.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:29 smithi006 ceph-mon[29446]: osdmap e379: 8 total, 8 up, 5 in 2024-03-26T04:05:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-26T04:05:30.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-26T04:05:30.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:30.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:30.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:30.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:30.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:30.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:30.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T04:05:30.887 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:30.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:30.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:30.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:30.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15540.0:173806 10.f e380/379 10:f4c268ee:::benchmark_data_smithi006_159841_object1711:head v 380'10876, mlcod=380'10874)", 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:30.993268+0000", 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0095737760000000009, 2024-03-26T04:05:31.005 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0096285260000000001, 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993268+0000", 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993268+0000", 2024-03-26T04:05:31.006 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993269+0000", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3200000000000005e-07 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993275+0000", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2179999999999996e-06 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993284+0000", 2024-03-26T04:05:31.007 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1220000000000001e-06 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993291+0000", 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.2899999999999999e-06 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993376+0000", 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5357000000000003e-05 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:05:31.008 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.993399+0000", 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2711999999999999e-05 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15540.0:173803 10.7 e380/379 10:ebd88be7:::benchmark_data_smithi006_159841_object1708:head v 380'10921, mlcod=380'10918)", 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:30.995033+0000", 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0078094619999999997, 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0079909500000000001, 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:05:31.009 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995033+0000", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995033+0000", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995033+0000", 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.65e-07 2024-03-26T04:05:31.010 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995041+0000", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7789999999999996e-06 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995048+0000", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8650000000000003e-06 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995053+0000", 2024-03-26T04:05:31.011 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6359999999999999e-06 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.996276+0000", 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0012231360000000001 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.996298+0000", 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.1087e-05 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.012 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15540.0:173809 10.3 e380/379 10:cce3b9c8:::benchmark_data_smithi006_159841_object1714:head v 380'10841, mlcod=380'10839)", 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:30.995088+0000", 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0077537989999999996, 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0080254479999999993, 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995088+0000", 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.013 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995088+0000", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995089+0000", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.97e-07 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995096+0000", 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.481e-06 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995101+0000", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4729999999999999e-06 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.995106+0000", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8459999999999999e-06 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.996806+0000", 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001699943 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.996831+0000", 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4810000000000001e-05 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15540.0:173816 10.9 e380/379 10:91673733:::benchmark_data_smithi006_159841_object1721:head v 380'10887, mlcod=380'10885)", 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:30.997274+0000", 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0055682830000000003, 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0059243389999999998, 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:05:31.016 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997274+0000", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997274+0000", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997275+0000", 2024-03-26T04:05:31.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8999999999999995e-07 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997284+0000", 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0950000000000002e-06 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997293+0000", 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7710000000000001e-06 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:05:31.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997300+0000", 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7959999999999998e-06 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.997990+0000", 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00069009799999999999 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.998017+0000", 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6910000000000002e-05 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.15540.0:173820 10.9 e380/379 10:94f94994:::benchmark_data_smithi006_159841_object1725:head v 380'10888, mlcod=380'10885)", 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:31.000625+0000", 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.0022174930000000001, 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026865209999999999, 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000625+0000", 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000625+0000", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000625+0000", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6199999999999997e-07 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000634+0000", 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5709999999999998e-06 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000642+0000", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0299999999999994e-06 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.000648+0000", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1480000000000003e-06 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.002306+0000", 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00165822 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:05:31.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:31.002397+0000", 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1057000000000006e-05 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 5 2024-03-26T04:05:31.023 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:05:31.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:31.096 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:31.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:31.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:31.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:31.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:31.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:31.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:31.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:31.506 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:31.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:31.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:31.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:31.773 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:31.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:31.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:31.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:31.774 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10845 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: osdmap e380: 8 total, 8 up, 5 in 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10807 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 377'10814 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10890 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 377'10834 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11065 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10840 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 377'10842 2024-03-26T04:05:31.775 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[37501]: pgmap v791: 158 pgs: 2 remapped+peering, 22 active+undersized+degraded, 36 active+undersized, 98 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.5 KiB/s rd, 30 MiB/s wr, 584 op/s; 62958/505806 objects degraded (12.447%) 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10845 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: osdmap e380: 8 total, 8 up, 5 in 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10807 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 377'10814 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10890 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 377'10834 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11065 2024-03-26T04:05:31.776 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10840 2024-03-26T04:05:31.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 377'10842 2024-03-26T04:05:31.777 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:31 smithi006 ceph-mon[29446]: pgmap v791: 158 pgs: 2 remapped+peering, 22 active+undersized+degraded, 36 active+undersized, 98 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.5 KiB/s rd, 30 MiB/s wr, 584 op/s; 62958/505806 objects degraded (12.447%) 2024-03-26T04:05:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.9 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10845 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: osdmap e380: 8 total, 8 up, 5 in 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.3 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10807 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.e starting backfill to osd.2 from (0'0,0'0] MAX to 377'10814 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10890 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.f starting backfill to osd.2 from (0'0,0'0] MAX to 377'10834 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.5 starting backfill to osd.2 from (0'0,0'0] MAX to 377'11065 2024-03-26T04:05:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.8 starting backfill to osd.2 from (0'0,0'0] MAX to 377'10840 2024-03-26T04:05:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 377'10842 2024-03-26T04:05:31.839 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:31 smithi086 ceph-mon[32965]: pgmap v791: 158 pgs: 2 remapped+peering, 22 active+undersized+degraded, 36 active+undersized, 98 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.5 KiB/s rd, 30 MiB/s wr, 584 op/s; 62958/505806 objects degraded (12.447%) 2024-03-26T04:05:31.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:31.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:31.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:31.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:32.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:32.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:32.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:32.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:32.181 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:32.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:32.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:32.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:32.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:05:32.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:32.472 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:32.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:32.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:32.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:32.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:32.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:32.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:32.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:32.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:32.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:32.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:33.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:33.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:33.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:32 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:32 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:33.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:33.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:33.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:33.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:33.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:32 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:33.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:33.462 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:33.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:33.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T04:05:33.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:33.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:33.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:33.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:33.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:33.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:33.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:33.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:34.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.054 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:34.136 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:34.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:33 smithi006 ceph-mon[29446]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 7 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_wait, 13 active+undersized, 130 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.39k op/s; 64240/512538 objects degraded (12.534%); 646 KiB/s, 10 objects/s recovering 2024-03-26T04:05:34.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:33 smithi006 ceph-mon[37501]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 7 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_wait, 13 active+undersized, 130 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.39k op/s; 64240/512538 objects degraded (12.534%); 646 KiB/s, 10 objects/s recovering 2024-03-26T04:05:34.141 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:34.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:34.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:34.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:33 smithi086 ceph-mon[32965]: pgmap v792: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 remapped+peering, 7 active+undersized+degraded, 3 active+undersized+degraded+remapped+backfill_wait, 13 active+undersized, 130 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.39k op/s; 64240/512538 objects degraded (12.534%); 646 KiB/s, 10 objects/s recovering 2024-03-26T04:05:34.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.371 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:34.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:34.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:34.569 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:34.601 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:34.645 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:34.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:34.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:34.900 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:34.937 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:34.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:34.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:35.039 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:35.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:35.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 64240/512538 objects degraded (12.534%), 13 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4289248159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 64240/512538 objects degraded (12.534%), 13 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4289248159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:34 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.179 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:35.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 64240/512538 objects degraded (12.534%), 13 pgs degraded, 20 pgs undersized (PG_DEGRADED) 2024-03-26T04:05:35.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:34 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4289248159' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.183 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:34 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:05:35.222 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:35.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:35.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:35.355 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:35.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:35.456 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:35.497 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:35.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:35.532 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:05:35.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:35.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:35.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:35.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:35.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:35.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:35.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:35.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:35.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:36.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:36.042 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[29446]: pgmap v793: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 55 MiB/s wr, 1.43k op/s; 84108/506931 objects degraded (16.592%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:05:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:05:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[29446]: osdmap e381: 8 total, 8 up, 5 in 2024-03-26T04:05:36.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:36.147 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[37501]: pgmap v793: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 55 MiB/s wr, 1.43k op/s; 84108/506931 objects degraded (16.592%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:05:36.147 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:36.147 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:05:36.147 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:35 smithi006 ceph-mon[37501]: osdmap e381: 8 total, 8 up, 5 in 2024-03-26T04:05:36.184 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:36.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.332 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:35 smithi086 ceph-mon[32965]: pgmap v793: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 55 MiB/s wr, 1.43k op/s; 84108/506931 objects degraded (16.592%); 7.8 MiB/s, 125 objects/s recovering 2024-03-26T04:05:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:35 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:05:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:35 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:05:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:35 smithi086 ceph-mon[32965]: osdmap e381: 8 total, 8 up, 5 in 2024-03-26T04:05:36.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:36.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.490 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:36.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:36.633 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:36.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:36.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:36.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:36.789 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:05:36.832 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:05:36.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:36.872 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:05:36.909 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:36.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:36.911 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:36.912 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:05:36.954 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:05:36.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:36.955 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/519241222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:36.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:36.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/519241222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:36.997 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:05:37.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:37.040 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:37.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:37.042 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:05:37.082 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:05:37.091 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:05:37.142 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:37.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.183 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:37.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:37.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:37.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/519241222' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:05:37.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:37.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:37.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.598 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:37.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:37.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:37.707 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:37.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:37.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:37.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:37.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:37.873 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:37.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:37.974 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:38.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:38.113 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:38.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:37 smithi006 ceph-mon[37501]: pgmap v795: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 45 MiB/s wr, 1.91k op/s; 80242/491115 objects degraded (16.339%); 16 MiB/s, 249 objects/s recovering 2024-03-26T04:05:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:37 smithi006 ceph-mon[29446]: pgmap v795: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 45 MiB/s wr, 1.91k op/s; 80242/491115 objects degraded (16.339%); 16 MiB/s, 249 objects/s recovering 2024-03-26T04:05:38.148 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:38.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:38.273 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:38.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:37 smithi086 ceph-mon[32965]: pgmap v795: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 45 MiB/s wr, 1.91k op/s; 80242/491115 objects degraded (16.339%); 16 MiB/s, 249 objects/s recovering 2024-03-26T04:05:38.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:38.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:38.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:38.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:38.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:38.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:38.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:38.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:38.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:38.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:38.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:38.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:38.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:38.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:38.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:39.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:39.090 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:39.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:39.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:38 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T04:05:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:38 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T04:05:39.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:38 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:05:39.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:39.377 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:39.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:39.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T04:05:39.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T04:05:39.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:05:39.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T04:05:39.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T04:05:39.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:38 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:05:39.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:39.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:39.650 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:39.701 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:39.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:39.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:39.847 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:39.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:39.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:39.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:40.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:40.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:40.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:40.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:40.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:40.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: pgmap v796: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 25 MiB/s wr, 1.40k op/s; 80242/491115 objects degraded (16.339%); 14 MiB/s, 219 objects/s recovering 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: 1.9 scrub starts 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: 1.9 scrub ok 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 80242/491115 objects degraded (16.339%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: osdmap e382: 8 total, 8 up, 5 in 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:39 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:40.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: pgmap v796: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 25 MiB/s wr, 1.40k op/s; 80242/491115 objects degraded (16.339%); 14 MiB/s, 219 objects/s recovering 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: 1.9 scrub starts 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: 1.9 scrub ok 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 80242/491115 objects degraded (16.339%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: osdmap e382: 8 total, 8 up, 5 in 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: pgmap v796: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 10 GiB data, 29 GiB used, 418 GiB / 447 GiB avail; 25 MiB/s wr, 1.40k op/s; 80242/491115 objects degraded (16.339%); 14 MiB/s, 219 objects/s recovering 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: 1.9 scrub starts 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: 1.9 scrub ok 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 80242/491115 objects degraded (16.339%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: osdmap e382: 8 total, 8 up, 5 in 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:40.392 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:39 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:40.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:40.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:40.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:40.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:40.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:40.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.684 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:40.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:40.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:40.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:40.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:40.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:40.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:40.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:05:41.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:41.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.094 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:41.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:41.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:41.218 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:41.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:41.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:41.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:41.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:41.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:41.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:41.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:41.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:41.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:41.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:41.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:41.751 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:41.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:41.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:41.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:41.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:42.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:42.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:42.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:42.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.301 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:42.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:41 smithi086 ceph-mon[32965]: pgmap v798: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.8 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 78431/483921 objects degraded (16.207%); 21 MiB/s, 333 objects/s recovering 2024-03-26T04:05:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:41 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T04:05:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:41 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T04:05:42.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[37501]: pgmap v798: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.8 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 78431/483921 objects degraded (16.207%); 21 MiB/s, 333 objects/s recovering 2024-03-26T04:05:42.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T04:05:42.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T04:05:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[29446]: pgmap v798: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.8 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 0 B/s wr, 1.19k op/s; 78431/483921 objects degraded (16.207%); 21 MiB/s, 333 objects/s recovering 2024-03-26T04:05:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T04:05:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:41 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T04:05:42.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:42.438 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:42.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:42.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:42.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:42.712 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:42.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:42.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:42.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:42.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:42.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:42.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:43.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:43.103 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:43.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:43.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:43.231 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:43.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:42 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T04:05:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:42 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T04:05:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.372 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:43.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:43.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:42 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T04:05:43.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:42 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T04:05:43.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:42 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T04:05:43.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:42 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T04:05:43.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:43.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:43.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:43.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:43.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:43.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:43.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:43.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:43.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:44.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:44.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:44.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:43 smithi006 ceph-mon[37501]: pgmap v799: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.5 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.59k op/s; 74544/468636 objects degraded (15.907%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:05:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:43 smithi006 ceph-mon[29446]: pgmap v799: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.5 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.59k op/s; 74544/468636 objects degraded (15.907%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:05:44.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:44.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:44.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:44.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:44.313 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:43 smithi086 ceph-mon[32965]: pgmap v799: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.5 GiB data, 28 GiB used, 419 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.59k op/s; 74544/468636 objects degraded (15.907%); 24 MiB/s, 386 objects/s recovering 2024-03-26T04:05:44.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:44.444 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:44.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:44.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:44.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:44.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:44.626 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:44.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:44.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:44.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:44.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:44.856 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:44.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:44.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:44.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:44.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:45.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:45.134 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:45.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:44 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:45.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:44 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:45.194 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 74544/468636 objects degraded (15.907%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:45.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:45.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:45.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:45.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:45.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:45.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:45.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 74544/468636 objects degraded (15.907%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:45.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:45.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:45.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 74544/468636 objects degraded (15.907%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:45.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:45.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:45.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:45.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:45.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:45.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:45.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:45.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:45.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:45.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:45.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:45.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:45.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:45.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:45.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:46.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:46.085 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:46.187 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:46.220 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:46.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:46.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:46.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: pgmap v800: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 72863/462246 objects degraded (15.763%); 27 MiB/s, 436 objects/s recovering 2024-03-26T04:05:46.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: osdmap e383: 8 total, 8 up, 5 in 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T04:05:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:45 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T04:05:46.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:46.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: pgmap v800: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 72863/462246 objects degraded (15.763%); 27 MiB/s, 436 objects/s recovering 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: osdmap e383: 8 total, 8 up, 5 in 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: pgmap v800: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.4 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.61k op/s; 72863/462246 objects degraded (15.763%); 27 MiB/s, 436 objects/s recovering 2024-03-26T04:05:46.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: osdmap e383: 8 total, 8 up, 5 in 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T04:05:46.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:45 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T04:05:46.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:46.494 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:46.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:46.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:46.628 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:46.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:46.681 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:05:46.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:46.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.868 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:46.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:46.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:46.992 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:46.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:47.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:47.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:47.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:47.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:47.273 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:47.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:47.417 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:47.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:47.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:47.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:47.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:47.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:47.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:47.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:47.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:47.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:47.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:47.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:48.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.107 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:48.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:48.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:48.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:47 smithi086 ceph-mon[32965]: pgmap v802: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 69301/448104 objects degraded (15.465%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:05:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:47 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:47 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:48.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:48.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:48.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[37501]: pgmap v802: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 69301/448104 objects degraded (15.465%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[29446]: pgmap v802: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.79k op/s; 69301/448104 objects degraded (15.465%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:48.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:47 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:48.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:48.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:48.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:48.640 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:48.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:48.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:48.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:48.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:48.908 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:48.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:49.009 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:49.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:49.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:49.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:49.195 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:49.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:48 smithi086 ceph-mon[32965]: osdmap e384: 8 total, 8 up, 5 in 2024-03-26T04:05:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:48 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:05:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:48 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:05:49.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[29446]: osdmap e384: 8 total, 8 up, 5 in 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[37501]: osdmap e384: 8 total, 8 up, 5 in 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:05:49.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:48 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:05:49.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:49.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:49.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:49.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:49.610 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:49.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:49.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:49.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:49.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:49.875 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:49.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:49.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:49.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:50.007 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:50.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:50.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:50.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:50.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:50.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:49 smithi086 ceph-mon[32965]: pgmap v804: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s; 69301/448104 objects degraded (15.465%); 25 MiB/s, 399 objects/s recovering 2024-03-26T04:05:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 69301/448104 objects degraded (15.465%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:50.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:50.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:49 smithi006 ceph-mon[29446]: pgmap v804: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s; 69301/448104 objects degraded (15.465%); 25 MiB/s, 399 objects/s recovering 2024-03-26T04:05:50.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 69301/448104 objects degraded (15.465%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:49 smithi006 ceph-mon[37501]: pgmap v804: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.1 GiB data, 27 GiB used, 420 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s; 69301/448104 objects degraded (15.465%); 25 MiB/s, 399 objects/s recovering 2024-03-26T04:05:50.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 69301/448104 objects degraded (15.465%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:50.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:50.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:50.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:50.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:50.690 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:50.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:50.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:50.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:50.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:50.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:50.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:50.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:51.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:51.099 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:51.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:05:51.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:05:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:05:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:05:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:51 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:51.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:51.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:51.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:51 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:05:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:51 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:05:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:51 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:05:51.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:51.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:51.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:51.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:51.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:51.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.751 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:51.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:51.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:51.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:51.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:51.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:51.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:52.022 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:52.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:52.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:52.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:52.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:52.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:52.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:52 smithi086 ceph-mon[32965]: pgmap v805: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 67526/441255 objects degraded (15.303%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:05:52.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:52.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:52 smithi006 ceph-mon[29446]: pgmap v805: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 67526/441255 objects degraded (15.303%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:05:52.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:52 smithi006 ceph-mon[37501]: pgmap v805: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 9.0 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.24k op/s; 67526/441255 objects degraded (15.303%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:05:52.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:52.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:52.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:52.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:52.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.701 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:52.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:52.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:52.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.841 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:52.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:52.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:52.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:52.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:53.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:53.101 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:53.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:53.234 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:53.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:53.375 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:53.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:53.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:53.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:53.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:53.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:53.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:53.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:53.745 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:53.778 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:53.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:53.924 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:53.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:54.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:54.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:54.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:54.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:54.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:54.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[37501]: pgmap v806: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 693 B/s rd, 0 B/s wr, 1.64k op/s; 63827/425910 objects degraded (14.986%); 27 MiB/s, 426 objects/s recovering 2024-03-26T04:05:54.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T04:05:54.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T04:05:54.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[29446]: pgmap v806: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 693 B/s rd, 0 B/s wr, 1.64k op/s; 63827/425910 objects degraded (14.986%); 27 MiB/s, 426 objects/s recovering 2024-03-26T04:05:54.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T04:05:54.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T04:05:54.272 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:54.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:54.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:54 smithi086 ceph-mon[32965]: pgmap v806: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 693 B/s rd, 0 B/s wr, 1.64k op/s; 63827/425910 objects degraded (14.986%); 27 MiB/s, 426 objects/s recovering 2024-03-26T04:05:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:54 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T04:05:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:54 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T04:05:54.374 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:05:54.374 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:05:54.418 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:54.463 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:54.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:54.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:54.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:54.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:05:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:05:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:05:54.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:54.744 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:54.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:54.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:54.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:54.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:54.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:54.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:55.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:55.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:55.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:55.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 63827/425910 objects degraded (14.986%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 63827/425910 objects degraded (14.986%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:55.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:55.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:55.289 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:55 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:05:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:55 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:05:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:55 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 63827/425910 objects degraded (14.986%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:05:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:05:55.338 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:05:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:05:55.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:55.428 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:55.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:55.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:55.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:55.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T04:05:55.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:55.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:55.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:55.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:55.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:55.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:55.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:55.998 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:55.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:56.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:56.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[37501]: pgmap v807: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 556 B/s rd, 0 B/s wr, 1.32k op/s; 63827/425910 objects degraded (14.986%); 21 MiB/s, 341 objects/s recovering 2024-03-26T04:05:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:05:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:05:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[29446]: pgmap v807: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 556 B/s rd, 0 B/s wr, 1.32k op/s; 63827/425910 objects degraded (14.986%); 21 MiB/s, 341 objects/s recovering 2024-03-26T04:05:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:05:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:56 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:05:56.233 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:56.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:56 smithi086 ceph-mon[32965]: pgmap v807: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.7 GiB data, 26 GiB used, 421 GiB / 447 GiB avail; 556 B/s rd, 0 B/s wr, 1.32k op/s; 63827/425910 objects degraded (14.986%); 21 MiB/s, 341 objects/s recovering 2024-03-26T04:05:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:56 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:05:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:56 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:05:56.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:56.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:56.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:56.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:56.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:56.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:56.692 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:56.785 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:05:56.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:56.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:56.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:56.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:56.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:56.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:57.039 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T04:05:57.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:57.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:57.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:57.262 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:57 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:05:57.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:57.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:57 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:05:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:57 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:05:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:05:57.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:57.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:57.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:57.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:57.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:57.677 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:57.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.820 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:57.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:57.960 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:57.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:57.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:58.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:58.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:58.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:58.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:58.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.231 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:58.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:58.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:05:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:58 smithi086 ceph-mon[32965]: pgmap v808: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.55k op/s; 58606/404370 objects degraded (14.493%); 27 MiB/s, 431 objects/s recovering 2024-03-26T04:05:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:05:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:58 smithi086 ceph-mon[32965]: osdmap e385: 8 total, 8 up, 5 in 2024-03-26T04:05:58.363 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:05:58.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[29446]: pgmap v808: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.55k op/s; 58606/404370 objects degraded (14.493%); 27 MiB/s, 431 objects/s recovering 2024-03-26T04:05:58.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:05:58.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[29446]: osdmap e385: 8 total, 8 up, 5 in 2024-03-26T04:05:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[37501]: pgmap v808: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.55k op/s; 58606/404370 objects degraded (14.493%); 27 MiB/s, 431 objects/s recovering 2024-03-26T04:05:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3277216466' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:05:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:58 smithi006 ceph-mon[37501]: osdmap e385: 8 total, 8 up, 5 in 2024-03-26T04:05:58.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:05:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.519 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:58.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:05:58.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:05:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.662 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:58.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:05:58.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:05:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:58.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:58.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:05:58.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:59.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.008 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:59.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:59.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:05:59.071 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:05:59.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:05:59.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:59.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:05:59.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:59.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:59 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:05:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:59 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:05:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:59 smithi086 ceph-mon[32965]: osdmap e386: 8 total, 8 up, 5 in 2024-03-26T04:05:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:05:59 smithi086 ceph-mon[32965]: pgmap v811: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s; 58606/404370 objects degraded (14.493%); 24 MiB/s, 383 objects/s recovering 2024-03-26T04:05:59.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[29446]: osdmap e386: 8 total, 8 up, 5 in 2024-03-26T04:05:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[29446]: pgmap v811: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s; 58606/404370 objects degraded (14.493%); 24 MiB/s, 383 objects/s recovering 2024-03-26T04:05:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:05:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:05:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[37501]: osdmap e386: 8 total, 8 up, 5 in 2024-03-26T04:05:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:05:59 smithi006 ceph-mon[37501]: pgmap v811: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.2 GiB data, 25 GiB used, 422 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.54k op/s; 58606/404370 objects degraded (14.493%); 24 MiB/s, 383 objects/s recovering 2024-03-26T04:05:59.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:05:59.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:05:59.530 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:59.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:59.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:59.673 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:05:59.710 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:05:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:05:59.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:59.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:05:59.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:05:59.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:05:59.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:05:59.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:00.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.114 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:00.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:00.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.250 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:00.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:00 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:06:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:00 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:06:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:00 smithi086 ceph-mon[32965]: osdmap e387: 8 total, 8 up, 5 in 2024-03-26T04:06:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:00 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 58606/404370 objects degraded (14.493%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:00 smithi086 ceph-mon[32965]: osdmap e388: 8 total, 8 up, 5 in 2024-03-26T04:06:00.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:00.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:00.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[29446]: osdmap e387: 8 total, 8 up, 5 in 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 58606/404370 objects degraded (14.493%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[29446]: osdmap e388: 8 total, 8 up, 5 in 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[37501]: osdmap e387: 8 total, 8 up, 5 in 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 58606/404370 objects degraded (14.493%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:00 smithi006 ceph-mon[37501]: osdmap e388: 8 total, 8 up, 5 in 2024-03-26T04:06:00.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:00.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:00.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:00.647 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:00.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:00.749 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:00.785 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:00.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:00.922 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:00.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:00.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:01.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:01.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:01.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:01.190 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:01.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:01.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:01.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:01.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:01 smithi006 ceph-mon[37501]: pgmap v814: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 568 op/s; 57121/397566 objects degraded (14.368%); 13 MiB/s, 200 objects/s recovering 2024-03-26T04:06:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:01 smithi006 ceph-mon[29446]: pgmap v814: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 568 op/s; 57121/397566 objects degraded (14.368%); 13 MiB/s, 200 objects/s recovering 2024-03-26T04:06:01.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:01.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:01.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:01.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:01.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:01 smithi086 ceph-mon[32965]: pgmap v814: 158 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 8.1 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 568 op/s; 57121/397566 objects degraded (14.368%); 13 MiB/s, 200 objects/s recovering 2024-03-26T04:06:01.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.597 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:01.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:01.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:01.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T04:06:01.740 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:01.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:01.807 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:01.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:01.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:01.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:01.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:02.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:02.103 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:02.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:02.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:02.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:02.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:02.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:02.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:02 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T04:06:02.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:02 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T04:06:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:02 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T04:06:02.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:02 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T04:06:02.402 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:02.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:02.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:02.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:02 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T04:06:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:02 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T04:06:02.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:02.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:02.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:02.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:02.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:02.953 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:02.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:03.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:03.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:03.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:03.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:03.321 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:03.347 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:03.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:03.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T04:06:03.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T04:06:03.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[29446]: pgmap v815: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 424 GiB / 447 GiB avail; 998 B/s rd, 0 B/s wr, 1.43k op/s; 53493/382416 objects degraded (13.988%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T04:06:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T04:06:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:03 smithi006 ceph-mon[37501]: pgmap v815: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 424 GiB / 447 GiB avail; 998 B/s rd, 0 B/s wr, 1.43k op/s; 53493/382416 objects degraded (13.988%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:03.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:03.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:03 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T04:06:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:03 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T04:06:03.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:03 smithi086 ceph-mon[32965]: pgmap v815: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 424 GiB / 447 GiB avail; 998 B/s rd, 0 B/s wr, 1.43k op/s; 53493/382416 objects degraded (13.988%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:03.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:03.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:03.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:03.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:03.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:03.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:03.889 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:03.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:03.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:03.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:04.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:04.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:06:04.085 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:06:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:06:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:06:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:06:04.086 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:06:04.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:04.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:04.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:04.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:04.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:04.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1654874202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:04.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1654874202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:04.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:04.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:04.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:04.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:04.541 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1654874202' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:04.644 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:04.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:04.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:04.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:04.821 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:04.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:04.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:04.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:04.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:05.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:05.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:05.222 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:05.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:05.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:05.361 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:05.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:06:05.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[37501]: pgmap v816: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 840 B/s rd, 0 B/s wr, 1.20k op/s; 53493/382416 objects degraded (13.988%); 19 MiB/s, 307 objects/s recovering 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 53493/382416 objects degraded (13.988%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[29446]: pgmap v816: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 840 B/s rd, 0 B/s wr, 1.20k op/s; 53493/382416 objects degraded (13.988%); 19 MiB/s, 307 objects/s recovering 2024-03-26T04:06:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:05 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 53493/382416 objects degraded (13.988%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:05.464 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:05.500 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:05.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:05.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:05.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:05 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:06:05.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:05 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:06:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:05 smithi086 ceph-mon[32965]: pgmap v816: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 1 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.8 GiB data, 24 GiB used, 423 GiB / 447 GiB avail; 840 B/s rd, 0 B/s wr, 1.20k op/s; 53493/382416 objects degraded (13.988%); 19 MiB/s, 307 objects/s recovering 2024-03-26T04:06:05.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:05 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 53493/382416 objects degraded (13.988%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:05.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:05.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:05.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:05.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:05.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:05.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:05.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:05.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:05.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:06.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:06.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:06.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:06.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:06.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:06.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:06.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:06.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:06.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:06.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.567 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:06.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:06.669 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:06.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:06.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:06.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:06.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:06.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:06.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:06.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:07.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:07.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:07.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:07.264 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:07.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:07.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T04:06:07.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T04:06:07.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[37501]: pgmap v817: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 48781/361947 objects degraded (13.477%); 30 MiB/s, 475 objects/s recovering 2024-03-26T04:06:07.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T04:06:07.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T04:06:07.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:07 smithi006 ceph-mon[29446]: pgmap v817: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 48781/361947 objects degraded (13.477%); 30 MiB/s, 475 objects/s recovering 2024-03-26T04:06:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:07.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:07.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:07.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:07 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T04:06:07.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:07 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T04:06:07.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:07 smithi086 ceph-mon[32965]: pgmap v817: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 48781/361947 objects degraded (13.477%); 30 MiB/s, 475 objects/s recovering 2024-03-26T04:06:07.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:07.670 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:07.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:07.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:07.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:07.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:07.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:08.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.071 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:08.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:08.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:08.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:08.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:08.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:08.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:08.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:08.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:08 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T04:06:08.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:08 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T04:06:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:08 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T04:06:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:08 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T04:06:08.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:08.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:08.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:08.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:08 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T04:06:08.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:08 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T04:06:08.599 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:08.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:08.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:08.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:08.846 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:08.886 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:08.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:08.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:09.021 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:09.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:09.160 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:06:09.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:09.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:09.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:09.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.300 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:09.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[37501]: 10.c scrub starts 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[37501]: pgmap v818: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.56k op/s; 48781/361947 objects degraded (13.477%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1946999971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[29446]: 10.c scrub starts 2024-03-26T04:06:09.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[29446]: pgmap v818: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.56k op/s; 48781/361947 objects degraded (13.477%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:06:09.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1946999971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:09.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:09.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:09.575 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:09.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:09.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:09 smithi086 ceph-mon[32965]: 10.c scrub starts 2024-03-26T04:06:09.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:09 smithi086 ceph-mon[32965]: pgmap v818: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.4 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.56k op/s; 48781/361947 objects degraded (13.477%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:06:09.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1946999971' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:06:09.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:09.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:09.755 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:09.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:09.856 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:09.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:09.988 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:09.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:09.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:10.090 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:10.133 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:10.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:10.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:10.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: osdmap e389: 8 total, 8 up, 5 in 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: 10.c scrub ok 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 48781/361947 objects degraded (13.477%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: osdmap e390: 8 total, 8 up, 5 in 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:06:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: osdmap e389: 8 total, 8 up, 5 in 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: 10.c scrub ok 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 48781/361947 objects degraded (13.477%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: osdmap e390: 8 total, 8 up, 5 in 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:10.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:10.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:10.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.552 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:10.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:06:10.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:06:10.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: osdmap e389: 8 total, 8 up, 5 in 2024-03-26T04:06:10.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: 10.c scrub ok 2024-03-26T04:06:10.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 48781/361947 objects degraded (13.477%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:10.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: osdmap e390: 8 total, 8 up, 5 in 2024-03-26T04:06:10.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:10.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:10.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:10.705 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:10.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:10.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:10.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:10.987 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:10.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:11.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:11.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:11.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:11.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:11.327 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:06:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:06:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:06:11.327 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:06:11.327 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:06:11.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:11.390 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:11.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:11.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:11.533 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:11.546 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:06:11.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:11 smithi086 ceph-mon[32965]: pgmap v821: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 47380/355317 objects degraded (13.335%); 22 MiB/s, 349 objects/s recovering 2024-03-26T04:06:11.603 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:11 smithi006 ceph-mon[37501]: pgmap v821: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 47380/355317 objects degraded (13.335%); 22 MiB/s, 349 objects/s recovering 2024-03-26T04:06:11.603 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:11 smithi006 ceph-mon[29446]: pgmap v821: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 7.2 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 47380/355317 objects degraded (13.335%); 22 MiB/s, 349 objects/s recovering 2024-03-26T04:06:11.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:11.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.676 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:11.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:11.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:11.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.814 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:11.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:11.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:11.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:11.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:12.066 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:12.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:12.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:12.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:12.234 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:12.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:12.336 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:12.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:12.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:12.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:12.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:12.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:12.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:06:12.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:12.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:12.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:12.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:12.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:12.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:13.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:13.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:13.152 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:13.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:13.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:13.285 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:13.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:13.353 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:13 smithi006 ceph-mon[37501]: pgmap v822: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 44048/340476 objects degraded (12.937%); 30 MiB/s, 477 objects/s recovering 2024-03-26T04:06:13.353 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:13 smithi006 ceph-mon[29446]: pgmap v822: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 44048/340476 objects degraded (12.937%); 30 MiB/s, 477 objects/s recovering 2024-03-26T04:06:13.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:13.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:13.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:13.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:13.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:13.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:13 smithi086 ceph-mon[32965]: pgmap v822: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.75k op/s; 44048/340476 objects degraded (12.937%); 30 MiB/s, 477 objects/s recovering 2024-03-26T04:06:13.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:13.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:13.804 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:13.841 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:13.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:13.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T04:06:13.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:14.007 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:14.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:14.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:14.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:14.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:14.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:14.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:14.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:14.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:14.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:14.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.570 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:14.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:14.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:14.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:14.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:14.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:14.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:14.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:14.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:14.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:15.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:15.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:15.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:15.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:15.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:15.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:15.386 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:15.389 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:15.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:15.500 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:15.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:15.518 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:15 smithi006 ceph-mon[37501]: pgmap v823: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 44048/340476 objects degraded (12.937%); 15 MiB/s, 236 objects/s recovering 2024-03-26T04:06:15.518 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:15 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 44048/340476 objects degraded (12.937%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:15.518 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:15 smithi006 ceph-mon[29446]: pgmap v823: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 44048/340476 objects degraded (12.937%); 15 MiB/s, 236 objects/s recovering 2024-03-26T04:06:15.519 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:15 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 44048/340476 objects degraded (12.937%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:15 smithi086 ceph-mon[32965]: pgmap v823: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.9 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 894 op/s; 44048/340476 objects degraded (12.937%); 15 MiB/s, 236 objects/s recovering 2024-03-26T04:06:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:15 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 44048/340476 objects degraded (12.937%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:15.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:15.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:15.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:15.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:15.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:15.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:15.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:15.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:15.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:15.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:16.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:16.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.045 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:16.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:16.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:16.190 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.202 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:16.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:16.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:16.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.331 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:16.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:16.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:16.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:06:16.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:06:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:06:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:16 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:06:16.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:16.477 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:16.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:16.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:16 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:06:16.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:16 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:06:16.612 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:16.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:16.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:16.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.762 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:16.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:16.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:16.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:16.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:16.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:16.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:17.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.039 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:17.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:17.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:17.180 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:17.264 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T04:06:17.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:17.322 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:17.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:17 smithi006 ceph-mon[37501]: pgmap v824: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 39608/319431 objects degraded (12.400%); 29 MiB/s, 460 objects/s recovering 2024-03-26T04:06:17.423 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:17 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:06:17.423 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:17 smithi006 ceph-mon[29446]: pgmap v824: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 39608/319431 objects degraded (12.400%); 29 MiB/s, 460 objects/s recovering 2024-03-26T04:06:17.424 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:17 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:06:17.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:17.461 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:17.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:17 smithi086 ceph-mon[32965]: pgmap v824: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 39608/319431 objects degraded (12.400%); 29 MiB/s, 460 objects/s recovering 2024-03-26T04:06:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:17 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:06:17.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.594 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:17.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:17.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:17.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:17.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:17.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:17.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:17.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:17.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:17.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:18.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:18.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:18.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:18.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:18.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:18.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:18.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:18.386 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:18.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:18.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:18.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:18.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:18.559 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:18 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:06:18.559 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:18 smithi006 ceph-mon[37501]: osdmap e391: 8 total, 8 up, 5 in 2024-03-26T04:06:18.559 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:18 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:06:18.559 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:18 smithi006 ceph-mon[29446]: osdmap e391: 8 total, 8 up, 5 in 2024-03-26T04:06:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:18 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3716077124' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:06:18.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:18 smithi086 ceph-mon[32965]: osdmap e391: 8 total, 8 up, 5 in 2024-03-26T04:06:18.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:18.647 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:18.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:18.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:18.875 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:18.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:18.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:18.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:19.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:19.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:19.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:19.164 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:19.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:19.302 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:19.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:19.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:19.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:19.576 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:19 smithi086 ceph-mon[32965]: pgmap v826: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 39608/319431 objects degraded (12.400%); 27 MiB/s, 425 objects/s recovering 2024-03-26T04:06:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:19 smithi086 ceph-mon[32965]: osdmap e392: 8 total, 8 up, 5 in 2024-03-26T04:06:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:19 smithi006 ceph-mon[37501]: pgmap v826: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 39608/319431 objects degraded (12.400%); 27 MiB/s, 425 objects/s recovering 2024-03-26T04:06:19.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:19 smithi006 ceph-mon[37501]: osdmap e392: 8 total, 8 up, 5 in 2024-03-26T04:06:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:19 smithi006 ceph-mon[29446]: pgmap v826: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.5 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 39608/319431 objects degraded (12.400%); 27 MiB/s, 425 objects/s recovering 2024-03-26T04:06:19.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:19 smithi006 ceph-mon[29446]: osdmap e392: 8 total, 8 up, 5 in 2024-03-26T04:06:19.677 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:19.716 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:19.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:19.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:19.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:19.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:19.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:20.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:20.134 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:20.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:20.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:20.370 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:20.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:20.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:20.533 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:20 smithi086 ceph-mon[32965]: osdmap e393: 8 total, 8 up, 5 in 2024-03-26T04:06:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:20 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 39608/319431 objects degraded (12.400%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:20.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:20 smithi086 ceph-mon[32965]: osdmap e394: 8 total, 8 up, 5 in 2024-03-26T04:06:20.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:20.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[29446]: osdmap e393: 8 total, 8 up, 5 in 2024-03-26T04:06:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 39608/319431 objects degraded (12.400%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:20.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[29446]: osdmap e394: 8 total, 8 up, 5 in 2024-03-26T04:06:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[37501]: osdmap e393: 8 total, 8 up, 5 in 2024-03-26T04:06:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 39608/319431 objects degraded (12.400%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:20.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:20 smithi006 ceph-mon[37501]: osdmap e394: 8 total, 8 up, 5 in 2024-03-26T04:06:20.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:20.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:20.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:20.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:20.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:20.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:20.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:20.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:21.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:21.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:21.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:21.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:21.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:21.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:21.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:21.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:21.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:21.468 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:21.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:21.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.600 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:21.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:21.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:21.739 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:21.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:21 smithi086 ceph-mon[32965]: pgmap v830: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.4 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 363 op/s; 39230/315084 objects degraded (12.451%) 2024-03-26T04:06:21.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:21.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:21.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:21.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:21 smithi006 ceph-mon[37501]: pgmap v830: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.4 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 363 op/s; 39230/315084 objects degraded (12.451%) 2024-03-26T04:06:21.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:21 smithi006 ceph-mon[29446]: pgmap v830: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.4 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 363 op/s; 39230/315084 objects degraded (12.451%) 2024-03-26T04:06:21.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:22.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:22.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:22.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:22.154 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:22.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:22.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:22.299 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:22.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:22.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:22.533 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:22.571 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:22.672 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:22.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.714 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:22.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:22.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:22 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:06:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:22 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:06:22.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:22.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:22 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:06:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:22 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:06:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:22 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:06:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:22 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:06:22.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:22.975 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:22.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:22.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:23.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:23.211 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:23.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:23.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:23.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:23.523 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:23.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:23.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.653 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:23.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:23.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:23.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:23 smithi086 ceph-mon[32965]: pgmap v831: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 35502/303120 objects degraded (11.712%); 26 MiB/s, 410 objects/s recovering 2024-03-26T04:06:23.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:23 smithi006 ceph-mon[37501]: pgmap v831: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 35502/303120 objects degraded (11.712%); 26 MiB/s, 410 objects/s recovering 2024-03-26T04:06:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:23 smithi006 ceph-mon[29446]: pgmap v831: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 150 active+clean; 6.2 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.11k op/s; 35502/303120 objects degraded (11.712%); 26 MiB/s, 410 objects/s recovering 2024-03-26T04:06:23.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:23.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:23.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:23.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:24.024 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:24.062 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:24.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:24.202 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:24.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:24.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:24.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:24.479 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:24.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:24.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:24.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 35502/303120 objects degraded (11.712%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:24.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 35502/303120 objects degraded (11.712%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:24.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:24.742 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:24.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 35502/303120 objects degraded (11.712%), 8 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:24.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:24.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:24.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:24.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:25.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.018 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:25.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:25.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:25.157 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T04:06:25.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.165 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:25.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:25.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:25.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:25.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:25.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:25.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:25.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:25.561 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:25.585 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:25.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:25.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:25.716 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:25.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:25 smithi086 ceph-mon[32965]: pgmap v832: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 34969/301167 objects degraded (11.611%); 26 MiB/s, 421 objects/s recovering 2024-03-26T04:06:25.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:25.857 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:25 smithi006 ceph-mon[29446]: pgmap v832: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 34969/301167 objects degraded (11.611%); 26 MiB/s, 421 objects/s recovering 2024-03-26T04:06:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:25.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:25 smithi006 ceph-mon[37501]: pgmap v832: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 6.1 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 34969/301167 objects degraded (11.611%); 26 MiB/s, 421 objects/s recovering 2024-03-26T04:06:25.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:25.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:25.986 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:25.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:25.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:26.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:26.129 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:26.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:26.261 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:26.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:26.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.412 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:26.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:26.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:26.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:26.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:26.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.703 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:26.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215917 8.c 8:328b5576:::benchmark_data_smithi006_103169_object85986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 518.25519684699998, 2024-03-26T04:06:26.737 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9868534169999998, 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215917 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:06:26.738 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475898+0000", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.3900000000000003e-07 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475904+0000", 2024-03-26T04:06:26.739 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.344e-06 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475915+0000", 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1496e-05 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.475935+0000", 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9800999999999999e-05 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.740 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476186+0000", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025108200000000002 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476303+0000", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117278 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 5,7", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.476416+0000", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011205999999999999 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.741 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.500056+0000", 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.023640960999999999 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.500146+0000", 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9951999999999995e-05 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.516444+0000", 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.016297282999999999 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.742 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.516463+0000", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9071000000000001e-05 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337701+0000", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.82123782300000003 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337727+0000", 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6482000000000002e-05 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.743 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.462673+0000", 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1249464010000001 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.462751+0000", 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7443999999999999e-05 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215940 8.c 8:3db160f6:::benchmark_data_smithi006_103169_object86009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:06:26.744 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 518.18903272700004, 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9264022000000001, 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215940 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:06:26.745 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542062+0000", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542063+0000", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6509999999999999e-06 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542070+0000", 2024-03-26T04:06:26.746 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.737e-06 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542086+0000", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6189999999999999e-05 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542096+0000", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5009999999999993e-06 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542168+0000", 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.1810999999999994e-05 2024-03-26T04:06:26.747 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.542179+0000", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1406000000000001e-05 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337766+0000", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.79558757499999999 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.337776+0000", 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7059999999999999e-06 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.748 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463110+0000", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.125333892 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463217+0000", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010647899999999999 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.463319+0000", 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000102587 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.749 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.467958+0000", 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046384590000000002 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.750 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468011+0000", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3446000000000003e-05 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468172+0000", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016070799999999999 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468290+0000", 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118597 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.751 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468349+0000", 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8641000000000003e-05 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468464+0000", 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114814 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.752 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:215954 8.c 8:39dff541:::benchmark_data_smithi006_103169_object86023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e214)", 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 518.15888171799998, 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.898915927, 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 215954 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.753 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572213+0000", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572214+0000", 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1519999999999999e-06 2024-03-26T04:06:26.754 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572219+0000", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1200000000000001e-06 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572230+0000", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0709e-05 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572236+0000", 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9800000000000003e-06 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.755 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572296+0000", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9914999999999998e-05 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:48.572308+0000", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2189e-05 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338781+0000", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.76647347099999996 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.756 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:49.338791+0000", 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6840000000000008e-06 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.465697+0000", 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.126906169 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.465834+0000", 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000136608 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.757 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.466026+0000", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019242099999999999 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468233+0000", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022071669999999999 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468277+0000", 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3503e-05 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.758 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.468472+0000", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019535200000000001 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471025+0000", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0025533539999999999 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471063+0000", 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7477e-05 2024-03-26T04:06:26.759 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:57:50.471129+0000", 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5655999999999996e-05 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15039.0:222643 8.c 8:3896b36c:::benchmark_data_smithi006_103169_object92712:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e226)", 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 493.35502671299997, 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0007649670000001, 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.760 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15039", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2267290914", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 222643 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.761 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376068+0000", 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376069+0000", 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.703e-06 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376077+0000", 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0260000000000007e-06 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.762 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376095+0000", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7682000000000001e-05 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376103+0000", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7430000000000002e-06 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376154+0000", 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1616000000000002e-05 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.763 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:13.376161+0000", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.075e-06 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.373847+0000", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99768531100000002 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.373971+0000", 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012385099999999999 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.764 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.374098+0000", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012708399999999999 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376704+0000", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026066800000000001 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376717+0000", 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2388999999999999e-05 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.765 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376761+0000", 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3745999999999998e-05 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376793+0000", 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2396e-05 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.766 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T03:58:14.376833+0000", 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9665000000000001e-05 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221614 9.9 9:9089f597:::benchmark_data_smithi006_132277_object89157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.591189+0000", 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.13990513499999, 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.058757736, 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.767 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221614 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591189+0000", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591189+0000", 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.768 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591191+0000", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.672e-06 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591199+0000", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0509999999999997e-06 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591212+0000", 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3405e-05 2024-03-26T04:06:26.769 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591233+0000", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0741999999999999e-05 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591664+0000", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00043043999999999998 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.591806+0000", 2024-03-26T04:06:26.770 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014293300000000001 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,5,7", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.592052+0000", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00024565999999999998 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.618863+0000", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.026811343000000001 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.623931+0000", 2024-03-26T04:06:26.771 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050675149999999999 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.623942+0000", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1080000000000001e-05 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.645526+0000", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.021584313 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.645650+0000", 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012347599999999999 2024-03-26T04:06:26.772 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.649874+0000", 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0042236579999999999 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.649947+0000", 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3448000000000006e-05 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.773 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221630 9.9 9:918db384:::benchmark_data_smithi006_132277_object89173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.629475+0000", 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.10161925400001, 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0207174539999999, 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221630 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.774 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629475+0000", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629475+0000", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629477+0000", 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6449999999999999e-06 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.775 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629484+0000", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.0820000000000001e-06 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629499+0000", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5458e-05 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.629509+0000", 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4469999999999995e-06 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.776 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.630109+0000", 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00060009899999999999 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.777 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.630118+0000", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4709999999999996e-06 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.646035+0000", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0159163550000001 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.646169+0000", 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000134246 2024-03-26T04:06:26.778 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.646314+0000", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014491999999999999 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.649586+0000", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032720409999999998 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.649850+0000", 2024-03-26T04:06:26.779 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00026380899999999999 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650016+0000", 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00016616800000000001 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650080+0000", 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4412999999999995e-05 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.780 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650193+0000", 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011230000000000001 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221641 9.9 9:95fa2196:::benchmark_data_smithi006_132277_object89184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.641132+0000", 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.08996223700001, 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0094712800000001, 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.781 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221641 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641132+0000", 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.782 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641132+0000", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641134+0000", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5889999999999999e-06 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641140+0000", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7499999999999997e-06 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.783 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641163+0000", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2296000000000001e-05 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641173+0000", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0550999999999999e-05 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641273+0000", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000100113 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.641317+0000", 2024-03-26T04:06:26.784 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3735000000000001e-05 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.648341+0000", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0070233749999999 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.648424+0000", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3658000000000004e-05 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.648505+0000", 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1075000000000006e-05 2024-03-26T04:06:26.785 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650209+0000", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0017038789999999999 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650253+0000", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3569999999999998e-05 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650452+0000", 2024-03-26T04:06:26.786 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00019913299999999999 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650507+0000", 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5488999999999997e-05 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650603+0000", 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6067e-05 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.787 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221659 9.9 9:993f0436:::benchmark_data_smithi006_132277_object89202:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.659525+0000", 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.07156937299999, 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0055576850000001, 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221659 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.788 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659525+0000", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659525+0000", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659527+0000", 2024-03-26T04:06:26.789 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.699e-06 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659533+0000", 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6789999999999999e-06 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659551+0000", 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8033000000000002e-05 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.790 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659560+0000", 2024-03-26T04:06:26.791 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1950000000000003e-06 2024-03-26T04:06:26.791 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.791 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:26.791 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659665+0000", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010582900000000001 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.659675+0000", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9140000000000003e-06 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650653+0000", 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.990977683 2024-03-26T04:06:26.792 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650769+0000", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000116193 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.650885+0000", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115668 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.654989+0000", 2024-03-26T04:06:26.793 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0041040970000000001 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.660309+0000", 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0053204330000000003 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.664922+0000", 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0046125630000000001 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.794 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.664975+0000", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2679999999999997e-05 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665083+0000", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000108019 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221675 9.9 9:945b4409:::benchmark_data_smithi006_132277_object89218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.671450+0000", 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.05964448, 2024-03-26T04:06:26.795 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99377169600000004, 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221675 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671450+0000", 2024-03-26T04:06:26.796 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671450+0000", 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671451+0000", 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8800000000000002e-07 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.797 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671455+0000", 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4209999999999997e-06 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671469+0000", 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3434e-05 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671476+0000", 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8800000000000008e-06 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.798 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671705+0000", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000228636 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.671729+0000", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3604999999999998e-05 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.652834+0000", 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98110567999999998 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.799 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.652955+0000", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012105800000000001 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.653094+0000", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013877 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.660194+0000", 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0071003020000000002 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.800 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.660346+0000", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000151292 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665090+0000", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047444760000000001 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665137+0000", 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6492000000000001e-05 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.801 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665222+0000", 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4862000000000002e-05 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:221676 9.9 9:91baf93c:::benchmark_data_smithi006_132277_object89219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:02:36.672864+0000", 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 230.05823083300001, 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99249003800000002, 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.802 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 221676 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672864+0000", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672864+0000", 2024-03-26T04:06:26.803 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672865+0000", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2300000000000001e-06 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672869+0000", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.639e-06 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672885+0000", 2024-03-26T04:06:26.804 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5701000000000001e-05 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672893+0000", 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4730000000000003e-06 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672959+0000", 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6496999999999997e-05 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.805 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:36.672967+0000", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6990000000000003e-06 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.655033+0000", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98206620899999997 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.655143+0000", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000109555 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 3,7", 2024-03-26T04:06:26.806 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.655261+0000", 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011882100000000001 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.660235+0000", 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0049737599999999998 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.660385+0000", 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014970200000000001 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.807 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665228+0000", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004843506 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665269+0000", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0904e-05 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:02:37.665354+0000", 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4342e-05 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.808 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15312.0:253214 9.b 9:d4dd5d1b:::benchmark_data_smithi006_132277_object120757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:03:12.548000+0000", 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 194.18309385500001, 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0630223320000001, 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15312", 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 253214 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.809 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548000+0000", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548000+0000", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548002+0000", 2024-03-26T04:06:26.810 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5510000000000001e-06 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548008+0000", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.003e-06 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548019+0000", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0803e-05 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.548027+0000", 2024-03-26T04:06:26.811 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8169999999999999e-06 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.550021+0000", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0019939279999999998 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.550145+0000", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000124098 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 1,2", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.550264+0000", 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011938399999999999 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.812 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.733982+0000", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.18371763299999999 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.738068+0000", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040865939999999998 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:12.738078+0000", 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0064000000000001e-05 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.813 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.610932+0000", 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.87285322499999995 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:03:13.611023+0000", 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1231999999999997e-05 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173269 10.8 10:10f4ce47:::benchmark_data_smithi006_159841_object1174:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T04:06:26.814 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:28.932158+0000", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.798935995000001, 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6858283269999998, 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173269 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932158+0000", 2024-03-26T04:06:26.815 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932158+0000", 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932159+0000", 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.02e-06 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.816 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932164+0000", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1999999999999996e-06 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932172+0000", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3499999999999997e-06 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932177+0000", 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4090000000000002e-06 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.817 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.818 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.818 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932222+0000", 2024-03-26T04:06:26.818 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4348999999999998e-05 2024-03-26T04:06:26.818 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.932296+0000", 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4220000000000004e-05 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.819 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 7", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.943375+0000", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011079556000000001 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980392+0000", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037016173999999999 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980404+0000", 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2295e-05 2024-03-26T04:06:26.820 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513202+0000", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.53279807199999996 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513217+0000", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5299999999999999e-05 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606266+0000", 2024-03-26T04:06:26.821 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0930483040000001 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606369+0000", 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000103494 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617914+0000", 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.011544644 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.822 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617987+0000", 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2940000000000003e-05 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173271 10.8 10:11d54799:::benchmark_data_smithi006_159841_object1176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:28.935969+0000", 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.795125355000003, 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.681914921, 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.823 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173271 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.935969+0000", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.935969+0000", 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.824 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.935970+0000", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.297e-06 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.935976+0000", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.037e-06 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.943552+0000", 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0075755370000000002 2024-03-26T04:06:26.825 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.943562+0000", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.06e-05 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.945926+0000", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002363118 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.946223+0000", 2024-03-26T04:06:26.826 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00029760600000000001 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 7", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.946361+0000", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000137658 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980432+0000", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.034071109000000002 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980439+0000", 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2930000000000003e-06 2024-03-26T04:06:26.827 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513259+0000", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.53282018200000003 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513270+0000", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0947e-05 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606586+0000", 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.093315453 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.828 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606679+0000", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2998999999999995e-05 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617796+0000", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.01111669 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617884+0000", 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8394999999999996e-05 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.829 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173279 10.8 10:17a76eb5:::benchmark_data_smithi006_159841_object1184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:28.972820+0000", 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.758274239999999, 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.644722528, 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173279 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.830 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972820+0000", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972820+0000", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972821+0000", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3000000000000001e-07 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.831 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972824+0000", 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.4910000000000001e-06 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972832+0000", 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4270000000000002e-06 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.972838+0000", 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.613e-06 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.832 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980464+0000", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0076255289999999998 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:28.980470+0000", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5869999999999998e-06 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513302+0000", 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.53283159300000005 2024-03-26T04:06:26.833 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513312+0000", 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.7249999999999992e-06 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606708+0000", 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.093396341 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.834 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606824+0000", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000115847 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.606947+0000", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012330500000000001 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.613646+0000", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0066989750000000002 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.835 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.616497+0000", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028512350000000001 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617378+0000", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00088102899999999997 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617435+0000", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6783999999999997e-05 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.836 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.617543+0000", 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010751699999999999 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173256 10.3 10:c03fcdcc:::benchmark_data_smithi006_159841_object1161:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.513898+0000", 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.217196344000001, 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.078734211, 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.837 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173256 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513898+0000", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513898+0000", 2024-03-26T04:06:26.838 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513900+0000", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.573e-06 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513907+0000", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5259999999999996e-06 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513961+0000", 2024-03-26T04:06:26.839 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3514999999999998e-05 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.513976+0000", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.507e-05 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514228+0000", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00025214399999999999 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514237+0000", 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5599999999999999e-06 2024-03-26T04:06:26.840 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.588925+0000", 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.074687138 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.592632+0000", 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0037076850000000001 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173259 10.7 10:efb806b5:::benchmark_data_smithi006_159841_object1164:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.841 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.514007+0000", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.217087509999999, 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0850688100000001, 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173259 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514007+0000", 2024-03-26T04:06:26.842 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514007+0000", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514008+0000", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6300000000000005e-07 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514012+0000", 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.1060000000000002e-06 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.843 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514020+0000", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3799999999999994e-06 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514027+0000", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.827e-06 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514842+0000", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00081530600000000002 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.844 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514850+0000", 2024-03-26T04:06:26.845 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3599999999999996e-06 2024-03-26T04:06:26.845 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.598171+0000", 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.083320923 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.599076+0000", 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00090424500000000003 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.846 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173265 10.5 10:a69bb224:::benchmark_data_smithi006_159841_object1170:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.514043+0000", 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.217050872999998, 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0701095920000001, 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173265 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.847 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514043+0000", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514043+0000", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514044+0000", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.51e-07 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.848 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514047+0000", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9299999999999999e-06 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514054+0000", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.764e-06 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514060+0000", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0610000000000001e-06 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514537+0000", 2024-03-26T04:06:26.849 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00047699 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514545+0000", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4109999999999996e-06 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.583835+0000", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.069289602 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.584153+0000", 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00031828299999999999 2024-03-26T04:06:26.850 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173270 10.3 10:c84c453c:::benchmark_data_smithi006_159841_object1175:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.514076+0000", 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.217018121999999, 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0789048880000001, 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173270 2024-03-26T04:06:26.851 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514076+0000", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514076+0000", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514077+0000", 2024-03-26T04:06:26.852 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3600000000000004e-07 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514080+0000", 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9919999999999999e-06 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514086+0000", 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6499999999999999e-06 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.853 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514093+0000", 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.3679999999999998e-06 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514265+0000", 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00017253099999999999 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514273+0000", 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.011e-06 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.854 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.592896+0000", 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0786229089999999 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.592981+0000", 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4890999999999997e-05 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173272 10.5 10:a3afe638:::benchmark_data_smithi006_159841_object1177:head [delete] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.855 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.514101+0000", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.216993125000002, 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.074570378, 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173272 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514101+0000", 2024-03-26T04:06:26.856 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514101+0000", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514102+0000", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.9400000000000001e-07 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514104+0000", 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.5840000000000002e-06 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.857 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514110+0000", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0909999999999998e-06 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514116+0000", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1029999999999998e-06 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514571+0000", 2024-03-26T04:06:26.858 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00045490299999999999 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514579+0000", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4580000000000004e-06 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.584988+0000", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0704095490000001 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for missing object", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.585141+0000", 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015234499999999999 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.859 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.588602+0000", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0034616690000000001 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.588672+0000", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.9282000000000005e-05 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:173286 10.3 10:ce98a2fa:::benchmark_data_smithi006_159841_object1191:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e379)", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:05:29.514124+0000", 2024-03-26T04:06:26.860 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 57.216969853000002, 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.09579922, 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 173286 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514124+0000", 2024-03-26T04:06:26.861 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514124+0000", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514125+0000", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3799999999999998e-07 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514127+0000", 2024-03-26T04:06:26.862 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.4779999999999998e-06 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514133+0000", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0039999999999996e-06 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514139+0000", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6200000000000004e-06 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514491+0000", 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00035169500000000001 2024-03-26T04:06:26.863 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:29.514500+0000", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9369999999999993e-06 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.593016+0000", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.078516475 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.594182+0000", 2024-03-26T04:06:26.864 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0011654809999999999 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.594312+0000", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00013045400000000001 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.600783+0000", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0064712729999999996 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.601515+0000", 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00073158299999999995 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.865 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.608537+0000", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0070215950000000003 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.608660+0000", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012311700000000001 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:05:30.609924+0000", 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00126407 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.866 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.867 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:06:26.867 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:06:26.867 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:06:26.886 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:26.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:26.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:27.019 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:27.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:27.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:27.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:27.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:27.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:27.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:27.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:27.551 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:27.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:27.692 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:27 smithi006 ceph-mon[29446]: pgmap v833: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 31061/280788 objects degraded (11.062%); 32 MiB/s, 504 objects/s recovering 2024-03-26T04:06:27.693 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:27 smithi006 ceph-mon[37501]: pgmap v833: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 31061/280788 objects degraded (11.062%); 32 MiB/s, 504 objects/s recovering 2024-03-26T04:06:27.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:27.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:27.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:27.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:27 smithi086 ceph-mon[32965]: pgmap v833: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.63k op/s; 31061/280788 objects degraded (11.062%); 32 MiB/s, 504 objects/s recovering 2024-03-26T04:06:27.932 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:06:27.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:27.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:27.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:27.979 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:06:28.023 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:06:28.064 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:06:28.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:28.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:28.104 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:06:28.144 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:06:28.191 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:06:28.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:28.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:28.232 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:06:28.244 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:06:28.245 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:06:28.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:28.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:28.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:28.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T04:06:28.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:28.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:28.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:28.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:28.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:06:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:06:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/313618867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:06:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:06:28.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/313618867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:28.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:28.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:28.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:28.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:29.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:29.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:28 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:06:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:28 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:06:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/313618867' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:29.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:29.166 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:29.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:29.268 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:29.293 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:29.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:29.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:29.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:29.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:29.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:29.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:29.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:29.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:29.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:29.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:29 smithi006 ceph-mon[37501]: pgmap v834: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 31061/279489 objects degraded (11.113%); 28 MiB/s, 447 objects/s recovering 2024-03-26T04:06:29.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 31061/279489 objects degraded (11.113%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:29 smithi006 ceph-mon[29446]: pgmap v834: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 31061/279489 objects degraded (11.113%); 28 MiB/s, 447 objects/s recovering 2024-03-26T04:06:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 31061/279489 objects degraded (11.113%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:29.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:29.971 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:29.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:29.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:30.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:29 smithi086 ceph-mon[32965]: pgmap v834: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.7 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.50k op/s; 31061/279489 objects degraded (11.113%); 28 MiB/s, 447 objects/s recovering 2024-03-26T04:06:30.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 31061/279489 objects degraded (11.113%), 7 pgs degraded (PG_DEGRADED) 2024-03-26T04:06:30.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.112 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:30.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:30.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:30.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:30.385 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:30.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:30.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:30.525 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:30.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:30.650 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:06:30.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:30.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:30.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:30.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T04:06:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/274653290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:30.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:30.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T04:06:30.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/274653290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:30.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:30 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:30.921 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:30.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:30.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:31.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:31.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:31.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:30 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T04:06:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:30 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/274653290' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:30 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:06:31.156 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:31.194 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:31.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:31.285 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:31.297 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:31.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:31.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:31.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:31.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:31.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:31.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:31.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:31.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:31.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:31.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T04:06:31.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[37501]: pgmap v835: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 30613/273951 objects degraded (11.175%); 23 MiB/s, 374 objects/s recovering 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[37501]: osdmap e395: 8 total, 8 up, 5 in 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[29446]: pgmap v835: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 30613/273951 objects degraded (11.175%); 23 MiB/s, 374 objects/s recovering 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:06:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:31 smithi006 ceph-mon[29446]: osdmap e395: 8 total, 8 up, 5 in 2024-03-26T04:06:31.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:32.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:32.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:31 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T04:06:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:31 smithi086 ceph-mon[32965]: pgmap v835: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.6 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.42k op/s; 30613/273951 objects degraded (11.175%); 23 MiB/s, 374 objects/s recovering 2024-03-26T04:06:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:31 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:06:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:31 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:06:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:31 smithi086 ceph-mon[32965]: osdmap e395: 8 total, 8 up, 5 in 2024-03-26T04:06:32.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:32.144 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:32.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:32.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:32.275 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:32.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:32.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:32.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:32.541 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:32.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:32.642 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:32.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:32.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:32.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:32.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:32.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:32.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:33.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:32 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:06:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:32 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:06:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:32 smithi086 ceph-mon[32965]: osdmap e396: 8 total, 8 up, 5 in 2024-03-26T04:06:33.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:06:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:06:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[37501]: osdmap e396: 8 total, 8 up, 5 in 2024-03-26T04:06:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:06:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:06:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:32 smithi006 ceph-mon[29446]: osdmap e396: 8 total, 8 up, 5 in 2024-03-26T04:06:33.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:33.215 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.216 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:33.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:33.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:33.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:33.468 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:33.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:33.599 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:33.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:33.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:33.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:33.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:33.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:33.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:33.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:33.984 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:33.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:33.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:34.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:33 smithi086 ceph-mon[32965]: pgmap v838: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 27343/259863 objects degraded (10.522%); 23 MiB/s, 366 objects/s recovering 2024-03-26T04:06:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:33 smithi086 ceph-mon[32965]: osdmap e397: 8 total, 8 up, 5 in 2024-03-26T04:06:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:34.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:34.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:33 smithi006 ceph-mon[37501]: pgmap v838: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 27343/259863 objects degraded (10.522%); 23 MiB/s, 366 objects/s recovering 2024-03-26T04:06:34.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:33 smithi006 ceph-mon[37501]: osdmap e397: 8 total, 8 up, 5 in 2024-03-26T04:06:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:33 smithi006 ceph-mon[29446]: pgmap v838: 158 pgs: 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.3 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s; 27343/259863 objects degraded (10.522%); 23 MiB/s, 366 objects/s recovering 2024-03-26T04:06:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:33 smithi006 ceph-mon[29446]: osdmap e397: 8 total, 8 up, 5 in 2024-03-26T04:06:34.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:34.240 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:34.342 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:34.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:34.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:34.505 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:34.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:34.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:34.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:34.643 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:34.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:34.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:34.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:34.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:34.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:34.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:34.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:35.002 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:35.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.042 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:35.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:34 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:06:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:34 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:06:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:34 smithi086 ceph-mon[32965]: osdmap e398: 8 total, 8 up, 5 in 2024-03-26T04:06:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 27343/259863 objects degraded (10.522%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[37501]: osdmap e398: 8 total, 8 up, 5 in 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 27343/259863 objects degraded (10.522%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[29446]: osdmap e398: 8 total, 8 up, 5 in 2024-03-26T04:06:35.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 27343/259863 objects degraded (10.522%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:35.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.168 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:35.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:35.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:35.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:35.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:35.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:35.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:35.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:35.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:35.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:35.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:35.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:35.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:35.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:35.831 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:35.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:35.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:35.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:35.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:35.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:36.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:35 smithi086 ceph-mon[32965]: pgmap v841: 158 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 27205/258192 objects degraded (10.537%); 25 MiB/s, 403 objects/s recovering 2024-03-26T04:06:36.094 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:36.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:36.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:35 smithi006 ceph-mon[37501]: pgmap v841: 158 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 27205/258192 objects degraded (10.537%); 25 MiB/s, 403 objects/s recovering 2024-03-26T04:06:36.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:35 smithi006 ceph-mon[29446]: pgmap v841: 158 pgs: 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_wait, 151 active+clean; 5.2 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.31k op/s; 27205/258192 objects degraded (10.537%); 25 MiB/s, 403 objects/s recovering 2024-03-26T04:06:36.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:36.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:36.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:36.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:36.366 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:36.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:36.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:36.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:36.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:36.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:36.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:36.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:36.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:36.754 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:36.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:36.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:36.887 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:36.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:36.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:36.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:37.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:37.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:37.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:37.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:37.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:37.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:37.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:37.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:37.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.557 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:37.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:37.686 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:37.788 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:37.825 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:37.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:37.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:37.995 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:37 smithi006 ceph-mon[37501]: pgmap v842: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 1.76k op/s; 23593/244617 objects degraded (9.645%); 41 MiB/s, 661 objects/s recovering 2024-03-26T04:06:37.995 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:37 smithi006 ceph-mon[29446]: pgmap v842: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 1.76k op/s; 23593/244617 objects degraded (9.645%); 41 MiB/s, 661 objects/s recovering 2024-03-26T04:06:38.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:37 smithi086 ceph-mon[32965]: pgmap v842: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 peering, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 151 active+clean; 5.0 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 1.76k op/s; 23593/244617 objects degraded (9.645%); 41 MiB/s, 661 objects/s recovering 2024-03-26T04:06:38.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:38.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:38.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:38.257 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:38.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:38.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:38.398 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:38.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:38.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:38.530 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:38.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:38.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:38.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:38.788 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:38.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:38.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:38.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:38.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:38.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:39.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:39.072 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:39.173 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:39.210 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:39.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:39.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:39.345 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:39.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:39.483 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:39.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:39.613 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:39.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:39.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:39.739 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.740 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:39.841 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:39.881 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:39.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:39.982 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:40.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:40.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:40.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:06:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:39 smithi086 ceph-mon[32965]: pgmap v843: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 782 B/s rd, 0 B/s wr, 881 op/s; 23593/242577 objects degraded (9.726%); 21 MiB/s, 335 objects/s recovering 2024-03-26T04:06:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 23593/242577 objects degraded (9.726%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:40.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:39 smithi006 ceph-mon[37501]: pgmap v843: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 782 B/s rd, 0 B/s wr, 881 op/s; 23593/242577 objects degraded (9.726%); 21 MiB/s, 335 objects/s recovering 2024-03-26T04:06:40.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 23593/242577 objects degraded (9.726%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:40.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:39 smithi006 ceph-mon[29446]: pgmap v843: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.9 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 782 B/s rd, 0 B/s wr, 881 op/s; 23593/242577 objects degraded (9.726%); 21 MiB/s, 335 objects/s recovering 2024-03-26T04:06:40.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 23593/242577 objects degraded (9.726%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:40.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:40.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:40.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:40.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:40.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:40.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:40.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:40.427 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:40.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:40.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:40.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:40.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:40.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.697 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:40.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:40.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:40.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:40.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:40.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:40.955 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:40.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:41.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:41.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:41.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:41.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:41.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:41.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:41.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:41.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:41.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:41.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:41.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:41.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:41.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:41.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:41.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:41.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:41.768 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:41.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:41.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:41.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:41.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:42.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:42.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:42.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:42.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:42.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:41 smithi006 ceph-mon[37501]: pgmap v844: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 23213/238143 objects degraded (9.748%); 17 MiB/s, 273 objects/s recovering 2024-03-26T04:06:42.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:41 smithi006 ceph-mon[29446]: pgmap v844: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 23213/238143 objects degraded (9.748%); 17 MiB/s, 273 objects/s recovering 2024-03-26T04:06:42.172 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:42.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:42.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.314 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:42.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:41 smithi086 ceph-mon[32965]: pgmap v844: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.8 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 905 op/s; 23213/238143 objects degraded (9.748%); 17 MiB/s, 273 objects/s recovering 2024-03-26T04:06:42.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:42.451 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:42.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:42.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:42.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:42.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:42.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:42.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:42.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:42.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:42.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:42.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:43.004 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:43.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:43.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:43.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:43.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:43.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:43.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.283 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:43.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:43.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:43.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:43.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:43.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:43.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:43.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:43.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:43.821 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:43.922 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:43.959 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:43.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:44.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:44.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:44.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:43 smithi006 ceph-mon[37501]: pgmap v845: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 20708/226467 objects degraded (9.144%); 22 MiB/s, 345 objects/s recovering 2024-03-26T04:06:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:43 smithi006 ceph-mon[29446]: pgmap v845: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 20708/226467 objects degraded (9.144%); 22 MiB/s, 345 objects/s recovering 2024-03-26T04:06:44.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:44.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:44.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:44.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:43 smithi086 ceph-mon[32965]: pgmap v845: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.6 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 20708/226467 objects degraded (9.144%); 22 MiB/s, 345 objects/s recovering 2024-03-26T04:06:44.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:44.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:44.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:44.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:44.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:44.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:44.624 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:44.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:44.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:44.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:44.752 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:44.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:44.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:44.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:44.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:44.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:45.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:45.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:45.127 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:45.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:45.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:45.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:45.256 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:44 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 20708/226467 objects degraded (9.144%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:45.256 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:44 smithi086 ceph-mon[32965]: osdmap e399: 8 total, 8 up, 5 in 2024-03-26T04:06:45.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:45.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:45.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:44 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 20708/226467 objects degraded (9.144%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:45.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:44 smithi006 ceph-mon[37501]: osdmap e399: 8 total, 8 up, 5 in 2024-03-26T04:06:45.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:44 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 20708/226467 objects degraded (9.144%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:45.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:44 smithi006 ceph-mon[29446]: osdmap e399: 8 total, 8 up, 5 in 2024-03-26T04:06:45.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:45.512 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:45.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:45.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:45.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:45.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:45.738 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:45.777 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:45.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:45.879 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:45.902 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:45.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:45.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:46.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:46.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:46.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:46.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:46.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:46.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.310 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:46.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:45 smithi086 ceph-mon[32965]: pgmap v846: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 974 B/s rd, 0 B/s wr, 1.09k op/s; 20708/223878 objects degraded (9.250%); 19 MiB/s, 300 objects/s recovering 2024-03-26T04:06:46.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:45 smithi086 ceph-mon[32965]: osdmap e400: 8 total, 8 up, 5 in 2024-03-26T04:06:46.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:45 smithi006 ceph-mon[37501]: pgmap v846: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 974 B/s rd, 0 B/s wr, 1.09k op/s; 20708/223878 objects degraded (9.250%); 19 MiB/s, 300 objects/s recovering 2024-03-26T04:06:46.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:45 smithi006 ceph-mon[37501]: osdmap e400: 8 total, 8 up, 5 in 2024-03-26T04:06:46.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:45 smithi006 ceph-mon[29446]: pgmap v846: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 2 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.5 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 974 B/s rd, 0 B/s wr, 1.09k op/s; 20708/223878 objects degraded (9.250%); 19 MiB/s, 300 objects/s recovering 2024-03-26T04:06:46.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:45 smithi006 ceph-mon[29446]: osdmap e400: 8 total, 8 up, 5 in 2024-03-26T04:06:46.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:46.442 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:46.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:46.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:46.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:46.699 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:46.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:46.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:46.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:46.955 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:46.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:47.057 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:47.096 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:47.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:47.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:47.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:47.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:47.228 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:47.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:47.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:47.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:47.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:47.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:47.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:47.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:47.628 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:47.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:47.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:47.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:47.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:47.899 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:47.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:47.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:48.001 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:48.026 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:48.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:48.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:48.282 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:48.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:47 smithi086 ceph-mon[32965]: pgmap v849: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 17566/209490 objects degraded (8.385%); 21 MiB/s, 331 objects/s recovering 2024-03-26T04:06:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:47 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:06:48.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:48.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:47 smithi006 ceph-mon[29446]: pgmap v849: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 17566/209490 objects degraded (8.385%); 21 MiB/s, 331 objects/s recovering 2024-03-26T04:06:48.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:47 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:06:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:47 smithi006 ceph-mon[37501]: pgmap v849: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 4 active+undersized+degraded+remapped+backfilling, 152 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 17566/209490 objects degraded (8.385%); 21 MiB/s, 331 objects/s recovering 2024-03-26T04:06:48.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:47 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:06:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:48.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:48.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:48.555 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:48.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:48.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:48.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:48.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:48.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:48.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:48.951 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:48.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:49.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:49.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:49.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:49.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:49.352 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:49.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:49.486 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:49.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:49.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:49.686 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:49.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:49.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:49.762 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:49.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:49.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:49.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:49.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:50.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:50.111 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:50.148 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:50.193 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:49 smithi006 ceph-mon[37501]: pgmap v850: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 17279/205935 objects degraded (8.391%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:50.194 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:49 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 17279/205935 objects degraded (8.391%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:50.194 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:49 smithi006 ceph-mon[29446]: pgmap v850: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 17279/205935 objects degraded (8.391%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:50.194 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:49 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 17279/205935 objects degraded (8.391%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:50.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:50.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:49 smithi086 ceph-mon[32965]: pgmap v850: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.2 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.34k op/s; 17279/205935 objects degraded (8.391%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:06:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:49 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 17279/205935 objects degraded (8.391%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:50.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:50.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:50.519 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:50.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:50.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:50.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:50.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:50.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:50.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:50.869 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:50.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:50.970 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:51.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:51.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:06:51.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:51.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:51.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:51.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:50 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:06:51.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:51.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:50 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:06:51.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:50 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:06:51.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:51.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:51.542 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:51.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:51.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:51.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:51.809 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:51.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:51.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:51.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:51.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:52.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:52.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:52.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:51 smithi006 ceph-mon[37501]: pgmap v851: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15816/201654 objects degraded (7.843%); 23 MiB/s, 372 objects/s recovering 2024-03-26T04:06:52.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:51 smithi006 ceph-mon[29446]: pgmap v851: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15816/201654 objects degraded (7.843%); 23 MiB/s, 372 objects/s recovering 2024-03-26T04:06:52.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:52.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:52.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:52.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:51 smithi086 ceph-mon[32965]: pgmap v851: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 4.1 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 15816/201654 objects degraded (7.843%); 23 MiB/s, 372 objects/s recovering 2024-03-26T04:06:52.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:52.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:52.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:52.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:52.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:52.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:52.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:52.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:52.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:52.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:52.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:52.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:52.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:53.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:53.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:53.082 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:06:53.082 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:06:53.082 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:53.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:06:53.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:06:53.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:52 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:53.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:53.173 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:53.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:53.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:06:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:52 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:06:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:52 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:06:53.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:53.452 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:53.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:53.593 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:53.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:53.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:53.732 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:53.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:53.865 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:53.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:53.974 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:06:53.974 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:06:53.980 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T04:06:53.995 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:53.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:54.097 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:54.136 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:54.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[37501]: pgmap v852: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 13585/190068 objects degraded (7.147%); 31 MiB/s, 488 objects/s recovering 2024-03-26T04:06:54.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[37501]: pgmap v853: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 13585/190068 objects degraded (7.147%); 30 MiB/s, 474 objects/s recovering 2024-03-26T04:06:54.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:06:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[29446]: pgmap v852: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 13585/190068 objects degraded (7.147%); 31 MiB/s, 488 objects/s recovering 2024-03-26T04:06:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[29446]: pgmap v853: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 13585/190068 objects degraded (7.147%); 30 MiB/s, 474 objects/s recovering 2024-03-26T04:06:54.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:06:54.238 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:54.279 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:54.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:53 smithi086 ceph-mon[32965]: pgmap v852: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s; 13585/190068 objects degraded (7.147%); 31 MiB/s, 488 objects/s recovering 2024-03-26T04:06:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:53 smithi086 ceph-mon[32965]: pgmap v853: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.9 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s; 13585/190068 objects degraded (7.147%); 30 MiB/s, 474 objects/s recovering 2024-03-26T04:06:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:06:54.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:54.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:54.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:54.565 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:54.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:06:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:06:54.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:54.696 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:54.764 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T04:06:54.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:54.856 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:54.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:54.947 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:06:54.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:54.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:54.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:55.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:55.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:55.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:55.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:55.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:55.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:06:55.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:54 smithi086 ceph-mon[32965]: osdmap e401: 8 total, 8 up, 5 in 2024-03-26T04:06:55.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:54 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 13585/190068 objects degraded (7.147%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:55.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:54 smithi086 ceph-mon[32965]: osdmap e402: 8 total, 8 up, 5 in 2024-03-26T04:06:55.268 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:55.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:55.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:55.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:55.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:06:55.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[37501]: osdmap e401: 8 total, 8 up, 5 in 2024-03-26T04:06:55.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 13585/190068 objects degraded (7.147%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[37501]: osdmap e402: 8 total, 8 up, 5 in 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2354781121' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[29446]: osdmap e401: 8 total, 8 up, 5 in 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 13585/190068 objects degraded (7.147%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[29446]: osdmap e402: 8 total, 8 up, 5 in 2024-03-26T04:06:55.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:06:55.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.533 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:55.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:55.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T04:06:55.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:06:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:06:55.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:55.663 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:55.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:55.816 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:55.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:55.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:55.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:56.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:56.082 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:56.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[37501]: pgmap v856: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.26k op/s; 12515/181002 objects degraded (6.914%); 28 MiB/s, 440 objects/s recovering 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[37501]: osdmap e403: 8 total, 8 up, 5 in 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[29446]: pgmap v856: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.26k op/s; 12515/181002 objects degraded (6.914%); 28 MiB/s, 440 objects/s recovering 2024-03-26T04:06:56.089 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:55 smithi006 ceph-mon[29446]: osdmap e403: 8 total, 8 up, 5 in 2024-03-26T04:06:56.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:56.228 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:56.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:55 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:06:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:55 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:06:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:55 smithi086 ceph-mon[32965]: pgmap v856: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.7 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.26k op/s; 12515/181002 objects degraded (6.914%); 28 MiB/s, 440 objects/s recovering 2024-03-26T04:06:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:55 smithi086 ceph-mon[32965]: osdmap e403: 8 total, 8 up, 5 in 2024-03-26T04:06:56.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:56.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:56.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:56.487 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:56.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:56.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:56.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:56.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:56.760 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:56.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:56.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:56.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:56.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:57.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:57.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:57.173 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:57.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:57.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:57.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:56 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:06:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:56 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:06:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:56 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2869102832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:56 smithi086 ceph-mon[32965]: osdmap e404: 8 total, 8 up, 5 in 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2869102832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[29446]: osdmap e404: 8 total, 8 up, 5 in 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2869102832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:06:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:56 smithi006 ceph-mon[37501]: osdmap e404: 8 total, 8 up, 5 in 2024-03-26T04:06:57.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:57.438 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:57.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:57.581 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:57.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:57.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:57.715 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:57.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:57.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.849 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:57.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:57.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:57.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:58.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:06:58.125 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:06:58.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:58.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:58.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:57 smithi086 ceph-mon[32965]: pgmap v859: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 10419/169173 objects degraded (6.159%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:06:58.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:06:58.372 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:06:58.373 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:06:58.374 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:06:58.374 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:06:58.374 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:06:58.374 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:06:57 smithi006 ceph-mon[37501]: pgmap v859: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 10419/169173 objects degraded (6.159%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:06:58.375 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:57 smithi006 ceph-mon[29446]: pgmap v859: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.4 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 10419/169173 objects degraded (6.159%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:06:58.389 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:58.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:06:58.524 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:06:58.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:06:58.656 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:58.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:06:58.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:58.799 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:58.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:58.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:58.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:58.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:59.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:59.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:59.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:06:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:59.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:06:59.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:59.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.328 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:59.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:59.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:06:59.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:06:59.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:06:59.597 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:59.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:06:59.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:06:59.739 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:06:59.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:06:59.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:06:59.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:06:59.874 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:06:59.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:06:59.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.017 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:00.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:00.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:00.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:00.256 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:00.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:00.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.291 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:00.310 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:59 smithi086 ceph-mon[32965]: pgmap v860: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.47k op/s; 10419/165132 objects degraded (6.309%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:07:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:06:59 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 10419/165132 objects degraded (6.309%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:00 smithi006 ceph-mon[37501]: pgmap v860: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.47k op/s; 10419/165132 objects degraded (6.309%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:07:00.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:00 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 10419/165132 objects degraded (6.309%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:00.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:59 smithi006 ceph-mon[29446]: pgmap v860: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.3 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.47k op/s; 10419/165132 objects degraded (6.309%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:07:00.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:06:59 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 10419/165132 objects degraded (6.309%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:00.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:00.424 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:00.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:00.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:00.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:00.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:00.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:00.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:00.830 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:00.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:00.932 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:00.977 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:00.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:01.079 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:01.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:01.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.122 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:01.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:01.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:01.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:01.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:01.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:01.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:01.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.532 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:01.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:01.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:01.679 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:01.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:01.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:01.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:01.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:01.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:01.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:02.017 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:07:02.046 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:07:02.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:02.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:02.229 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:02.263 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:02 smithi086 ceph-mon[32965]: pgmap v861: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.40k op/s; 9399/160089 objects degraded (5.871%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:07:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:02 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1183226929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:02 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[37501]: pgmap v861: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.40k op/s; 9399/160089 objects degraded (5.871%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:07:02.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1183226929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.338 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.338 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[29446]: pgmap v861: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.2 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.40k op/s; 9399/160089 objects degraded (5.871%); 26 MiB/s, 419 objects/s recovering 2024-03-26T04:07:02.339 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1183226929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.339 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:02 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:02.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:02.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:02.490 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:02.519 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:02.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:02.651 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:02.753 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:02.780 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:02.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:02.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:02.914 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:02.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:03.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:03.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:03.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:03.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:03 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:03.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:03 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:03.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:03 smithi086 ceph-mon[32965]: osdmap e405: 8 total, 8 up, 5 in 2024-03-26T04:07:03.365 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:03.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:03.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:03.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:03.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[29446]: osdmap e405: 8 total, 8 up, 5 in 2024-03-26T04:07:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:03.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:03 smithi006 ceph-mon[37501]: osdmap e405: 8 total, 8 up, 5 in 2024-03-26T04:07:03.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:03.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:03.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:03.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:03.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:03.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:03.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:03.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:03.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:03.866 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:03.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:03.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:03.993 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:03.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:03.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:04.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:04.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:04.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:04.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:04.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:04.272 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:07:04.272 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:07:04.272 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:07:04.272 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:07:04.272 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:07:04.272 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:04 smithi006 ceph-mon[37501]: pgmap v863: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 709 B/s rd, 0 B/s wr, 1.52k op/s; 7503/148053 objects degraded (5.068%); 23 MiB/s, 368 objects/s recovering 2024-03-26T04:07:04.272 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:04 smithi006 ceph-mon[37501]: osdmap e406: 8 total, 8 up, 5 in 2024-03-26T04:07:04.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:04 smithi006 ceph-mon[29446]: pgmap v863: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 709 B/s rd, 0 B/s wr, 1.52k op/s; 7503/148053 objects degraded (5.068%); 23 MiB/s, 368 objects/s recovering 2024-03-26T04:07:04.273 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:04 smithi006 ceph-mon[29446]: osdmap e406: 8 total, 8 up, 5 in 2024-03-26T04:07:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:04 smithi086 ceph-mon[32965]: pgmap v863: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 153 active+clean; 3.0 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 709 B/s rd, 0 B/s wr, 1.52k op/s; 7503/148053 objects degraded (5.068%); 23 MiB/s, 368 objects/s recovering 2024-03-26T04:07:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:04 smithi086 ceph-mon[32965]: osdmap e406: 8 total, 8 up, 5 in 2024-03-26T04:07:04.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:04.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:04.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:04.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:04.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.512 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:04.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:04.592 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:04.613 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:04.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.655 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:04.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:04.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:04.789 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:04.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:04.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:04.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:04.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:04.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:05.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:05.054 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:05.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[29446]: osdmap e407: 8 total, 8 up, 5 in 2024-03-26T04:07:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 7503/148053 objects degraded (5.068%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[29446]: osdmap e408: 8 total, 8 up, 5 in 2024-03-26T04:07:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[37501]: osdmap e407: 8 total, 8 up, 5 in 2024-03-26T04:07:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 7503/148053 objects degraded (5.068%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:05.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:05 smithi006 ceph-mon[37501]: osdmap e408: 8 total, 8 up, 5 in 2024-03-26T04:07:05.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:05.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:05.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:05.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:05 smithi086 ceph-mon[32965]: osdmap e407: 8 total, 8 up, 5 in 2024-03-26T04:07:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:05 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 7503/148053 objects degraded (5.068%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:05 smithi086 ceph-mon[32965]: osdmap e408: 8 total, 8 up, 5 in 2024-03-26T04:07:05.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:05.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:05.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:05.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:05.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:05.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:05.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:05.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:05.839 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:05.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:05.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:05.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:05.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:06.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:06.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:06.074 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T04:07:06.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:06.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:06.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:06.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:06.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:06 smithi086 ceph-mon[32965]: pgmap v867: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 154 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 6861/141216 objects degraded (4.859%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:07:06.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:06.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:06 smithi006 ceph-mon[37501]: pgmap v867: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 154 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 6861/141216 objects degraded (4.859%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:07:06.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:06 smithi006 ceph-mon[29446]: pgmap v867: 158 pgs: 1 active+undersized+degraded+remapped+backfill_wait, 1 active+undersized+degraded+remapped+backfill_wait+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 154 active+clean; 2.9 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.57k op/s; 6861/141216 objects degraded (4.859%); 23 MiB/s, 365 objects/s recovering 2024-03-26T04:07:06.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:06.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:06.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:06.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:06.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:06.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:06.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:06.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:06.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:06.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:07.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:07.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:07.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:07.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:07.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:07.334 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:07.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:07 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:07.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:07 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:07.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:07 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:07.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:07.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.478 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:07.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:07.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:07.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:07.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:07:07.713 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.745 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:07.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:07.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:07.876 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:07.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:07.978 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:08.019 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:08.120 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:08.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:08.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:08.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:08.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:08 smithi086 ceph-mon[32965]: pgmap v868: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 4973/134415 objects degraded (3.700%); 26 MiB/s, 416 objects/s recovering 2024-03-26T04:07:08.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:08 smithi006 ceph-mon[37501]: pgmap v868: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 4973/134415 objects degraded (3.700%); 26 MiB/s, 416 objects/s recovering 2024-03-26T04:07:08.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:08 smithi006 ceph-mon[29446]: pgmap v868: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 155 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 955 op/s; 4973/134415 objects degraded (3.700%); 26 MiB/s, 416 objects/s recovering 2024-03-26T04:07:08.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:08.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:08.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:08.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:08.555 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:08.657 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:08.692 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:08.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:08.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:08.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:08.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:08.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:08.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:09.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:09.084 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:09.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:09.217 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:09.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:09.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:07:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:07:09.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:09.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:07:09.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:07:09.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:07:09.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:07:09.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:09.490 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:09.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:09.642 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:09.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:09.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:09.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:09.875 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:09.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:09.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:09.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:09.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:10.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:10.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:10.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:10.056 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: pgmap v869: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 4181/129786 objects degraded (3.221%); 29 MiB/s, 457 objects/s recovering 2024-03-26T04:07:10.056 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: osdmap e409: 8 total, 8 up, 5 in 2024-03-26T04:07:10.056 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:07:10.077 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:10.137 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:10.140 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.174 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:10.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:10.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:10.304 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: pgmap v869: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 4181/129786 objects degraded (3.221%); 29 MiB/s, 457 objects/s recovering 2024-03-26T04:07:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: osdmap e409: 8 total, 8 up, 5 in 2024-03-26T04:07:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:07:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:07:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 4181/129786 objects degraded (3.221%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: osdmap e410: 8 total, 8 up, 5 in 2024-03-26T04:07:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 4181/129786 objects degraded (3.221%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: osdmap e410: 8 total, 8 up, 5 in 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: pgmap v869: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 4181/129786 objects degraded (3.221%); 29 MiB/s, 457 objects/s recovering 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: osdmap e409: 8 total, 8 up, 5 in 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:07:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:07:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 4181/129786 objects degraded (3.221%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: osdmap e410: 8 total, 8 up, 5 in 2024-03-26T04:07:10.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:10.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:10.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:10.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.564 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:10.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:10.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:10.694 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:10.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:10.936 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:10.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:10.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:11.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:11.110 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:11.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:11.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:11.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:11.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:11.345 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:11.382 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:11.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:11.522 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:11.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:11.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:11.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:11.805 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:11.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:11.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:11.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:12.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:12.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:12.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.199 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:12.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:12.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:12.308 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[37501]: pgmap v872: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.08k op/s; 4181/124326 objects degraded (3.363%); 27 MiB/s, 439 objects/s recovering 2024-03-26T04:07:12.309 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:07:12.309 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:07:12.309 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[29446]: pgmap v872: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.08k op/s; 4181/124326 objects degraded (3.363%); 27 MiB/s, 439 objects/s recovering 2024-03-26T04:07:12.309 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:07:12.309 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:12 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:07:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:12 smithi086 ceph-mon[32965]: pgmap v872: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 156 active+clean; 2.5 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.08k op/s; 4181/124326 objects degraded (3.363%); 27 MiB/s, 439 objects/s recovering 2024-03-26T04:07:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:12 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:07:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:12 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:07:12.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:12.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:12.486 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:12.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:12.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:12.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:12.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:12.760 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:12.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:12.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:12.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:12.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:13.033 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:13.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:13.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:13.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[37501]: osdmap e411: 8 total, 8 up, 5 in 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[37501]: pgmap v874: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 3055/113715 objects degraded (2.687%); 13 MiB/s, 213 objects/s recovering 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[29446]: osdmap e411: 8 total, 8 up, 5 in 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:07:13.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:13 smithi006 ceph-mon[29446]: pgmap v874: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 3055/113715 objects degraded (2.687%); 13 MiB/s, 213 objects/s recovering 2024-03-26T04:07:13.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:13 smithi086 ceph-mon[32965]: osdmap e411: 8 total, 8 up, 5 in 2024-03-26T04:07:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:13 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:07:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:13 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:07:13.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:13 smithi086 ceph-mon[32965]: pgmap v874: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.3 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 3055/113715 objects degraded (2.687%); 13 MiB/s, 213 objects/s recovering 2024-03-26T04:07:13.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:13.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:13.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:13.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:13.579 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:13.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:13.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:13.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:13.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:13.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:13.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:13.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:13.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:13.995 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:13.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:14.096 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:14.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.134 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:14.170 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:14 smithi006 ceph-mon[37501]: osdmap e412: 8 total, 8 up, 5 in 2024-03-26T04:07:14.171 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:14 smithi006 ceph-mon[29446]: osdmap e412: 8 total, 8 up, 5 in 2024-03-26T04:07:14.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:14.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:14 smithi086 ceph-mon[32965]: osdmap e412: 8 total, 8 up, 5 in 2024-03-26T04:07:14.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:14.431 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:14.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:14.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:14.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:14.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:14.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:14.850 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:14.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:14.989 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:14.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:15.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:15.121 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:15.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:15.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:15.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:15.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:15 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 3055/113715 objects degraded (2.687%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:15.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:15 smithi006 ceph-mon[37501]: pgmap v876: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s; 3055/106890 objects degraded (2.858%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T04:07:15.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:15 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 3055/113715 objects degraded (2.687%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:15.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:15 smithi006 ceph-mon[29446]: pgmap v876: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s; 3055/106890 objects degraded (2.858%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T04:07:15.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:15.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:15 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 3055/113715 objects degraded (2.687%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:15.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:15 smithi086 ceph-mon[32965]: pgmap v876: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s; 3055/106890 objects degraded (2.858%); 5.7 MiB/s, 91 objects/s recovering 2024-03-26T04:07:15.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:15.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:15.650 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:15.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:15.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:15.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:15.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:15.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:16.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:16.049 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:16.134 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T04:07:16.151 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:16.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.189 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:16.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:16.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:16.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:16.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3527408864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3527408864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:16.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:16.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:16.556 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:16 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3527408864' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:16 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:07:16.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.597 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:16.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:16.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:16.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:16.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:16.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:16.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:16.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:16.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:16.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:17.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:17.130 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:17.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:17.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:17.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:17.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:17.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:07:17.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[29446]: osdmap e413: 8 total, 8 up, 5 in 2024-03-26T04:07:17.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[29446]: pgmap v878: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.85k op/s; 1977/91011 objects degraded (2.172%); 11 MiB/s, 170 objects/s recovering 2024-03-26T04:07:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:07:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[37501]: osdmap e413: 8 total, 8 up, 5 in 2024-03-26T04:07:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:17 smithi006 ceph-mon[37501]: pgmap v878: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.85k op/s; 1977/91011 objects degraded (2.172%); 11 MiB/s, 170 objects/s recovering 2024-03-26T04:07:17.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:17.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:17.524 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:17 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:07:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:17 smithi086 ceph-mon[32965]: osdmap e413: 8 total, 8 up, 5 in 2024-03-26T04:07:17.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:17 smithi086 ceph-mon[32965]: pgmap v878: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.8 GiB data, 9.2 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.85k op/s; 1977/91011 objects degraded (2.172%); 11 MiB/s, 170 objects/s recovering 2024-03-26T04:07:17.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:17.663 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:17.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:17.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:17.803 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:17.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:17.905 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:17.941 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:17.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:17.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:18.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:18.092 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:18.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:18.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:18.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:18.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:18.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:18.351 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:18.447 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:07:18.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:18.466 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:07:18.476 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:18.487 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:07:18.507 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:07:18.527 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:07:18.547 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:07:18.567 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:07:18.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:18.587 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:07:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:07:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:07:18.588 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:07:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:07:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:07:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:07:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:07:18.589 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:07:18.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:18.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:18.725 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:18.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:18.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:18.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:18.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:18.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:19.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:19.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:19.132 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:19.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:19.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:19.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:19.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[37501]: osdmap e414: 8 total, 8 up, 5 in 2024-03-26T04:07:19.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/737673481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:19.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[37501]: pgmap v880: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.57k op/s; 1977/85389 objects degraded (2.315%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T04:07:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[29446]: osdmap e414: 8 total, 8 up, 5 in 2024-03-26T04:07:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/737673481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:19 smithi006 ceph-mon[29446]: pgmap v880: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.57k op/s; 1977/85389 objects degraded (2.315%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T04:07:19.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:19.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:19.537 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:19 smithi086 ceph-mon[32965]: osdmap e414: 8 total, 8 up, 5 in 2024-03-26T04:07:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/737673481' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:19.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:19 smithi086 ceph-mon[32965]: pgmap v880: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 9.0 GiB used, 438 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.57k op/s; 1977/85389 objects degraded (2.315%); 5.2 MiB/s, 83 objects/s recovering 2024-03-26T04:07:19.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:19.668 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:19.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:19.800 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:19.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:19.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:19.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:20.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:20.081 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:20.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:20.201 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T04:07:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.229 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:20.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:20.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:20.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:20.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:20.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[37501]: osdmap e415: 8 total, 8 up, 5 in 2024-03-26T04:07:20.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1977/85389 objects degraded (2.315%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:20.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[37501]: osdmap e416: 8 total, 8 up, 5 in 2024-03-26T04:07:20.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[29446]: osdmap e415: 8 total, 8 up, 5 in 2024-03-26T04:07:20.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1977/85389 objects degraded (2.315%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:20.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:20 smithi006 ceph-mon[29446]: osdmap e416: 8 total, 8 up, 5 in 2024-03-26T04:07:20.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:20.524 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:20.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:20 smithi086 ceph-mon[32965]: osdmap e415: 8 total, 8 up, 5 in 2024-03-26T04:07:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:20 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1977/85389 objects degraded (2.315%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-26T04:07:20.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:20 smithi086 ceph-mon[32965]: osdmap e416: 8 total, 8 up, 5 in 2024-03-26T04:07:20.626 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:20.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:20.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:20.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:20.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:20.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:20.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:20.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:20.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:20.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:21.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:21.080 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:21.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:21.163 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:07:21.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:21.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:21.278 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:07:21.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:21.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:21.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:21 smithi006 ceph-mon[37501]: pgmap v883: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 1977/81003 objects degraded (2.441%); 6.7 MiB/s, 107 objects/s recovering 2024-03-26T04:07:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:07:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:21 smithi006 ceph-mon[29446]: pgmap v883: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 1977/81003 objects degraded (2.441%); 6.7 MiB/s, 107 objects/s recovering 2024-03-26T04:07:21.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:07:21.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:21.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:21.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:21.576 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:21 smithi086 ceph-mon[32965]: pgmap v883: 158 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.5 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.24k op/s; 1977/81003 objects degraded (2.441%); 6.7 MiB/s, 107 objects/s recovering 2024-03-26T04:07:21.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:07:21.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:21.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:21.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:21.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:21.864 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:21.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:21.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:21.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:22.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:22.033 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:22.135 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:22.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:22.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:22.282 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:07:22.282 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:07:22.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:22.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:22.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:07:22.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[37501]: osdmap e417: 8 total, 8 up, 5 in 2024-03-26T04:07:22.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:22.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:07:22.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:22 smithi006 ceph-mon[29446]: osdmap e417: 8 total, 8 up, 5 in 2024-03-26T04:07:22.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:22.445 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:22.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:22.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:22 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/948033406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:07:22.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:22 smithi086 ceph-mon[32965]: osdmap e417: 8 total, 8 up, 5 in 2024-03-26T04:07:22.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:22.725 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:22.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:22.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:22.850 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:22.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:22.983 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:22.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:23.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:23.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:23.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:23.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:23.358 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:23.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:23.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:23.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:23.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:23 smithi086 ceph-mon[32965]: 1.a scrub starts 2024-03-26T04:07:23.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:23 smithi086 ceph-mon[32965]: 1.a scrub ok 2024-03-26T04:07:23.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:23 smithi086 ceph-mon[32965]: pgmap v885: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.4 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 1385/71550 objects degraded (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-03-26T04:07:23.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:23.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[37501]: 1.a scrub starts 2024-03-26T04:07:23.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[37501]: 1.a scrub ok 2024-03-26T04:07:23.640 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[37501]: pgmap v885: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.4 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 1385/71550 objects degraded (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-03-26T04:07:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[29446]: 1.a scrub starts 2024-03-26T04:07:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[29446]: 1.a scrub ok 2024-03-26T04:07:23.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:23 smithi006 ceph-mon[29446]: pgmap v885: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.4 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 994 op/s; 1385/71550 objects degraded (1.936%); 1.9 MiB/s, 30 objects/s recovering 2024-03-26T04:07:23.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:23.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:23.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:23.896 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:23.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:23.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:24.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:24.064 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:24.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:24.206 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:24.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:24.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:24.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:24.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.487 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:24.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:24.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:24.589 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:24.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:24.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:24.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:24.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:24.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:24.881 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:24.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:24.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:24.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:24 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 1385/71550 objects degraded (1.936%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:24.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:24 smithi006 ceph-mon[37501]: osdmap e418: 8 total, 8 up, 5 in 2024-03-26T04:07:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:24 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 1385/71550 objects degraded (1.936%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:24.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:24 smithi006 ceph-mon[29446]: osdmap e418: 8 total, 8 up, 5 in 2024-03-26T04:07:24.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:25.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:25.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:25.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:24 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 1385/71550 objects degraded (1.936%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:25.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:24 smithi086 ceph-mon[32965]: osdmap e418: 8 total, 8 up, 5 in 2024-03-26T04:07:25.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:25.159 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:25.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:25.292 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:25.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:25.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:25.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:25.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:25.553 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:25.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:25.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:25.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:25.684 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:25.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:25.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:25.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:25.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: pgmap v887: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1385/65058 objects degraded (2.129%); 1.5 MiB/s, 25 objects/s recovering 2024-03-26T04:07:25.865 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: pgmap v887: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1385/65058 objects degraded (2.129%); 1.5 MiB/s, 25 objects/s recovering 2024-03-26T04:07:25.866 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:25.878 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:25.881 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:25.918 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:25.972 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:25.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:25.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:26.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: pgmap v887: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1.3 GiB data, 7.7 GiB used, 439 GiB / 447 GiB avail; 0 B/s wr, 1.20k op/s; 1385/65058 objects degraded (2.129%); 1.5 MiB/s, 25 objects/s recovering 2024-03-26T04:07:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:26.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:26.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:26.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:26.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:26.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:26.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:26.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:26.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:26.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:26.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.526 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:26.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:26.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:26.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:26.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:26.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:07:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:26 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:07:26.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:26.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:26.931 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:26.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:27.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.077 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:27.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:26 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T04:07:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:26 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T04:07:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:26 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:07:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:26 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:07:27.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:27.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:27.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:27.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:27.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:07:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:27.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:27.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:27.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.513 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:27.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:27.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:27.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:27.754 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:27.769 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T04:07:27.769 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T04:07:27.769 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[37501]: pgmap v888: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1003 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 883/48789 objects degraded (1.810%); 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:27.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T04:07:27.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T04:07:27.770 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:27 smithi006 ceph-mon[29446]: pgmap v888: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1003 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 883/48789 objects degraded (1.810%); 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:27.804 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:27.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:27.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:27.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:27.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:27.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:28.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:28.081 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:28.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:27 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T04:07:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:27 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T04:07:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:27 smithi086 ceph-mon[32965]: pgmap v888: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 1003 MiB data, 6.8 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.68k op/s; 883/48789 objects degraded (1.810%); 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:28.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:28.213 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:28.315 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:28.359 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:28.461 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:28.494 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.517 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:28.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:28.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:28.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:28.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:28.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:28.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T04:07:28.869 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:28 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T04:07:28.882 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:28.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:28.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:28 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:07:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:28 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:07:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:28 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T04:07:28.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:28 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T04:07:29.021 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:29.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:29.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:29.190 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:29.225 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:07:29.268 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:07:29.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:29.314 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:07:29.332 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:29.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:29.358 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:07:29.405 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:07:29.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:29.447 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:07:29.482 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:29.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:29.489 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:07:29.539 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:07:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:07:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:07:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:07:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:07:29.552 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:07:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:07:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:07:29.553 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:07:29.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:29.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:29.716 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:29.755 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:29.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:29.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:29.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:29.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[29446]: 4.1d scrub starts 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[29446]: 4.1d scrub ok 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[29446]: pgmap v889: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 880 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 883/42912 objects degraded (2.058%); 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1438351718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 883/48789 objects degraded (1.810%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[37501]: 4.1d scrub starts 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[37501]: 4.1d scrub ok 2024-03-26T04:07:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[37501]: pgmap v889: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 880 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 883/42912 objects degraded (2.058%); 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:29.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1438351718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:29.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:29 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 883/48789 objects degraded (1.810%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:29.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:30.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:29 smithi086 ceph-mon[32965]: 4.1d scrub starts 2024-03-26T04:07:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:29 smithi086 ceph-mon[32965]: 4.1d scrub ok 2024-03-26T04:07:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:29 smithi086 ceph-mon[32965]: pgmap v889: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 880 MiB data, 6.5 GiB used, 440 GiB / 447 GiB avail; 0 B/s wr, 1.58k op/s; 883/42912 objects degraded (2.058%); 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:29 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1438351718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:30.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:29 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 883/48789 objects degraded (1.810%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:30.118 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:30.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:30.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:30.289 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:30.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:30.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:30.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:30.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:30.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:30.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:30.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:30.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:30.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:30.769 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T04:07:30.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:30.839 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:30.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.840 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:30.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:30.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:30.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: 1.7 scrub starts 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: 1.7 scrub ok 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2714655701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: 1.7 scrub starts 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: 1.7 scrub ok 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2714655701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.002 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:30 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: 1.7 scrub starts 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: 1.7 scrub ok 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2714655701' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:30 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:07:31.123 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:31.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:31.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:31.359 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:31.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:31.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:31.529 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:31.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:31.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.678 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:31.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:31.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:31.818 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:31.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: 6.2 scrub starts 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: 6.2 scrub ok 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: osdmap e419: 8 total, 8 up, 5 in 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[37501]: pgmap v891: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 753 MiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.44k op/s; 883/36813 objects degraded (2.399%) 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: 6.2 scrub starts 2024-03-26T04:07:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: 6.2 scrub ok 2024-03-26T04:07:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:07:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: osdmap e419: 8 total, 8 up, 5 in 2024-03-26T04:07:31.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:31 smithi006 ceph-mon[29446]: pgmap v891: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 753 MiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.44k op/s; 883/36813 objects degraded (2.399%) 2024-03-26T04:07:31.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:31.959 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:31.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:32.028 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T04:07:32.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:07:32.062 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T04:07:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T04:07:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: 6.2 scrub starts 2024-03-26T04:07:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: 6.2 scrub ok 2024-03-26T04:07:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:07:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: osdmap e419: 8 total, 8 up, 5 in 2024-03-26T04:07:32.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:31 smithi086 ceph-mon[32965]: pgmap v891: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 753 MiB data, 5.9 GiB used, 441 GiB / 447 GiB avail; 0 B/s wr, 1.44k op/s; 883/36813 objects degraded (2.399%) 2024-03-26T04:07:32.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.122 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:32.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:32.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:32.253 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:32.355 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:32.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:32.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:32.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:32.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:32.539 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:32.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:32.642 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:07:32.643 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:07:32.643 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T04:07:32.643 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:07:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:32.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:32.791 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:32.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:32.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:32.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:32 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T04:07:32.845 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:32 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T04:07:32.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:32 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T04:07:32.845 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:32 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T04:07:32.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:32.976 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:32.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:33.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:32 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T04:07:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:32 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T04:07:33.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:33.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:33.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:33.247 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:33.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:33.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:33.389 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:33.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:33.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:33.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:07:33.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:33.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:33.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:33.809 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:33.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[29446]: pgmap v892: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 503 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 363/24825 objects degraded (1.462%) 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T04:07:33.813 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:33 smithi006 ceph-mon[37501]: pgmap v892: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 503 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 363/24825 objects degraded (1.462%) 2024-03-26T04:07:33.861 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T04:07:33.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:34.003 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:33 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T04:07:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:33 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T04:07:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:33 smithi086 ceph-mon[32965]: pgmap v892: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 503 MiB data, 5.3 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 363/24825 objects degraded (1.462%) 2024-03-26T04:07:34.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:34.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:34.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:34.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:34.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:34.429 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:34.444 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:34.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:34.537 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:34.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:34.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:34.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:34.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:34.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:34.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: 3.13 scrub starts 2024-03-26T04:07:34.861 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: 3.13 scrub ok 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 363/24825 objects degraded (1.462%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[37501]: osdmap e420: 8 total, 8 up, 5 in 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: 3.13 scrub starts 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: 3.13 scrub ok 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 363/24825 objects degraded (1.462%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:34.862 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:34 smithi006 ceph-mon[29446]: osdmap e420: 8 total, 8 up, 5 in 2024-03-26T04:07:34.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:34.951 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:34.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:34.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:34.964 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:35.053 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:35.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: 3.13 scrub starts 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: 3.13 scrub ok 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 363/24825 objects degraded (1.462%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:34 smithi086 ceph-mon[32965]: osdmap e420: 8 total, 8 up, 5 in 2024-03-26T04:07:35.180 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.204 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:35.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:35.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:35.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:35.455 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:35.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:35.585 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:35.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:35.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:35.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:35.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:35.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:35.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:35.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T04:07:35.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[29446]: pgmap v894: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 353 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 363/17622 objects degraded (2.060%) 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[29446]: osdmap e421: 8 total, 8 up, 5 in 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[37501]: pgmap v894: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 353 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 363/17622 objects degraded (2.060%) 2024-03-26T04:07:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:35 smithi006 ceph-mon[37501]: osdmap e421: 8 total, 8 up, 5 in 2024-03-26T04:07:35.959 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:35.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:35.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:35.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:35 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T04:07:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:35 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T04:07:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:35 smithi086 ceph-mon[32965]: pgmap v894: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 353 MiB data, 4.9 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 363/17622 objects degraded (2.060%) 2024-03-26T04:07:36.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:35 smithi086 ceph-mon[32965]: osdmap e421: 8 total, 8 up, 5 in 2024-03-26T04:07:36.094 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Removed 172092 objects 2024-03-26T04:07:36.094 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Clean up completed and total clean up time :127.702 2024-03-26T04:07:36.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:36.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:36.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.249 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:36.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:36.350 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:36.373 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:36.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:36.474 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:36.498 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:36.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:36.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:36.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:36.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:36.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:36.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:36.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:36.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:36.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:36.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:36.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:37.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:37.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:37.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:37.137 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:37.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:37.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:37.267 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:37.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:37.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:37.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:37.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:37.525 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:37.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:37.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:37.652 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:37.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:37.701 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:07:37.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:37.779 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:37.832 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-26T04:07:37.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[37501]: pgmap v896: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 91 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 833 B/s rd, 0 B/s wr, 1.73k op/s; 363/5019 objects degraded (7.233%) 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1973333115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:37.858 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T04:07:37.859 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T04:07:37.859 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[29446]: pgmap v896: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 91 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 833 B/s rd, 0 B/s wr, 1.73k op/s; 363/5019 objects degraded (7.233%) 2024-03-26T04:07:37.859 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1973333115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:37.859 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:37 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:37.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:37.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:37.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:37.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:38.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:38.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:38.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:37 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T04:07:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:37 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T04:07:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:37 smithi086 ceph-mon[32965]: pgmap v896: 158 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 157 active+clean; 91 MiB data, 4.0 GiB used, 443 GiB / 447 GiB avail; 833 B/s rd, 0 B/s wr, 1.73k op/s; 363/5019 objects degraded (7.233%) 2024-03-26T04:07:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1973333115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:37 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:07:38.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:38.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:38.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:38.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:38.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:38.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:38.407 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:38.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:38.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:38.534 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:38.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.661 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:38.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:38.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:38.787 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:38.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:38.913 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:38.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:39.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:39.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T04:07:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T04:07:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:07:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: osdmap e422: 8 total, 8 up, 5 in 2024-03-26T04:07:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T04:07:39.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:38 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T04:07:39.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:07:39.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: osdmap e422: 8 total, 8 up, 5 in 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:07:39.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: osdmap e422: 8 total, 8 up, 5 in 2024-03-26T04:07:39.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T04:07:39.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:38 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T04:07:39.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:39.266 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:39.294 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:39.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:39.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:39.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.421 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:39.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:39.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:39.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:39.773 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:39.798 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:39.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: 6.5 deep-scrub starts 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: 6.5 deep-scrub ok 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: pgmap v898: 158 pgs: 158 active+clean; 16 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2.1 MiB/s, 32 objects/s recovering 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[37501]: Health check update: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: 6.5 deep-scrub starts 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: 6.5 deep-scrub ok 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: pgmap v898: 158 pgs: 158 active+clean; 16 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2.1 MiB/s, 32 objects/s recovering 2024-03-26T04:07:39.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T04:07:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T04:07:39.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:39 smithi006 ceph-mon[29446]: Health check update: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:39.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:39.925 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:39.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:40.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:40.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: 6.5 deep-scrub starts 2024-03-26T04:07:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: 6.5 deep-scrub ok 2024-03-26T04:07:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: pgmap v898: 158 pgs: 158 active+clean; 16 MiB data, 3.9 GiB used, 443 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.30k op/s; 2.1 MiB/s, 32 objects/s recovering 2024-03-26T04:07:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T04:07:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T04:07:40.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:39 smithi086 ceph-mon[32965]: Health check update: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized (PG_DEGRADED) 2024-03-26T04:07:40.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:40.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:40.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:40.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:40.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:40.437 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:40.539 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:40.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:40.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:40.696 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:40.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:40.824 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:40.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:40.953 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:40.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:41.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.080 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:41.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: 1.5 scrub starts 2024-03-26T04:07:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: 1.5 scrub ok 2024-03-26T04:07:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:07:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized) 2024-03-26T04:07:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:41.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: 1.5 scrub starts 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: 1.5 scrub ok 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized) 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:41.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: 1.5 scrub starts 2024-03-26T04:07:41.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: 1.5 scrub ok 2024-03-26T04:07:41.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:07:41.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 363/5019 objects degraded (7.233%), 1 pg degraded, 1 pg undersized) 2024-03-26T04:07:41.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:41.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:41.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:41.210 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:41.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:41.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:41.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.341 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:41.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:41.442 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.470 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:41.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:41.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:41.595 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:41.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:41.708 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T04:07:41.708 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T04:07:41.708 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[29446]: pgmap v899: 158 pgs: 158 active+clean; 966 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 980 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:07:41.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T04:07:41.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T04:07:41.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[37501]: pgmap v899: 158 pgs: 158 active+clean; 966 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 980 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:07:41.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T04:07:41.709 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T04:07:41.710 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T04:07:41.710 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:41 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T04:07:41.724 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:41.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:41.791 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:07:41.791 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:07:41.792 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:07:41.792 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:07:41.792 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:07:41.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:41.852 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:41.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:41.980 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:41.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:42.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:41 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T04:07:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:41 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T04:07:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:41 smithi086 ceph-mon[32965]: pgmap v899: 158 pgs: 158 active+clean; 966 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 980 op/s; 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:07:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:41 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T04:07:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:41 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T04:07:42.106 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:42.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:42.231 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:42.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:42.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:42.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:42.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:42.486 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:42.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:42.588 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:42.610 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:42.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:42.639 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:07:42.642 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:07:42.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:42.740 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:42.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:42.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:42.868 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:42.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:42 smithi006 ceph-mon[29446]: 4.14 deep-scrub starts 2024-03-26T04:07:42.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:42 smithi006 ceph-mon[29446]: 4.14 deep-scrub ok 2024-03-26T04:07:42.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:42 smithi006 ceph-mon[37501]: 4.14 deep-scrub starts 2024-03-26T04:07:42.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:42 smithi006 ceph-mon[37501]: 4.14 deep-scrub ok 2024-03-26T04:07:42.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:42.996 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:42.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:42.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:42 smithi086 ceph-mon[32965]: 4.14 deep-scrub starts 2024-03-26T04:07:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:42 smithi086 ceph-mon[32965]: 4.14 deep-scrub ok 2024-03-26T04:07:43.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:43.124 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:43.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:43.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:43.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:43.378 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:43.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:43.506 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:43.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:43.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:43.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:43.711 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":899,"stamp":"2024-03-26T04:07:40.781960+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":989608,"num_objects":236,"num_object_clones":0,"num_object_copies":708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":236,"num_whiteouts":0,"num_read":2551,"num_read_kb":2271,"num_write":345094,"num_write_kb":11013934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38173,"num_bytes_recovered":2471309329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54450,"ondisk_log_size":54450,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3985056,"kb_used_data":72420,"kb_used_omap":12,"kb_used_meta":3912563,"kb_avail":464740704,"statfs":{"total":479975178240,"available":475894480896,"internally_reserved":0,"allocated":74158080,"data_stored":69171011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13250,"internal_metadata":4006464574},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-399966232,"num_objects":-6104,"num_object_clones":0,"num_object_copies":-18312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-363,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6104,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6104,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.232708"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":773,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701088+0000","last_change":"2024-03-26T04:05:03.632715+0000","last_active":"2024-03-26T04:07:37.701088+0000","last_peered":"2024-03-26T04:07:37.701088+0000","last_clean":"2024-03-26T04:07:37.701088+0000","last_became_active":"2024-03-26T04:01:35.797821+0000","last_became_peered":"2024-03-26T04:01:35.797821+0000","last_unstale":"2024-03-26T04:07:37.701088+0000","last_undegraded":"2024-03-26T04:07:37.701088+0000","last_fullsized":"2024-03-26T04:07:37.701088+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:03.632384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:03.632384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":774,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584490+0000","last_change":"2024-03-26T04:05:29.541099+0000","last_active":"2024-03-26T04:07:35.584490+0000","last_peered":"2024-03-26T04:07:35.584490+0000","last_clean":"2024-03-26T04:07:35.584490+0000","last_became_active":"2024-03-26T04:05:29.540706+0000","last_became_peered":"2024-03-26T04:05:29.540706+0000","last_unstale":"2024-03-26T04:07:35.584490+0000","last_undegraded":"2024-03-26T04:07:35.584490+0000","last_fullsized":"2024-03-26T04:07:35.584490+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:33.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.560099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":794,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563199+0000","last_change":"2024-03-26T04:05:29.539024+0000","last_active":"2024-03-26T04:07:35.563199+0000","last_peered":"2024-03-26T04:07:35.563199+0000","last_clean":"2024-03-26T04:07:35.563199+0000","last_became_active":"2024-03-26T04:05:29.538364+0000","last_became_peered":"2024-03-26T04:05:29.538364+0000","last_unstale":"2024-03-26T04:07:35.563199+0000","last_undegraded":"2024-03-26T04:07:35.563199+0000","last_fullsized":"2024-03-26T04:07:35.563199+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:52.117692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:52.117692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":876,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700839+0000","last_change":"2024-03-26T04:05:29.520520+0000","last_active":"2024-03-26T04:07:37.700839+0000","last_peered":"2024-03-26T04:07:37.700839+0000","last_clean":"2024-03-26T04:07:37.700839+0000","last_became_active":"2024-03-26T04:05:29.520084+0000","last_became_peered":"2024-03-26T04:05:29.520084+0000","last_unstale":"2024-03-26T04:07:37.700839+0000","last_undegraded":"2024-03-26T04:07:37.700839+0000","last_fullsized":"2024-03-26T04:07:37.700839+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:01:20.107206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.107206+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832594+0000","last_change":"2024-03-26T04:06:29.930571+0000","last_active":"2024-03-26T04:07:37.832594+0000","last_peered":"2024-03-26T04:07:37.832594+0000","last_clean":"2024-03-26T04:07:37.832594+0000","last_became_active":"2024-03-26T04:02:52.504637+0000","last_became_peered":"2024-03-26T04:02:52.504637+0000","last_unstale":"2024-03-26T04:07:37.832594+0000","last_undegraded":"2024-03-26T04:07:37.832594+0000","last_fullsized":"2024-03-26T04:07:37.832594+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:29.930462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:29.930462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:07:49.086851+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":769,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700751+0000","last_change":"2024-03-26T04:05:29.519865+0000","last_active":"2024-03-26T04:07:37.700751+0000","last_peered":"2024-03-26T04:07:37.700751+0000","last_clean":"2024-03-26T04:07:37.700751+0000","last_became_active":"2024-03-26T04:05:29.519486+0000","last_became_peered":"2024-03-26T04:05:29.519486+0000","last_unstale":"2024-03-26T04:07:37.700751+0000","last_undegraded":"2024-03-26T04:07:37.700751+0000","last_fullsized":"2024-03-26T04:07:37.700751+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.069804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.069804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":335,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835357+0000","last_change":"2024-03-26T04:07:30.526662+0000","last_active":"2024-03-26T04:07:37.835357+0000","last_peered":"2024-03-26T04:07:37.835357+0000","last_clean":"2024-03-26T04:07:37.835357+0000","last_became_active":"2024-03-26T04:05:29.536149+0000","last_became_peered":"2024-03-26T04:05:29.536149+0000","last_unstale":"2024-03-26T04:07:37.835357+0000","last_undegraded":"2024-03-26T04:07:37.835357+0000","last_fullsized":"2024-03-26T04:07:37.835357+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:44.136302+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":237,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832542+0000","last_change":"2024-03-26T04:05:29.533974+0000","last_active":"2024-03-26T04:07:37.832542+0000","last_peered":"2024-03-26T04:07:37.832542+0000","last_clean":"2024-03-26T04:07:37.832542+0000","last_became_active":"2024-03-26T04:05:29.533552+0000","last_became_peered":"2024-03-26T04:05:29.533552+0000","last_unstale":"2024-03-26T04:07:37.832542+0000","last_undegraded":"2024-03-26T04:07:37.832542+0000","last_fullsized":"2024-03-26T04:07:37.832542+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:01:20.776205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.776205+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":756,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561951+0000","last_change":"2024-03-26T04:04:21.537611+0000","last_active":"2024-03-26T04:07:35.561951+0000","last_peered":"2024-03-26T04:07:35.561951+0000","last_clean":"2024-03-26T04:07:35.561951+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:07:35.561951+0000","last_undegraded":"2024-03-26T04:07:35.561951+0000","last_fullsized":"2024-03-26T04:07:35.561951+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:21.537431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:21.537431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":517,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562350+0000","last_change":"2024-03-26T04:03:53.342268+0000","last_active":"2024-03-26T04:07:35.562350+0000","last_peered":"2024-03-26T04:07:35.562350+0000","last_clean":"2024-03-26T04:07:35.562350+0000","last_became_active":"2024-03-26T04:01:35.802827+0000","last_became_peered":"2024-03-26T04:01:35.802827+0000","last_unstale":"2024-03-26T04:07:35.562350+0000","last_undegraded":"2024-03-26T04:07:35.562350+0000","last_fullsized":"2024-03-26T04:07:35.562350+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:53.342172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.342172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":475,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702535+0000","last_change":"2024-03-26T04:04:55.527963+0000","last_active":"2024-03-26T04:07:37.702535+0000","last_peered":"2024-03-26T04:07:37.702535+0000","last_clean":"2024-03-26T04:07:37.702535+0000","last_became_active":"2024-03-26T04:02:52.503960+0000","last_became_peered":"2024-03-26T04:02:52.503960+0000","last_unstale":"2024-03-26T04:07:37.702535+0000","last_undegraded":"2024-03-26T04:07:37.702535+0000","last_fullsized":"2024-03-26T04:07:37.702535+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:55.527836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.527836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":885,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701239+0000","last_change":"2024-03-26T04:07:26.622146+0000","last_active":"2024-03-26T04:07:37.701239+0000","last_peered":"2024-03-26T04:07:37.701239+0000","last_clean":"2024-03-26T04:07:37.701239+0000","last_became_active":"2024-03-26T04:05:29.521715+0000","last_became_peered":"2024-03-26T04:05:29.521715+0000","last_unstale":"2024-03-26T04:07:37.701239+0000","last_undegraded":"2024-03-26T04:07:37.701239+0000","last_fullsized":"2024-03-26T04:07:37.701239+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T04:07:26.622024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.622024+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:36.899154+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":568,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584680+0000","last_change":"2024-03-26T04:05:29.533183+0000","last_active":"2024-03-26T04:07:35.584680+0000","last_peered":"2024-03-26T04:07:35.584680+0000","last_clean":"2024-03-26T04:07:35.584680+0000","last_became_active":"2024-03-26T04:05:29.532847+0000","last_became_peered":"2024-03-26T04:05:29.532847+0000","last_unstale":"2024-03-26T04:07:35.584680+0000","last_undegraded":"2024-03-26T04:07:35.584680+0000","last_fullsized":"2024-03-26T04:07:35.584680+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.396869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.396869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":792,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832771+0000","last_change":"2024-03-26T04:07:24.766554+0000","last_active":"2024-03-26T04:07:37.832771+0000","last_peered":"2024-03-26T04:07:37.832771+0000","last_clean":"2024-03-26T04:07:37.832771+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:07:37.832771+0000","last_undegraded":"2024-03-26T04:07:37.832771+0000","last_fullsized":"2024-03-26T04:07:37.832771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.766403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.766403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:28.353556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":494,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701776+0000","last_change":"2024-03-26T04:04:01.292305+0000","last_active":"2024-03-26T04:07:37.701776+0000","last_peered":"2024-03-26T04:07:37.701776+0000","last_clean":"2024-03-26T04:07:37.701776+0000","last_became_active":"2024-03-26T04:01:35.797214+0000","last_became_peered":"2024-03-26T04:01:35.797214+0000","last_unstale":"2024-03-26T04:07:37.701776+0000","last_undegraded":"2024-03-26T04:07:37.701776+0000","last_fullsized":"2024-03-26T04:07:37.701776+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:01.292182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:01.292182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":837,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701758+0000","last_change":"2024-03-26T04:05:29.520607+0000","last_active":"2024-03-26T04:07:37.701758+0000","last_peered":"2024-03-26T04:07:37.701758+0000","last_clean":"2024-03-26T04:07:37.701758+0000","last_became_active":"2024-03-26T04:05:29.519855+0000","last_became_peered":"2024-03-26T04:05:29.519855+0000","last_unstale":"2024-03-26T04:07:37.701758+0000","last_undegraded":"2024-03-26T04:07:37.701758+0000","last_fullsized":"2024-03-26T04:07:37.701758+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:01:08.181298+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.181298+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":799,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702630+0000","last_change":"2024-03-26T04:05:02.671720+0000","last_active":"2024-03-26T04:07:37.702630+0000","last_peered":"2024-03-26T04:07:37.702630+0000","last_clean":"2024-03-26T04:07:37.702630+0000","last_became_active":"2024-03-26T04:02:52.500023+0000","last_became_peered":"2024-03-26T04:02:52.500023+0000","last_unstale":"2024-03-26T04:07:37.702630+0000","last_undegraded":"2024-03-26T04:07:37.702630+0000","last_fullsized":"2024-03-26T04:07:37.702630+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:02.671615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:02.671615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":766,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585234+0000","last_change":"2024-03-26T04:04:51.186722+0000","last_active":"2024-03-26T04:07:35.585234+0000","last_peered":"2024-03-26T04:07:35.585234+0000","last_clean":"2024-03-26T04:07:35.585234+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:07:35.585234+0000","last_undegraded":"2024-03-26T04:07:35.585234+0000","last_fullsized":"2024-03-26T04:07:35.585234+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.186605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.186605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":776,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562649+0000","last_change":"2024-03-26T04:03:54.321625+0000","last_active":"2024-03-26T04:07:35.562649+0000","last_peered":"2024-03-26T04:07:35.562649+0000","last_clean":"2024-03-26T04:07:35.562649+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:07:35.562649+0000","last_undegraded":"2024-03-26T04:07:35.562649+0000","last_fullsized":"2024-03-26T04:07:35.562649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:54.321548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:54.321548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":800,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833687+0000","last_change":"2024-03-26T04:05:29.533402+0000","last_active":"2024-03-26T04:07:37.833687+0000","last_peered":"2024-03-26T04:07:37.833687+0000","last_clean":"2024-03-26T04:07:37.833687+0000","last_became_active":"2024-03-26T04:05:29.525450+0000","last_became_peered":"2024-03-26T04:05:29.525450+0000","last_unstale":"2024-03-26T04:07:37.833687+0000","last_undegraded":"2024-03-26T04:07:37.833687+0000","last_fullsized":"2024-03-26T04:07:37.833687+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:02:45.014541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:45.014541+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":399,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585578+0000","last_change":"2024-03-26T04:05:29.533637+0000","last_active":"2024-03-26T04:07:35.585578+0000","last_peered":"2024-03-26T04:07:35.585578+0000","last_clean":"2024-03-26T04:07:35.585578+0000","last_became_active":"2024-03-26T04:05:29.533287+0000","last_became_peered":"2024-03-26T04:05:29.533287+0000","last_unstale":"2024-03-26T04:07:35.585578+0000","last_undegraded":"2024-03-26T04:07:35.585578+0000","last_fullsized":"2024-03-26T04:07:35.585578+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:22.809763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.809763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":789,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702022+0000","last_change":"2024-03-26T04:07:37.463272+0000","last_active":"2024-03-26T04:07:37.702022+0000","last_peered":"2024-03-26T04:07:37.702022+0000","last_clean":"2024-03-26T04:07:37.702022+0000","last_became_active":"2024-03-26T04:05:29.536639+0000","last_became_peered":"2024-03-26T04:05:29.536639+0000","last_unstale":"2024-03-26T04:07:37.702022+0000","last_undegraded":"2024-03-26T04:07:37.702022+0000","last_fullsized":"2024-03-26T04:07:37.702022+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.463158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.463158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.787883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":779,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701345+0000","last_change":"2024-03-26T04:07:24.624918+0000","last_active":"2024-03-26T04:07:37.701345+0000","last_peered":"2024-03-26T04:07:37.701345+0000","last_clean":"2024-03-26T04:07:37.701345+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:07:37.701345+0000","last_undegraded":"2024-03-26T04:07:37.701345+0000","last_fullsized":"2024-03-26T04:07:37.701345+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.624793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.624793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.269799+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":322,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585014+0000","last_change":"2024-03-26T04:07:28.446713+0000","last_active":"2024-03-26T04:07:35.585014+0000","last_peered":"2024-03-26T04:07:35.585014+0000","last_clean":"2024-03-26T04:07:35.585014+0000","last_became_active":"2024-03-26T04:05:29.521493+0000","last_became_peered":"2024-03-26T04:05:29.521493+0000","last_unstale":"2024-03-26T04:07:35.585014+0000","last_undegraded":"2024-03-26T04:07:35.585014+0000","last_fullsized":"2024-03-26T04:07:35.585014+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:07:28.446562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:28.446562+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:32.529078+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":238,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833941+0000","last_change":"2024-03-26T04:05:29.548371+0000","last_active":"2024-03-26T04:07:37.833941+0000","last_peered":"2024-03-26T04:07:37.833941+0000","last_clean":"2024-03-26T04:07:37.833941+0000","last_became_active":"2024-03-26T04:05:29.545095+0000","last_became_peered":"2024-03-26T04:05:29.545095+0000","last_unstale":"2024-03-26T04:07:37.833941+0000","last_undegraded":"2024-03-26T04:07:37.833941+0000","last_fullsized":"2024-03-26T04:07:37.833941+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:02:40.908630+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:02:40.908630+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"379'11","reported_seq":943,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702972+0000","last_change":"2024-03-26T04:05:29.535165+0000","last_active":"2024-03-26T04:07:37.702972+0000","last_peered":"2024-03-26T04:07:37.702972+0000","last_clean":"2024-03-26T04:07:37.702972+0000","last_became_active":"2024-03-26T04:05:29.534648+0000","last_became_peered":"2024-03-26T04:05:29.534648+0000","last_unstale":"2024-03-26T04:07:37.702972+0000","last_undegraded":"2024-03-26T04:07:37.702972+0000","last_fullsized":"2024-03-26T04:07:37.702972+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:35.094793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.094793+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":806,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584373+0000","last_change":"2024-03-26T04:04:18.067687+0000","last_active":"2024-03-26T04:07:35.584373+0000","last_peered":"2024-03-26T04:07:35.584373+0000","last_clean":"2024-03-26T04:07:35.584373+0000","last_became_active":"2024-03-26T04:02:52.514953+0000","last_became_peered":"2024-03-26T04:02:52.514953+0000","last_unstale":"2024-03-26T04:07:35.584373+0000","last_undegraded":"2024-03-26T04:07:35.584373+0000","last_fullsized":"2024-03-26T04:07:35.584373+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:04:18.067462+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.067462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"356'32","reported_seq":874,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563242+0000","last_change":"2024-03-26T04:04:05.345019+0000","last_active":"2024-03-26T04:07:35.563242+0000","last_peered":"2024-03-26T04:07:35.563242+0000","last_clean":"2024-03-26T04:07:35.563242+0000","last_became_active":"2024-03-26T04:02:52.497638+0000","last_became_peered":"2024-03-26T04:02:52.497638+0000","last_unstale":"2024-03-26T04:07:35.563242+0000","last_undegraded":"2024-03-26T04:07:35.563242+0000","last_fullsized":"2024-03-26T04:07:35.563242+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'32","last_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:04:05.344849+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":770,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585044+0000","last_change":"2024-03-26T04:05:29.520660+0000","last_active":"2024-03-26T04:07:35.585044+0000","last_peered":"2024-03-26T04:07:35.585044+0000","last_clean":"2024-03-26T04:07:35.585044+0000","last_became_active":"2024-03-26T04:05:29.520352+0000","last_became_peered":"2024-03-26T04:05:29.520352+0000","last_unstale":"2024-03-26T04:07:35.585044+0000","last_undegraded":"2024-03-26T04:07:35.585044+0000","last_fullsized":"2024-03-26T04:07:35.585044+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:21.633920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.633920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":771,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563578+0000","last_change":"2024-03-26T04:07:29.534176+0000","last_active":"2024-03-26T04:07:35.563578+0000","last_peered":"2024-03-26T04:07:35.563578+0000","last_clean":"2024-03-26T04:07:35.563578+0000","last_became_active":"2024-03-26T04:05:29.536597+0000","last_became_peered":"2024-03-26T04:05:29.536597+0000","last_unstale":"2024-03-26T04:07:35.563578+0000","last_undegraded":"2024-03-26T04:07:35.563578+0000","last_fullsized":"2024-03-26T04:07:35.563578+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:29.533765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.533765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:31.454229+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":362,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561809+0000","last_change":"2024-03-26T04:07:31.484578+0000","last_active":"2024-03-26T04:07:35.561809+0000","last_peered":"2024-03-26T04:07:35.561809+0000","last_clean":"2024-03-26T04:07:35.561809+0000","last_became_active":"2024-03-26T04:05:29.540582+0000","last_became_peered":"2024-03-26T04:05:29.540582+0000","last_unstale":"2024-03-26T04:07:35.561809+0000","last_undegraded":"2024-03-26T04:07:35.561809+0000","last_fullsized":"2024-03-26T04:07:35.561809+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:31.484355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.484355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:56.172779+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":599,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835034+0000","last_change":"2024-03-26T04:05:29.537371+0000","last_active":"2024-03-26T04:07:37.835034+0000","last_peered":"2024-03-26T04:07:37.835034+0000","last_clean":"2024-03-26T04:07:37.835034+0000","last_became_active":"2024-03-26T04:05:29.536977+0000","last_became_peered":"2024-03-26T04:05:29.536977+0000","last_unstale":"2024-03-26T04:07:37.835034+0000","last_undegraded":"2024-03-26T04:07:37.835034+0000","last_fullsized":"2024-03-26T04:07:37.835034+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":782,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832937+0000","last_change":"2024-03-26T04:05:44.853422+0000","last_active":"2024-03-26T04:07:37.832937+0000","last_peered":"2024-03-26T04:07:37.832937+0000","last_clean":"2024-03-26T04:07:37.832937+0000","last_became_active":"2024-03-26T04:05:29.533627+0000","last_became_peered":"2024-03-26T04:05:29.533627+0000","last_unstale":"2024-03-26T04:07:37.832937+0000","last_undegraded":"2024-03-26T04:07:37.832937+0000","last_fullsized":"2024-03-26T04:07:37.832937+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:44.853303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:32.737823+0000","last_clean_scrub_stamp":"2024-03-26T04:05:44.853303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":701,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701036+0000","last_change":"2024-03-26T04:04:17.385877+0000","last_active":"2024-03-26T04:07:37.701036+0000","last_peered":"2024-03-26T04:07:37.701036+0000","last_clean":"2024-03-26T04:07:37.701036+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:07:37.701036+0000","last_undegraded":"2024-03-26T04:07:37.701036+0000","last_fullsized":"2024-03-26T04:07:37.701036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:17.385765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.385765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":802,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563415+0000","last_change":"2024-03-26T04:05:29.539237+0000","last_active":"2024-03-26T04:07:35.563415+0000","last_peered":"2024-03-26T04:07:35.563415+0000","last_clean":"2024-03-26T04:07:35.563415+0000","last_became_active":"2024-03-26T04:05:29.538915+0000","last_became_peered":"2024-03-26T04:05:29.538915+0000","last_unstale":"2024-03-26T04:07:35.563415+0000","last_undegraded":"2024-03-26T04:07:35.563415+0000","last_fullsized":"2024-03-26T04:07:35.563415+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:16.271586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:16.271586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":296,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584458+0000","last_change":"2024-03-26T04:05:29.520954+0000","last_active":"2024-03-26T04:07:35.584458+0000","last_peered":"2024-03-26T04:07:35.584458+0000","last_clean":"2024-03-26T04:07:35.584458+0000","last_became_active":"2024-03-26T04:05:29.520584+0000","last_became_peered":"2024-03-26T04:05:29.520584+0000","last_unstale":"2024-03-26T04:07:35.584458+0000","last_undegraded":"2024-03-26T04:07:35.584458+0000","last_fullsized":"2024-03-26T04:07:35.584458+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:00:38.458453+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.458453+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":421,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562150+0000","last_change":"2024-03-26T04:05:29.537044+0000","last_active":"2024-03-26T04:07:35.562150+0000","last_peered":"2024-03-26T04:07:35.562150+0000","last_clean":"2024-03-26T04:07:35.562150+0000","last_became_active":"2024-03-26T04:05:29.536699+0000","last_became_peered":"2024-03-26T04:05:29.536699+0000","last_unstale":"2024-03-26T04:07:35.562150+0000","last_undegraded":"2024-03-26T04:07:35.562150+0000","last_fullsized":"2024-03-26T04:07:35.562150+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.635002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.635002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":791,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832589+0000","last_change":"2024-03-26T04:05:48.811855+0000","last_active":"2024-03-26T04:07:37.832589+0000","last_peered":"2024-03-26T04:07:37.832589+0000","last_clean":"2024-03-26T04:07:37.832589+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:07:37.832589+0000","last_undegraded":"2024-03-26T04:07:37.832589+0000","last_fullsized":"2024-03-26T04:07:37.832589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:48.811706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:48.811706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":507,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585529+0000","last_change":"2024-03-26T04:05:29.540075+0000","last_active":"2024-03-26T04:07:35.585529+0000","last_peered":"2024-03-26T04:07:35.585529+0000","last_clean":"2024-03-26T04:07:35.585529+0000","last_became_active":"2024-03-26T04:05:29.539760+0000","last_became_peered":"2024-03-26T04:05:29.539760+0000","last_unstale":"2024-03-26T04:07:35.585529+0000","last_undegraded":"2024-03-26T04:07:35.585529+0000","last_fullsized":"2024-03-26T04:07:35.585529+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:19.384487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:19.384487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":439,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561875+0000","last_change":"2024-03-26T04:05:29.545345+0000","last_active":"2024-03-26T04:07:35.561875+0000","last_peered":"2024-03-26T04:07:35.561875+0000","last_clean":"2024-03-26T04:07:35.561875+0000","last_became_active":"2024-03-26T04:05:29.542754+0000","last_became_peered":"2024-03-26T04:05:29.542754+0000","last_unstale":"2024-03-26T04:07:35.561875+0000","last_undegraded":"2024-03-26T04:07:35.561875+0000","last_fullsized":"2024-03-26T04:07:35.561875+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:01:19.700471+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.700471+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":778,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561436+0000","last_change":"2024-03-26T04:03:07.149962+0000","last_active":"2024-03-26T04:07:35.561436+0000","last_peered":"2024-03-26T04:07:35.561436+0000","last_clean":"2024-03-26T04:07:35.561436+0000","last_became_active":"2024-03-26T04:02:52.506113+0000","last_became_peered":"2024-03-26T04:02:52.506113+0000","last_unstale":"2024-03-26T04:07:35.561436+0000","last_undegraded":"2024-03-26T04:07:35.561436+0000","last_fullsized":"2024-03-26T04:07:35.561436+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:07.149881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.149881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":363,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562247+0000","last_change":"2024-03-26T04:04:51.740064+0000","last_active":"2024-03-26T04:07:35.562247+0000","last_peered":"2024-03-26T04:07:35.562247+0000","last_clean":"2024-03-26T04:07:35.562247+0000","last_became_active":"2024-03-26T04:02:52.489626+0000","last_became_peered":"2024-03-26T04:02:52.489626+0000","last_unstale":"2024-03-26T04:07:35.562247+0000","last_undegraded":"2024-03-26T04:07:35.562247+0000","last_fullsized":"2024-03-26T04:07:35.562247+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.739925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.739925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":810,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833197+0000","last_change":"2024-03-26T04:05:54.943954+0000","last_active":"2024-03-26T04:07:37.833197+0000","last_peered":"2024-03-26T04:07:37.833197+0000","last_clean":"2024-03-26T04:07:37.833197+0000","last_became_active":"2024-03-26T04:02:52.498923+0000","last_became_peered":"2024-03-26T04:02:52.498923+0000","last_unstale":"2024-03-26T04:07:37.833197+0000","last_undegraded":"2024-03-26T04:07:37.833197+0000","last_fullsized":"2024-03-26T04:07:37.833197+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:54.943879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:54.943879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":825,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585721+0000","last_change":"2024-03-26T04:05:29.535609+0000","last_active":"2024-03-26T04:07:35.585721+0000","last_peered":"2024-03-26T04:07:35.585721+0000","last_clean":"2024-03-26T04:07:35.585721+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:07:35.585721+0000","last_undegraded":"2024-03-26T04:07:35.585721+0000","last_fullsized":"2024-03-26T04:07:35.585721+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:01:36.382121+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.382121+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":399,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562040+0000","last_change":"2024-03-26T04:05:29.538705+0000","last_active":"2024-03-26T04:07:35.562040+0000","last_peered":"2024-03-26T04:07:35.562040+0000","last_clean":"2024-03-26T04:07:35.562040+0000","last_became_active":"2024-03-26T04:05:29.537974+0000","last_became_peered":"2024-03-26T04:05:29.537974+0000","last_unstale":"2024-03-26T04:07:35.562040+0000","last_undegraded":"2024-03-26T04:07:35.562040+0000","last_fullsized":"2024-03-26T04:07:35.562040+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.709457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.709457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":792,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585581+0000","last_change":"2024-03-26T04:03:57.362649+0000","last_active":"2024-03-26T04:07:35.585581+0000","last_peered":"2024-03-26T04:07:35.585581+0000","last_clean":"2024-03-26T04:07:35.585581+0000","last_became_active":"2024-03-26T04:02:52.512541+0000","last_became_peered":"2024-03-26T04:02:52.512541+0000","last_unstale":"2024-03-26T04:07:35.585581+0000","last_undegraded":"2024-03-26T04:07:35.585581+0000","last_fullsized":"2024-03-26T04:07:35.585581+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:57.362566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:57.362566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":786,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701895+0000","last_change":"2024-03-26T04:05:29.536775+0000","last_active":"2024-03-26T04:07:37.701895+0000","last_peered":"2024-03-26T04:07:37.701895+0000","last_clean":"2024-03-26T04:07:37.701895+0000","last_became_active":"2024-03-26T04:05:29.536500+0000","last_became_peered":"2024-03-26T04:05:29.536500+0000","last_unstale":"2024-03-26T04:07:37.701895+0000","last_undegraded":"2024-03-26T04:07:37.701895+0000","last_fullsized":"2024-03-26T04:07:37.701895+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:01:35.104690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":844,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701892+0000","last_change":"2024-03-26T04:05:29.521309+0000","last_active":"2024-03-26T04:07:37.701892+0000","last_peered":"2024-03-26T04:07:37.701892+0000","last_clean":"2024-03-26T04:07:37.701892+0000","last_became_active":"2024-03-26T04:05:29.520817+0000","last_became_peered":"2024-03-26T04:05:29.520817+0000","last_unstale":"2024-03-26T04:07:37.701892+0000","last_undegraded":"2024-03-26T04:07:37.701892+0000","last_fullsized":"2024-03-26T04:07:37.701892+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:02:51.525641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:51.525641+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":774,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562983+0000","last_change":"2024-03-26T04:04:58.704933+0000","last_active":"2024-03-26T04:07:35.562983+0000","last_peered":"2024-03-26T04:07:35.562983+0000","last_clean":"2024-03-26T04:07:35.562983+0000","last_became_active":"2024-03-26T04:02:52.497109+0000","last_became_peered":"2024-03-26T04:02:52.497109+0000","last_unstale":"2024-03-26T04:07:35.562983+0000","last_undegraded":"2024-03-26T04:07:35.562983+0000","last_fullsized":"2024-03-26T04:07:35.562983+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:04:58.704813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":424,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585243+0000","last_change":"2024-03-26T04:03:53.367019+0000","last_active":"2024-03-26T04:07:35.585243+0000","last_peered":"2024-03-26T04:07:35.585243+0000","last_clean":"2024-03-26T04:07:35.585243+0000","last_became_active":"2024-03-26T04:02:52.511718+0000","last_became_peered":"2024-03-26T04:02:52.511718+0000","last_unstale":"2024-03-26T04:07:35.585243+0000","last_undegraded":"2024-03-26T04:07:35.585243+0000","last_fullsized":"2024-03-26T04:07:35.585243+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:53.366906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.366906+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":802,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832597+0000","last_change":"2024-03-26T04:07:32.845599+0000","last_active":"2024-03-26T04:07:37.832597+0000","last_peered":"2024-03-26T04:07:37.832597+0000","last_clean":"2024-03-26T04:07:37.832597+0000","last_became_active":"2024-03-26T04:05:29.533466+0000","last_became_peered":"2024-03-26T04:05:29.533466+0000","last_unstale":"2024-03-26T04:07:37.832597+0000","last_undegraded":"2024-03-26T04:07:37.832597+0000","last_fullsized":"2024-03-26T04:07:37.832597+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:32.845486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:32.845486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:54.200681+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"241'32","reported_seq":325,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.586430+0000","last_change":"2024-03-26T04:05:29.540362+0000","last_active":"2024-03-26T04:07:35.586430+0000","last_peered":"2024-03-26T04:07:35.586430+0000","last_clean":"2024-03-26T04:07:35.586430+0000","last_became_active":"2024-03-26T04:05:29.540006+0000","last_became_peered":"2024-03-26T04:05:29.540006+0000","last_unstale":"2024-03-26T04:07:35.586430+0000","last_undegraded":"2024-03-26T04:07:35.586430+0000","last_fullsized":"2024-03-26T04:07:35.586430+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'32","last_scrub_stamp":"2024-03-26T04:00:17.438826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.438826+0000","objects_scrubbed":7,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":39,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":774,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.703023+0000","last_change":"2024-03-26T04:05:29.522233+0000","last_active":"2024-03-26T04:07:37.703023+0000","last_peered":"2024-03-26T04:07:37.703023+0000","last_clean":"2024-03-26T04:07:37.703023+0000","last_became_active":"2024-03-26T04:05:29.521886+0000","last_became_peered":"2024-03-26T04:05:29.521886+0000","last_unstale":"2024-03-26T04:07:37.703023+0000","last_undegraded":"2024-03-26T04:07:37.703023+0000","last_fullsized":"2024-03-26T04:07:37.703023+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.428232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.428232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"379'14","reported_seq":197,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833794+0000","last_change":"2024-03-26T04:05:29.541223+0000","last_active":"2024-03-26T04:07:37.833794+0000","last_peered":"2024-03-26T04:07:37.833794+0000","last_clean":"2024-03-26T04:07:37.833794+0000","last_became_active":"2024-03-26T04:05:29.540855+0000","last_became_peered":"2024-03-26T04:05:29.540855+0000","last_unstale":"2024-03-26T04:07:37.833794+0000","last_undegraded":"2024-03-26T04:07:37.833794+0000","last_fullsized":"2024-03-26T04:07:37.833794+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'12","last_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_deep_scrub":"334'12","last_deep_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.907441+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":819,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562342+0000","last_change":"2024-03-26T04:04:17.541024+0000","last_active":"2024-03-26T04:07:35.562342+0000","last_peered":"2024-03-26T04:07:35.562342+0000","last_clean":"2024-03-26T04:07:35.562342+0000","last_became_active":"2024-03-26T04:02:52.492361+0000","last_became_peered":"2024-03-26T04:02:52.492361+0000","last_unstale":"2024-03-26T04:07:35.562342+0000","last_undegraded":"2024-03-26T04:07:35.562342+0000","last_fullsized":"2024-03-26T04:07:35.562342+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.538770+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":741,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833416+0000","last_change":"2024-03-26T04:06:21.728756+0000","last_active":"2024-03-26T04:07:37.833416+0000","last_peered":"2024-03-26T04:07:37.833416+0000","last_clean":"2024-03-26T04:07:37.833416+0000","last_became_active":"2024-03-26T04:02:52.505421+0000","last_became_peered":"2024-03-26T04:02:52.505421+0000","last_unstale":"2024-03-26T04:07:37.833416+0000","last_undegraded":"2024-03-26T04:07:37.833416+0000","last_fullsized":"2024-03-26T04:07:37.833416+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:06:21.728608+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:06:21.728608+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":687,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834206+0000","last_change":"2024-03-26T04:05:29.548434+0000","last_active":"2024-03-26T04:07:37.834206+0000","last_peered":"2024-03-26T04:07:37.834206+0000","last_clean":"2024-03-26T04:07:37.834206+0000","last_became_active":"2024-03-26T04:05:29.546602+0000","last_became_peered":"2024-03-26T04:05:29.546602+0000","last_unstale":"2024-03-26T04:07:37.834206+0000","last_undegraded":"2024-03-26T04:07:37.834206+0000","last_fullsized":"2024-03-26T04:07:37.834206+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:01:13.279325+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:01:13.279325+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":353,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585479+0000","last_change":"2024-03-26T04:03:58.337609+0000","last_active":"2024-03-26T04:07:35.585479+0000","last_peered":"2024-03-26T04:07:35.585479+0000","last_clean":"2024-03-26T04:07:35.585479+0000","last_became_active":"2024-03-26T04:02:52.505042+0000","last_became_peered":"2024-03-26T04:02:52.505042+0000","last_unstale":"2024-03-26T04:07:35.585479+0000","last_undegraded":"2024-03-26T04:07:35.585479+0000","last_fullsized":"2024-03-26T04:07:35.585479+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:58.337496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:58.337496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":479,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701201+0000","last_change":"2024-03-26T04:04:57.523173+0000","last_active":"2024-03-26T04:07:37.701201+0000","last_peered":"2024-03-26T04:07:37.701201+0000","last_clean":"2024-03-26T04:07:37.701201+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:07:37.701201+0000","last_undegraded":"2024-03-26T04:07:37.701201+0000","last_fullsized":"2024-03-26T04:07:37.701201+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:57.522794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:57.522794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"241'27","reported_seq":874,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563471+0000","last_change":"2024-03-26T04:05:29.537906+0000","last_active":"2024-03-26T04:07:35.563471+0000","last_peered":"2024-03-26T04:07:35.563471+0000","last_clean":"2024-03-26T04:07:35.563471+0000","last_became_active":"2024-03-26T04:05:29.537550+0000","last_became_peered":"2024-03-26T04:05:29.537550+0000","last_unstale":"2024-03-26T04:07:35.563471+0000","last_undegraded":"2024-03-26T04:07:35.563471+0000","last_fullsized":"2024-03-26T04:07:35.563471+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:00:25.343896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:25.343896+0000","objects_scrubbed":6,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":794,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833187+0000","last_change":"2024-03-26T04:06:05.957492+0000","last_active":"2024-03-26T04:07:37.833187+0000","last_peered":"2024-03-26T04:07:37.833187+0000","last_clean":"2024-03-26T04:07:37.833187+0000","last_became_active":"2024-03-26T04:01:35.799877+0000","last_became_peered":"2024-03-26T04:01:35.799877+0000","last_unstale":"2024-03-26T04:07:37.833187+0000","last_undegraded":"2024-03-26T04:07:37.833187+0000","last_fullsized":"2024-03-26T04:07:37.833187+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:05.957415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:05.957415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":507,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585697+0000","last_change":"2024-03-26T04:04:10.223664+0000","last_active":"2024-03-26T04:07:35.585697+0000","last_peered":"2024-03-26T04:07:35.585697+0000","last_clean":"2024-03-26T04:07:35.585697+0000","last_became_active":"2024-03-26T04:02:52.502225+0000","last_became_peered":"2024-03-26T04:02:52.502225+0000","last_unstale":"2024-03-26T04:07:35.585697+0000","last_undegraded":"2024-03-26T04:07:35.585697+0000","last_fullsized":"2024-03-26T04:07:35.585697+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:04:10.223350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":741,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561715+0000","last_change":"2024-03-26T04:05:29.540410+0000","last_active":"2024-03-26T04:07:35.561715+0000","last_peered":"2024-03-26T04:07:35.561715+0000","last_clean":"2024-03-26T04:07:35.561715+0000","last_became_active":"2024-03-26T04:05:29.540070+0000","last_became_peered":"2024-03-26T04:05:29.540070+0000","last_unstale":"2024-03-26T04:07:35.561715+0000","last_undegraded":"2024-03-26T04:07:35.561715+0000","last_fullsized":"2024-03-26T04:07:35.561715+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:48.062863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:02:48.062863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":225,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585678+0000","last_change":"2024-03-26T04:05:29.539905+0000","last_active":"2024-03-26T04:07:35.585678+0000","last_peered":"2024-03-26T04:07:35.585678+0000","last_clean":"2024-03-26T04:07:35.585678+0000","last_became_active":"2024-03-26T04:05:29.539596+0000","last_became_peered":"2024-03-26T04:05:29.539596+0000","last_unstale":"2024-03-26T04:07:35.585678+0000","last_undegraded":"2024-03-26T04:07:35.585678+0000","last_fullsized":"2024-03-26T04:07:35.585678+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:00:21.475271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.475271+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"421'21382","reported_seq":21537,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.080368+0000","last_change":"2024-03-26T04:03:47.643174+0000","last_active":"2024-03-26T04:07:36.080368+0000","last_peered":"2024-03-26T04:07:36.080368+0000","last_clean":"2024-03-26T04:07:36.080368+0000","last_became_active":"2024-03-26T04:03:47.642762+0000","last_became_peered":"2024-03-26T04:03:47.642762+0000","last_unstale":"2024-03-26T04:07:36.080368+0000","last_undegraded":"2024-03-26T04:07:36.080368+0000","last_fullsized":"2024-03-26T04:07:36.080368+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21382,"num_write_kb":684224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":493,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562527+0000","last_change":"2024-03-26T04:05:29.537569+0000","last_active":"2024-03-26T04:07:35.562527+0000","last_peered":"2024-03-26T04:07:35.562527+0000","last_clean":"2024-03-26T04:07:35.562527+0000","last_became_active":"2024-03-26T04:05:29.537211+0000","last_became_peered":"2024-03-26T04:05:29.537211+0000","last_unstale":"2024-03-26T04:07:35.562527+0000","last_undegraded":"2024-03-26T04:07:35.562527+0000","last_fullsized":"2024-03-26T04:07:35.562527+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:50.068294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:50.068294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":806,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833920+0000","last_change":"2024-03-26T04:06:27.796450+0000","last_active":"2024-03-26T04:07:37.833920+0000","last_peered":"2024-03-26T04:07:37.833920+0000","last_clean":"2024-03-26T04:07:37.833920+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:07:37.833920+0000","last_undegraded":"2024-03-26T04:07:37.833920+0000","last_fullsized":"2024-03-26T04:07:37.833920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T04:06:27.796333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:27.796333+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":773,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832466+0000","last_change":"2024-03-26T04:07:25.796444+0000","last_active":"2024-03-26T04:07:37.832466+0000","last_peered":"2024-03-26T04:07:37.832466+0000","last_clean":"2024-03-26T04:07:37.832466+0000","last_became_active":"2024-03-26T04:02:52.505008+0000","last_became_peered":"2024-03-26T04:02:52.505008+0000","last_unstale":"2024-03-26T04:07:37.832466+0000","last_undegraded":"2024-03-26T04:07:37.832466+0000","last_fullsized":"2024-03-26T04:07:37.832466+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:07:25.796341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.796341+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:27.516967+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":1031,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832958+0000","last_change":"2024-03-26T04:05:45.805637+0000","last_active":"2024-03-26T04:07:37.832958+0000","last_peered":"2024-03-26T04:07:37.832958+0000","last_clean":"2024-03-26T04:07:37.832958+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:07:37.832958+0000","last_undegraded":"2024-03-26T04:07:37.832958+0000","last_fullsized":"2024-03-26T04:07:37.832958+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:05:45.805513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:45.805513+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"421'21634","reported_seq":22203,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834437+0000","last_change":"2024-03-26T04:07:35.582310+0000","last_active":"2024-03-26T04:07:37.834437+0000","last_peered":"2024-03-26T04:07:37.834437+0000","last_clean":"2024-03-26T04:07:37.834437+0000","last_became_active":"2024-03-26T04:07:35.562368+0000","last_became_peered":"2024-03-26T04:07:35.562368+0000","last_unstale":"2024-03-26T04:07:37.834437+0000","last_undegraded":"2024-03-26T04:07:37.834437+0000","last_fullsized":"2024-03-26T04:07:37.834437+0000","mapping_epoch":420,"log_start":"404'18500","ondisk_log_start":"404'18500","created":349,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":10,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21634,"num_write_kb":692288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":371,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832732+0000","last_change":"2024-03-26T04:06:01.998434+0000","last_active":"2024-03-26T04:07:37.832732+0000","last_peered":"2024-03-26T04:07:37.832732+0000","last_clean":"2024-03-26T04:07:37.832732+0000","last_became_active":"2024-03-26T04:02:52.505076+0000","last_became_peered":"2024-03-26T04:02:52.505076+0000","last_unstale":"2024-03-26T04:07:37.832732+0000","last_undegraded":"2024-03-26T04:07:37.832732+0000","last_fullsized":"2024-03-26T04:07:37.832732+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:01.998326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:06:01.998326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":793,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833453+0000","last_change":"2024-03-26T04:06:06.861624+0000","last_active":"2024-03-26T04:07:37.833453+0000","last_peered":"2024-03-26T04:07:37.833453+0000","last_clean":"2024-03-26T04:07:37.833453+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:07:37.833453+0000","last_undegraded":"2024-03-26T04:07:37.833453+0000","last_fullsized":"2024-03-26T04:07:37.833453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:06.861506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:06.861506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":889,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701278+0000","last_change":"2024-03-26T04:04:19.443761+0000","last_active":"2024-03-26T04:07:37.701278+0000","last_peered":"2024-03-26T04:07:37.701278+0000","last_clean":"2024-03-26T04:07:37.701278+0000","last_became_active":"2024-03-26T04:02:52.513309+0000","last_became_peered":"2024-03-26T04:02:52.513309+0000","last_unstale":"2024-03-26T04:07:37.701278+0000","last_undegraded":"2024-03-26T04:07:37.701278+0000","last_fullsized":"2024-03-26T04:07:37.701278+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:04:19.443608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.443608+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":690,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832649+0000","last_change":"2024-03-26T04:05:29.533866+0000","last_active":"2024-03-26T04:07:37.832649+0000","last_peered":"2024-03-26T04:07:37.832649+0000","last_clean":"2024-03-26T04:07:37.832649+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:07:37.832649+0000","last_undegraded":"2024-03-26T04:07:37.832649+0000","last_fullsized":"2024-03-26T04:07:37.832649+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:11.271561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:11.271561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"421'21466","reported_seq":21621,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.080323+0000","last_change":"2024-03-26T04:03:47.643711+0000","last_active":"2024-03-26T04:07:36.080323+0000","last_peered":"2024-03-26T04:07:36.080323+0000","last_clean":"2024-03-26T04:07:36.080323+0000","last_became_active":"2024-03-26T04:03:47.643384+0000","last_became_peered":"2024-03-26T04:03:47.643384+0000","last_unstale":"2024-03-26T04:07:36.080323+0000","last_undegraded":"2024-03-26T04:07:36.080323+0000","last_fullsized":"2024-03-26T04:07:36.080323+0000","mapping_epoch":349,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21466,"num_write_kb":686912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":645,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832510+0000","last_change":"2024-03-26T04:05:29.543839+0000","last_active":"2024-03-26T04:07:37.832510+0000","last_peered":"2024-03-26T04:07:37.832510+0000","last_clean":"2024-03-26T04:07:37.832510+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:07:37.832510+0000","last_undegraded":"2024-03-26T04:07:37.832510+0000","last_fullsized":"2024-03-26T04:07:37.832510+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:37.891864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.891864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":690,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833181+0000","last_change":"2024-03-26T04:05:29.535670+0000","last_active":"2024-03-26T04:07:37.833181+0000","last_peered":"2024-03-26T04:07:37.833181+0000","last_clean":"2024-03-26T04:07:37.833181+0000","last_became_active":"2024-03-26T04:05:29.534559+0000","last_became_peered":"2024-03-26T04:05:29.534559+0000","last_unstale":"2024-03-26T04:07:37.833181+0000","last_undegraded":"2024-03-26T04:07:37.833181+0000","last_fullsized":"2024-03-26T04:07:37.833181+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.965300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":578,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701786+0000","last_change":"2024-03-26T04:04:14.382770+0000","last_active":"2024-03-26T04:07:37.701786+0000","last_peered":"2024-03-26T04:07:37.701786+0000","last_clean":"2024-03-26T04:07:37.701786+0000","last_became_active":"2024-03-26T04:02:52.500569+0000","last_became_peered":"2024-03-26T04:02:52.500569+0000","last_unstale":"2024-03-26T04:07:37.701786+0000","last_undegraded":"2024-03-26T04:07:37.701786+0000","last_fullsized":"2024-03-26T04:07:37.701786+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:14.382655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:14.382655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"356'51","reported_seq":861,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584980+0000","last_change":"2024-03-26T04:04:03.371506+0000","last_active":"2024-03-26T04:07:35.584980+0000","last_peered":"2024-03-26T04:07:35.584980+0000","last_clean":"2024-03-26T04:07:35.584980+0000","last_became_active":"2024-03-26T04:01:35.818705+0000","last_became_peered":"2024-03-26T04:01:35.818705+0000","last_unstale":"2024-03-26T04:07:35.584980+0000","last_undegraded":"2024-03-26T04:07:35.584980+0000","last_fullsized":"2024-03-26T04:07:35.584980+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'51","last_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.371335+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"421'22006","reported_seq":29865,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834341+0000","last_change":"2024-03-26T04:07:04.562327+0000","last_active":"2024-03-26T04:07:37.834341+0000","last_peered":"2024-03-26T04:07:37.834341+0000","last_clean":"2024-03-26T04:07:37.834341+0000","last_became_active":"2024-03-26T04:07:04.537258+0000","last_became_peered":"2024-03-26T04:07:04.537258+0000","last_unstale":"2024-03-26T04:07:37.834341+0000","last_undegraded":"2024-03-26T04:07:37.834341+0000","last_fullsized":"2024-03-26T04:07:37.834341+0000","mapping_epoch":407,"log_start":"400'18300","ondisk_log_start":"400'18300","created":349,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22006,"num_write_kb":704192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3844,"num_bytes_recovered":251854848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":758,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562172+0000","last_change":"2024-03-26T04:05:29.543248+0000","last_active":"2024-03-26T04:07:35.562172+0000","last_peered":"2024-03-26T04:07:35.562172+0000","last_clean":"2024-03-26T04:07:35.562172+0000","last_became_active":"2024-03-26T04:05:29.541737+0000","last_became_peered":"2024-03-26T04:05:29.541737+0000","last_unstale":"2024-03-26T04:07:35.562172+0000","last_undegraded":"2024-03-26T04:07:35.562172+0000","last_fullsized":"2024-03-26T04:07:35.562172+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.410420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.410420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":707,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832687+0000","last_change":"2024-03-26T04:07:37.592031+0000","last_active":"2024-03-26T04:07:37.832687+0000","last_peered":"2024-03-26T04:07:37.832687+0000","last_clean":"2024-03-26T04:07:37.832687+0000","last_became_active":"2024-03-26T04:05:29.524644+0000","last_became_peered":"2024-03-26T04:05:29.524644+0000","last_unstale":"2024-03-26T04:07:37.832687+0000","last_undegraded":"2024-03-26T04:07:37.832687+0000","last_fullsized":"2024-03-26T04:07:37.832687+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.591946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.591946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.371327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":539,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585927+0000","last_change":"2024-03-26T04:07:34.405175+0000","last_active":"2024-03-26T04:07:35.585927+0000","last_peered":"2024-03-26T04:07:35.585927+0000","last_clean":"2024-03-26T04:07:35.585927+0000","last_became_active":"2024-03-26T04:05:29.531951+0000","last_became_peered":"2024-03-26T04:05:29.531951+0000","last_unstale":"2024-03-26T04:07:35.585927+0000","last_undegraded":"2024-03-26T04:07:35.585927+0000","last_fullsized":"2024-03-26T04:07:35.585927+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.404972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.404972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.885121+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":855,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562875+0000","last_change":"2024-03-26T04:04:55.743092+0000","last_active":"2024-03-26T04:07:35.562875+0000","last_peered":"2024-03-26T04:07:35.562875+0000","last_clean":"2024-03-26T04:07:35.562875+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:07:35.562875+0000","last_undegraded":"2024-03-26T04:07:35.562875+0000","last_fullsized":"2024-03-26T04:07:35.562875+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:04:55.742967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.742967+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"421'21544","reported_seq":30915,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.093306+0000","last_change":"2024-03-26T04:07:03.053873+0000","last_active":"2024-03-26T04:07:36.093306+0000","last_peered":"2024-03-26T04:07:36.093306+0000","last_clean":"2024-03-26T04:07:36.093306+0000","last_became_active":"2024-03-26T04:07:03.030042+0000","last_became_peered":"2024-03-26T04:07:03.030042+0000","last_unstale":"2024-03-26T04:07:36.093306+0000","last_undegraded":"2024-03-26T04:07:36.093306+0000","last_fullsized":"2024-03-26T04:07:36.093306+0000","mapping_epoch":405,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21544,"num_write_kb":689408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4603,"num_bytes_recovered":301662208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":580,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561671+0000","last_change":"2024-03-26T04:05:29.538976+0000","last_active":"2024-03-26T04:07:35.561671+0000","last_peered":"2024-03-26T04:07:35.561671+0000","last_clean":"2024-03-26T04:07:35.561671+0000","last_became_active":"2024-03-26T04:05:29.538433+0000","last_became_peered":"2024-03-26T04:05:29.538433+0000","last_unstale":"2024-03-26T04:07:35.561671+0000","last_undegraded":"2024-03-26T04:07:35.561671+0000","last_fullsized":"2024-03-26T04:07:35.561671+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:37.939498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.939498+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":366,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702339+0000","last_change":"2024-03-26T04:04:03.264229+0000","last_active":"2024-03-26T04:07:37.702339+0000","last_peered":"2024-03-26T04:07:37.702339+0000","last_clean":"2024-03-26T04:07:37.702339+0000","last_became_active":"2024-03-26T04:02:52.520574+0000","last_became_peered":"2024-03-26T04:02:52.520574+0000","last_unstale":"2024-03-26T04:07:37.702339+0000","last_undegraded":"2024-03-26T04:07:37.702339+0000","last_fullsized":"2024-03-26T04:07:37.702339+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:03.264133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.264133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":663,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584670+0000","last_change":"2024-03-26T04:04:17.110459+0000","last_active":"2024-03-26T04:07:35.584670+0000","last_peered":"2024-03-26T04:07:35.584670+0000","last_clean":"2024-03-26T04:07:35.584670+0000","last_became_active":"2024-03-26T04:02:52.495493+0000","last_became_peered":"2024-03-26T04:02:52.495493+0000","last_unstale":"2024-03-26T04:07:35.584670+0000","last_undegraded":"2024-03-26T04:07:35.584670+0000","last_fullsized":"2024-03-26T04:07:35.584670+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:04:17.110315+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.110315+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":778,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701683+0000","last_change":"2024-03-26T04:04:52.483535+0000","last_active":"2024-03-26T04:07:37.701683+0000","last_peered":"2024-03-26T04:07:37.701683+0000","last_clean":"2024-03-26T04:07:37.701683+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:07:37.701683+0000","last_undegraded":"2024-03-26T04:07:37.701683+0000","last_fullsized":"2024-03-26T04:07:37.701683+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:52.483429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:04:52.483429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"379'51","reported_seq":1170,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.586431+0000","last_change":"2024-03-26T04:05:29.540308+0000","last_active":"2024-03-26T04:07:35.586431+0000","last_peered":"2024-03-26T04:07:35.586431+0000","last_clean":"2024-03-26T04:07:35.586431+0000","last_became_active":"2024-03-26T04:05:29.539914+0000","last_became_peered":"2024-03-26T04:05:29.539914+0000","last_unstale":"2024-03-26T04:07:35.586431+0000","last_undegraded":"2024-03-26T04:07:35.586431+0000","last_fullsized":"2024-03-26T04:07:35.586431+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T04:01:23.451122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.451122+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":769,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563166+0000","last_change":"2024-03-26T04:05:04.831962+0000","last_active":"2024-03-26T04:07:35.563166+0000","last_peered":"2024-03-26T04:07:35.563166+0000","last_clean":"2024-03-26T04:07:35.563166+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:07:35.563166+0000","last_undegraded":"2024-03-26T04:07:35.563166+0000","last_fullsized":"2024-03-26T04:07:35.563166+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:04.831457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:04.831457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"421'21530","reported_seq":38483,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832979+0000","last_change":"2024-03-26T04:06:19.310400+0000","last_active":"2024-03-26T04:07:37.832979+0000","last_peered":"2024-03-26T04:07:37.832979+0000","last_clean":"2024-03-26T04:07:37.832979+0000","last_became_active":"2024-03-26T04:06:19.282014+0000","last_became_peered":"2024-03-26T04:06:19.282014+0000","last_unstale":"2024-03-26T04:07:37.832979+0000","last_undegraded":"2024-03-26T04:07:37.832979+0000","last_fullsized":"2024-03-26T04:07:37.832979+0000","mapping_epoch":392,"log_start":"400'17775","ondisk_log_start":"400'17775","created":349,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":1247,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21530,"num_write_kb":688960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8394,"num_bytes_recovered":550109184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":612,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562039+0000","last_change":"2024-03-26T04:04:11.458855+0000","last_active":"2024-03-26T04:07:35.562039+0000","last_peered":"2024-03-26T04:07:35.562039+0000","last_clean":"2024-03-26T04:07:35.562039+0000","last_became_active":"2024-03-26T04:00:32.425416+0000","last_became_peered":"2024-03-26T04:00:32.425416+0000","last_unstale":"2024-03-26T04:07:35.562039+0000","last_undegraded":"2024-03-26T04:07:35.562039+0000","last_fullsized":"2024-03-26T04:07:35.562039+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:11.458734+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:04:11.458734+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":816,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:38.023167+0000","last_change":"2024-03-26T04:07:38.023167+0000","last_active":"2024-03-26T04:07:38.023167+0000","last_peered":"2024-03-26T04:07:38.023167+0000","last_clean":"2024-03-26T04:07:38.023167+0000","last_became_active":"2024-03-26T04:05:29.542557+0000","last_became_peered":"2024-03-26T04:05:29.542557+0000","last_unstale":"2024-03-26T04:07:38.023167+0000","last_undegraded":"2024-03-26T04:07:38.023167+0000","last_fullsized":"2024-03-26T04:07:38.023167+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.023066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.026959+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":791,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584515+0000","last_change":"2024-03-26T04:05:29.535985+0000","last_active":"2024-03-26T04:07:35.584515+0000","last_peered":"2024-03-26T04:07:35.584515+0000","last_clean":"2024-03-26T04:07:35.584515+0000","last_became_active":"2024-03-26T04:05:29.533776+0000","last_became_peered":"2024-03-26T04:05:29.533776+0000","last_unstale":"2024-03-26T04:07:35.584515+0000","last_undegraded":"2024-03-26T04:07:35.584515+0000","last_fullsized":"2024-03-26T04:07:35.584515+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.324845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.324845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":716,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832889+0000","last_change":"2024-03-26T04:05:29.544757+0000","last_active":"2024-03-26T04:07:37.832889+0000","last_peered":"2024-03-26T04:07:37.832889+0000","last_clean":"2024-03-26T04:07:37.832889+0000","last_became_active":"2024-03-26T04:05:29.543967+0000","last_became_peered":"2024-03-26T04:05:29.543967+0000","last_unstale":"2024-03-26T04:07:37.832889+0000","last_undegraded":"2024-03-26T04:07:37.832889+0000","last_fullsized":"2024-03-26T04:07:37.832889+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:27.325285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:01:27.325285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"371'29","reported_seq":851,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585204+0000","last_change":"2024-03-26T04:04:59.217984+0000","last_active":"2024-03-26T04:07:35.585204+0000","last_peered":"2024-03-26T04:07:35.585204+0000","last_clean":"2024-03-26T04:07:35.585204+0000","last_became_active":"2024-03-26T04:01:35.795965+0000","last_became_peered":"2024-03-26T04:01:35.795965+0000","last_unstale":"2024-03-26T04:07:35.585204+0000","last_undegraded":"2024-03-26T04:07:35.585204+0000","last_fullsized":"2024-03-26T04:07:35.585204+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'29","last_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:04:59.217868+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":710,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835079+0000","last_change":"2024-03-26T04:05:29.536479+0000","last_active":"2024-03-26T04:07:37.835079+0000","last_peered":"2024-03-26T04:07:37.835079+0000","last_clean":"2024-03-26T04:07:37.835079+0000","last_became_active":"2024-03-26T04:05:29.536165+0000","last_became_peered":"2024-03-26T04:05:29.536165+0000","last_unstale":"2024-03-26T04:07:37.835079+0000","last_undegraded":"2024-03-26T04:07:37.835079+0000","last_fullsized":"2024-03-26T04:07:37.835079+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.247532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.247532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"421'21542","reported_seq":21697,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.061377+0000","last_change":"2024-03-26T04:03:47.644681+0000","last_active":"2024-03-26T04:07:36.061377+0000","last_peered":"2024-03-26T04:07:36.061377+0000","last_clean":"2024-03-26T04:07:36.061377+0000","last_became_active":"2024-03-26T04:03:47.644384+0000","last_became_peered":"2024-03-26T04:03:47.644384+0000","last_unstale":"2024-03-26T04:07:36.061377+0000","last_undegraded":"2024-03-26T04:07:36.061377+0000","last_fullsized":"2024-03-26T04:07:36.061377+0000","mapping_epoch":349,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131072,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21542,"num_write_kb":689408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":506,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702360+0000","last_change":"2024-03-26T04:04:02.403906+0000","last_active":"2024-03-26T04:07:37.702360+0000","last_peered":"2024-03-26T04:07:37.702360+0000","last_clean":"2024-03-26T04:07:37.702360+0000","last_became_active":"2024-03-26T04:02:52.504066+0000","last_became_peered":"2024-03-26T04:02:52.504066+0000","last_unstale":"2024-03-26T04:07:37.702360+0000","last_undegraded":"2024-03-26T04:07:37.702360+0000","last_fullsized":"2024-03-26T04:07:37.702360+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:04:02.403784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1068,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832325+0000","last_change":"2024-03-26T04:06:03.860318+0000","last_active":"2024-03-26T04:07:37.832325+0000","last_peered":"2024-03-26T04:07:37.832325+0000","last_clean":"2024-03-26T04:07:37.832325+0000","last_became_active":"2024-03-26T04:00:27.442487+0000","last_became_peered":"2024-03-26T04:00:27.442487+0000","last_unstale":"2024-03-26T04:07:37.832325+0000","last_undegraded":"2024-03-26T04:07:37.832325+0000","last_fullsized":"2024-03-26T04:07:37.832325+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:06:03.860237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:06:03.860237+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700854+0000","last_change":"2024-03-26T04:04:56.545589+0000","last_active":"2024-03-26T04:07:37.700854+0000","last_peered":"2024-03-26T04:07:37.700854+0000","last_clean":"2024-03-26T04:07:37.700854+0000","last_became_active":"2024-03-26T04:02:52.499614+0000","last_became_peered":"2024-03-26T04:02:52.499614+0000","last_unstale":"2024-03-26T04:07:37.700854+0000","last_undegraded":"2024-03-26T04:07:37.700854+0000","last_fullsized":"2024-03-26T04:07:37.700854+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:56.545464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:56.545464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":984,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702601+0000","last_change":"2024-03-26T04:04:20.432746+0000","last_active":"2024-03-26T04:07:37.702601+0000","last_peered":"2024-03-26T04:07:37.702601+0000","last_clean":"2024-03-26T04:07:37.702601+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:07:37.702601+0000","last_undegraded":"2024-03-26T04:07:37.702601+0000","last_fullsized":"2024-03-26T04:07:37.702601+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:04:20.432551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:20.432551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":592,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834087+0000","last_change":"2024-03-26T04:05:29.547970+0000","last_active":"2024-03-26T04:07:37.834087+0000","last_peered":"2024-03-26T04:07:37.834087+0000","last_clean":"2024-03-26T04:07:37.834087+0000","last_became_active":"2024-03-26T04:05:29.544566+0000","last_became_peered":"2024-03-26T04:05:29.544566+0000","last_unstale":"2024-03-26T04:07:37.834087+0000","last_undegraded":"2024-03-26T04:07:37.834087+0000","last_fullsized":"2024-03-26T04:07:37.834087+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.381181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.381181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"421'21530","reported_seq":26671,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.062672+0000","last_change":"2024-03-26T04:07:19.157035+0000","last_active":"2024-03-26T04:07:36.062672+0000","last_peered":"2024-03-26T04:07:36.062672+0000","last_clean":"2024-03-26T04:07:36.062672+0000","last_became_active":"2024-03-26T04:07:19.140559+0000","last_became_peered":"2024-03-26T04:07:19.140559+0000","last_unstale":"2024-03-26T04:07:36.062672+0000","last_undegraded":"2024-03-26T04:07:36.062672+0000","last_fullsized":"2024-03-26T04:07:36.062672+0000","mapping_epoch":414,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21530,"num_write_kb":688960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":162856960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":666,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584875+0000","last_change":"2024-03-26T04:04:07.285956+0000","last_active":"2024-03-26T04:07:35.584875+0000","last_peered":"2024-03-26T04:07:35.584875+0000","last_clean":"2024-03-26T04:07:35.584875+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:07:35.584875+0000","last_undegraded":"2024-03-26T04:07:35.584875+0000","last_fullsized":"2024-03-26T04:07:35.584875+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:07.285700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:07.285700+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":790,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.586070+0000","last_change":"2024-03-26T04:05:29.522174+0000","last_active":"2024-03-26T04:07:35.586070+0000","last_peered":"2024-03-26T04:07:35.586070+0000","last_clean":"2024-03-26T04:07:35.586070+0000","last_became_active":"2024-03-26T04:05:29.521425+0000","last_became_peered":"2024-03-26T04:05:29.521425+0000","last_unstale":"2024-03-26T04:07:35.586070+0000","last_undegraded":"2024-03-26T04:07:35.586070+0000","last_fullsized":"2024-03-26T04:07:35.586070+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.552445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.552445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":805,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561957+0000","last_change":"2024-03-26T04:03:56.329590+0000","last_active":"2024-03-26T04:07:35.561957+0000","last_peered":"2024-03-26T04:07:35.561957+0000","last_clean":"2024-03-26T04:07:35.561957+0000","last_became_active":"2024-03-26T04:02:52.508294+0000","last_became_peered":"2024-03-26T04:02:52.508294+0000","last_unstale":"2024-03-26T04:07:35.561957+0000","last_undegraded":"2024-03-26T04:07:35.561957+0000","last_fullsized":"2024-03-26T04:07:35.561957+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:56.329511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:56.329511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1941,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:38.290441+0000","last_change":"2024-03-26T04:07:31.850944+0000","last_active":"2024-03-26T04:07:38.290441+0000","last_peered":"2024-03-26T04:07:38.290441+0000","last_clean":"2024-03-26T04:07:38.290441+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:07:38.290441+0000","last_undegraded":"2024-03-26T04:07:38.290441+0000","last_fullsized":"2024-03-26T04:07:38.290441+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:07:31.850790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.850790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.959779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1127,"num_read_kb":1122,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":802,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563122+0000","last_change":"2024-03-26T04:05:29.542004+0000","last_active":"2024-03-26T04:07:35.563122+0000","last_peered":"2024-03-26T04:07:35.563122+0000","last_clean":"2024-03-26T04:07:35.563122+0000","last_became_active":"2024-03-26T04:05:29.538986+0000","last_became_peered":"2024-03-26T04:05:29.538986+0000","last_unstale":"2024-03-26T04:07:35.563122+0000","last_undegraded":"2024-03-26T04:07:35.563122+0000","last_fullsized":"2024-03-26T04:07:35.563122+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:00:18.345305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.345305+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":482,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585808+0000","last_change":"2024-03-26T04:05:01.160620+0000","last_active":"2024-03-26T04:07:35.585808+0000","last_peered":"2024-03-26T04:07:35.585808+0000","last_clean":"2024-03-26T04:07:35.585808+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:07:35.585808+0000","last_undegraded":"2024-03-26T04:07:35.585808+0000","last_fullsized":"2024-03-26T04:07:35.585808+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:05:01.160427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"421'24449","reported_seq":24809,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702239+0000","last_change":"2024-03-26T04:05:23.910568+0000","last_active":"2024-03-26T04:07:37.702239+0000","last_peered":"2024-03-26T04:07:37.702239+0000","last_clean":"2024-03-26T04:07:37.702239+0000","last_became_active":"2024-03-26T04:03:47.643956+0000","last_became_peered":"2024-03-26T04:03:47.643956+0000","last_unstale":"2024-03-26T04:07:37.702239+0000","last_undegraded":"2024-03-26T04:07:37.702239+0000","last_fullsized":"2024-03-26T04:07:37.702239+0000","mapping_epoch":349,"log_start":"404'21403","ondisk_log_start":"404'21403","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":4910,"log_size":3046,"log_dups_size":3000,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21172,"num_write_kb":677441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":621,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561242+0000","last_change":"2024-03-26T04:04:53.785962+0000","last_active":"2024-03-26T04:07:35.561242+0000","last_peered":"2024-03-26T04:07:35.561242+0000","last_clean":"2024-03-26T04:07:35.561242+0000","last_became_active":"2024-03-26T04:02:55.535259+0000","last_became_peered":"2024-03-26T04:02:55.535259+0000","last_unstale":"2024-03-26T04:07:35.561242+0000","last_undegraded":"2024-03-26T04:07:35.561242+0000","last_fullsized":"2024-03-26T04:07:35.561242+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:53.785762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:53.785762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":771,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584628+0000","last_change":"2024-03-26T04:05:29.541309+0000","last_active":"2024-03-26T04:07:35.584628+0000","last_peered":"2024-03-26T04:07:35.584628+0000","last_clean":"2024-03-26T04:07:35.584628+0000","last_became_active":"2024-03-26T04:05:29.540971+0000","last_became_peered":"2024-03-26T04:05:29.540971+0000","last_unstale":"2024-03-26T04:07:35.584628+0000","last_undegraded":"2024-03-26T04:07:35.584628+0000","last_fullsized":"2024-03-26T04:07:35.584628+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.502151+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":792,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.586279+0000","last_change":"2024-03-26T04:05:29.532330+0000","last_active":"2024-03-26T04:07:35.586279+0000","last_peered":"2024-03-26T04:07:35.586279+0000","last_clean":"2024-03-26T04:07:35.586279+0000","last_became_active":"2024-03-26T04:05:29.532029+0000","last_became_peered":"2024-03-26T04:05:29.532029+0000","last_unstale":"2024-03-26T04:07:35.586279+0000","last_undegraded":"2024-03-26T04:07:35.586279+0000","last_fullsized":"2024-03-26T04:07:35.586279+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:34.550087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:00:34.550087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"358'44","reported_seq":881,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562796+0000","last_change":"2024-03-26T04:04:19.574250+0000","last_active":"2024-03-26T04:07:35.562796+0000","last_peered":"2024-03-26T04:07:35.562796+0000","last_clean":"2024-03-26T04:07:35.562796+0000","last_became_active":"2024-03-26T04:01:35.824185+0000","last_became_peered":"2024-03-26T04:01:35.824185+0000","last_unstale":"2024-03-26T04:07:35.562796+0000","last_undegraded":"2024-03-26T04:07:35.562796+0000","last_fullsized":"2024-03-26T04:07:35.562796+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.571532+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":724,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562216+0000","last_change":"2024-03-26T04:05:29.537174+0000","last_active":"2024-03-26T04:07:35.562216+0000","last_peered":"2024-03-26T04:07:35.562216+0000","last_clean":"2024-03-26T04:07:35.562216+0000","last_became_active":"2024-03-26T04:05:29.536647+0000","last_became_peered":"2024-03-26T04:05:29.536647+0000","last_unstale":"2024-03-26T04:07:35.562216+0000","last_undegraded":"2024-03-26T04:07:35.562216+0000","last_fullsized":"2024-03-26T04:07:35.562216+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.058039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.058039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"421'21270","reported_seq":21597,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833110+0000","last_change":"2024-03-26T04:06:09.371173+0000","last_active":"2024-03-26T04:07:37.833110+0000","last_peered":"2024-03-26T04:07:37.833110+0000","last_clean":"2024-03-26T04:07:37.833110+0000","last_became_active":"2024-03-26T04:03:47.646236+0000","last_became_peered":"2024-03-26T04:03:47.646236+0000","last_unstale":"2024-03-26T04:07:37.833110+0000","last_undegraded":"2024-03-26T04:07:37.833110+0000","last_fullsized":"2024-03-26T04:07:37.833110+0000","mapping_epoch":349,"log_start":"404'18100","ondisk_log_start":"404'18100","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'14177","last_scrub_stamp":"2024-03-26T04:06:09.370906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:06:09.370906+0000","objects_scrubbed":7160,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21270,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":469,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834907+0000","last_change":"2024-03-26T04:07:29.526841+0000","last_active":"2024-03-26T04:07:37.834907+0000","last_peered":"2024-03-26T04:07:37.834907+0000","last_clean":"2024-03-26T04:07:37.834907+0000","last_became_active":"2024-03-26T04:05:29.536837+0000","last_became_peered":"2024-03-26T04:05:29.536837+0000","last_unstale":"2024-03-26T04:07:37.834907+0000","last_undegraded":"2024-03-26T04:07:37.834907+0000","last_fullsized":"2024-03-26T04:07:37.834907+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:29.526591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.526591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:46.626775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":787,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.561464+0000","last_change":"2024-03-26T04:05:29.539255+0000","last_active":"2024-03-26T04:07:35.561464+0000","last_peered":"2024-03-26T04:07:35.561464+0000","last_clean":"2024-03-26T04:07:35.561464+0000","last_became_active":"2024-03-26T04:05:29.538900+0000","last_became_peered":"2024-03-26T04:05:29.538900+0000","last_unstale":"2024-03-26T04:07:35.561464+0000","last_undegraded":"2024-03-26T04:07:35.561464+0000","last_fullsized":"2024-03-26T04:07:35.561464+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:10.378037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:10.378037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":431,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834653+0000","last_change":"2024-03-26T04:05:29.544209+0000","last_active":"2024-03-26T04:07:37.834653+0000","last_peered":"2024-03-26T04:07:37.834653+0000","last_clean":"2024-03-26T04:07:37.834653+0000","last_became_active":"2024-03-26T04:05:29.542761+0000","last_became_peered":"2024-03-26T04:05:29.542761+0000","last_unstale":"2024-03-26T04:07:37.834653+0000","last_undegraded":"2024-03-26T04:07:37.834653+0000","last_fullsized":"2024-03-26T04:07:37.834653+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:03.918739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:03.918739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":843,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585843+0000","last_change":"2024-03-26T04:04:12.249958+0000","last_active":"2024-03-26T04:07:35.585843+0000","last_peered":"2024-03-26T04:07:35.585843+0000","last_clean":"2024-03-26T04:07:35.585843+0000","last_became_active":"2024-03-26T04:01:35.819144+0000","last_became_peered":"2024-03-26T04:01:35.819144+0000","last_unstale":"2024-03-26T04:07:35.585843+0000","last_undegraded":"2024-03-26T04:07:35.585843+0000","last_fullsized":"2024-03-26T04:07:35.585843+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:04:12.249693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:12.249693+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":1051,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562505+0000","last_change":"2024-03-26T04:02:52.536740+0000","last_active":"2024-03-26T04:07:35.562505+0000","last_peered":"2024-03-26T04:07:35.562505+0000","last_clean":"2024-03-26T04:07:35.562505+0000","last_became_active":"2024-03-26T04:02:52.495925+0000","last_became_peered":"2024-03-26T04:02:52.495925+0000","last_unstale":"2024-03-26T04:07:35.562505+0000","last_undegraded":"2024-03-26T04:07:35.562505+0000","last_fullsized":"2024-03-26T04:07:35.562505+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-03-26T04:02:46.076593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:46.076593+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"421'21498","reported_seq":37029,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832197+0000","last_change":"2024-03-26T04:06:33.714049+0000","last_active":"2024-03-26T04:07:37.832197+0000","last_peered":"2024-03-26T04:07:37.832197+0000","last_clean":"2024-03-26T04:07:37.832197+0000","last_became_active":"2024-03-26T04:06:33.689489+0000","last_became_peered":"2024-03-26T04:06:33.689489+0000","last_unstale":"2024-03-26T04:07:37.832197+0000","last_undegraded":"2024-03-26T04:07:37.832197+0000","last_fullsized":"2024-03-26T04:07:37.832197+0000","mapping_epoch":397,"log_start":"400'17800","ondisk_log_start":"400'17800","created":349,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3698,"log_dups_size":0,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21498,"num_write_kb":687936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7684,"num_bytes_recovered":503578624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":977,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832986+0000","last_change":"2024-03-26T04:07:26.838873+0000","last_active":"2024-03-26T04:07:37.832986+0000","last_peered":"2024-03-26T04:07:37.832986+0000","last_clean":"2024-03-26T04:07:37.832986+0000","last_became_active":"2024-03-26T04:02:55.533229+0000","last_became_peered":"2024-03-26T04:02:55.533229+0000","last_unstale":"2024-03-26T04:07:37.832986+0000","last_undegraded":"2024-03-26T04:07:37.832986+0000","last_fullsized":"2024-03-26T04:07:37.832986+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:26.838741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.838741+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.279852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":789,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832936+0000","last_change":"2024-03-26T04:05:41.943296+0000","last_active":"2024-03-26T04:07:37.832936+0000","last_peered":"2024-03-26T04:07:37.832936+0000","last_clean":"2024-03-26T04:07:37.832936+0000","last_became_active":"2024-03-26T04:05:29.542416+0000","last_became_peered":"2024-03-26T04:05:29.542416+0000","last_unstale":"2024-03-26T04:07:37.832936+0000","last_undegraded":"2024-03-26T04:07:37.832936+0000","last_fullsized":"2024-03-26T04:07:37.832936+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:41.943187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:41.943187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":779,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563427+0000","last_change":"2024-03-26T04:05:07.862785+0000","last_active":"2024-03-26T04:07:35.563427+0000","last_peered":"2024-03-26T04:07:35.563427+0000","last_clean":"2024-03-26T04:07:35.563427+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:07:35.563427+0000","last_undegraded":"2024-03-26T04:07:35.563427+0000","last_fullsized":"2024-03-26T04:07:35.563427+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:05:07.862659+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.862659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":879,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701258+0000","last_change":"2024-03-26T04:04:16.357551+0000","last_active":"2024-03-26T04:07:37.701258+0000","last_peered":"2024-03-26T04:07:37.701258+0000","last_clean":"2024-03-26T04:07:37.701258+0000","last_became_active":"2024-03-26T04:01:35.799996+0000","last_became_peered":"2024-03-26T04:01:35.799996+0000","last_unstale":"2024-03-26T04:07:37.701258+0000","last_undegraded":"2024-03-26T04:07:37.701258+0000","last_fullsized":"2024-03-26T04:07:37.701258+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:04:16.357311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:16.357311+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":485,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585342+0000","last_change":"2024-03-26T04:03:51.382658+0000","last_active":"2024-03-26T04:07:35.585342+0000","last_peered":"2024-03-26T04:07:35.585342+0000","last_clean":"2024-03-26T04:07:35.585342+0000","last_became_active":"2024-03-26T04:02:52.515390+0000","last_became_peered":"2024-03-26T04:02:52.515390+0000","last_unstale":"2024-03-26T04:07:35.585342+0000","last_undegraded":"2024-03-26T04:07:35.585342+0000","last_fullsized":"2024-03-26T04:07:35.585342+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:51.382563+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:03:51.382563+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"421'21472","reported_seq":28780,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702169+0000","last_change":"2024-03-26T04:07:04.052128+0000","last_active":"2024-03-26T04:07:37.702169+0000","last_peered":"2024-03-26T04:07:37.702169+0000","last_clean":"2024-03-26T04:07:37.702169+0000","last_became_active":"2024-03-26T04:07:04.035244+0000","last_became_peered":"2024-03-26T04:07:04.035244+0000","last_unstale":"2024-03-26T04:07:37.702169+0000","last_undegraded":"2024-03-26T04:07:37.702169+0000","last_fullsized":"2024-03-26T04:07:37.702169+0000","mapping_epoch":406,"log_start":"404'18500","ondisk_log_start":"404'18500","created":349,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":3000,"ondisk_log_size":2972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21472,"num_write_kb":687104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":233897984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":627,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.474278+0000","last_change":"2024-03-26T04:07:39.474303+0000","last_active":"2024-03-26T04:07:39.474278+0000","last_peered":"2024-03-26T04:07:39.474278+0000","last_clean":"2024-03-26T04:07:39.474278+0000","last_became_active":"2024-03-26T04:05:29.521761+0000","last_became_peered":"2024-03-26T04:05:29.521761+0000","last_unstale":"2024-03-26T04:07:39.474278+0000","last_undegraded":"2024-03-26T04:07:39.474278+0000","last_fullsized":"2024-03-26T04:07:39.474278+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:39.474219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.474219+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:43.579453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":779,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701129+0000","last_change":"2024-03-26T04:07:30.569585+0000","last_active":"2024-03-26T04:07:37.701129+0000","last_peered":"2024-03-26T04:07:37.701129+0000","last_clean":"2024-03-26T04:07:37.701129+0000","last_became_active":"2024-03-26T04:05:29.532854+0000","last_became_peered":"2024-03-26T04:05:29.532854+0000","last_unstale":"2024-03-26T04:07:37.701129+0000","last_undegraded":"2024-03-26T04:07:37.701129+0000","last_fullsized":"2024-03-26T04:07:37.701129+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.569498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.569498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.566912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":282,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.563208+0000","last_change":"2024-03-26T04:03:06.122009+0000","last_active":"2024-03-26T04:07:35.563208+0000","last_peered":"2024-03-26T04:07:35.563208+0000","last_clean":"2024-03-26T04:07:35.563208+0000","last_became_active":"2024-03-26T04:02:52.496389+0000","last_became_peered":"2024-03-26T04:02:52.496389+0000","last_unstale":"2024-03-26T04:07:35.563208+0000","last_undegraded":"2024-03-26T04:07:35.563208+0000","last_fullsized":"2024-03-26T04:07:35.563208+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:06.121917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:06.121917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":355,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832626+0000","last_change":"2024-03-26T04:07:35.968314+0000","last_active":"2024-03-26T04:07:37.832626+0000","last_peered":"2024-03-26T04:07:37.832626+0000","last_clean":"2024-03-26T04:07:37.832626+0000","last_became_active":"2024-03-26T04:02:52.511833+0000","last_became_peered":"2024-03-26T04:02:52.511833+0000","last_unstale":"2024-03-26T04:07:37.832626+0000","last_undegraded":"2024-03-26T04:07:37.832626+0000","last_fullsized":"2024-03-26T04:07:37.832626+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:07:35.968179+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:07:35.968179+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:00.483729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":782,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701055+0000","last_change":"2024-03-26T04:04:06.297839+0000","last_active":"2024-03-26T04:07:37.701055+0000","last_peered":"2024-03-26T04:07:37.701055+0000","last_clean":"2024-03-26T04:07:37.701055+0000","last_became_active":"2024-03-26T04:02:52.498870+0000","last_became_peered":"2024-03-26T04:02:52.498870+0000","last_unstale":"2024-03-26T04:07:37.701055+0000","last_undegraded":"2024-03-26T04:07:37.701055+0000","last_fullsized":"2024-03-26T04:07:37.701055+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.297701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"421'21798","reported_seq":21955,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700718+0000","last_change":"2024-03-26T04:03:47.644772+0000","last_active":"2024-03-26T04:07:37.700718+0000","last_peered":"2024-03-26T04:07:37.700718+0000","last_clean":"2024-03-26T04:07:37.700718+0000","last_became_active":"2024-03-26T04:03:47.644103+0000","last_became_peered":"2024-03-26T04:03:47.644103+0000","last_unstale":"2024-03-26T04:07:37.700718+0000","last_undegraded":"2024-03-26T04:07:37.700718+0000","last_fullsized":"2024-03-26T04:07:37.700718+0000","mapping_epoch":349,"log_start":"404'18800","ondisk_log_start":"404'18800","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":3000,"ondisk_log_size":2998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21798,"num_write_kb":697536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":464,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833151+0000","last_change":"2024-03-26T04:07:22.530369+0000","last_active":"2024-03-26T04:07:37.833151+0000","last_peered":"2024-03-26T04:07:37.833151+0000","last_clean":"2024-03-26T04:07:37.833151+0000","last_became_active":"2024-03-26T04:05:29.524691+0000","last_became_peered":"2024-03-26T04:05:29.524691+0000","last_unstale":"2024-03-26T04:07:37.833151+0000","last_undegraded":"2024-03-26T04:07:37.833151+0000","last_fullsized":"2024-03-26T04:07:37.833151+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:22.530143+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:07:22.530143+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:22.608960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":453,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585122+0000","last_change":"2024-03-26T04:05:29.532951+0000","last_active":"2024-03-26T04:07:35.585122+0000","last_peered":"2024-03-26T04:07:35.585122+0000","last_clean":"2024-03-26T04:07:35.585122+0000","last_became_active":"2024-03-26T04:05:29.532692+0000","last_became_peered":"2024-03-26T04:05:29.532692+0000","last_unstale":"2024-03-26T04:07:35.585122+0000","last_undegraded":"2024-03-26T04:07:35.585122+0000","last_fullsized":"2024-03-26T04:07:35.585122+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:25.365447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.365447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"379'13","reported_seq":1024,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702987+0000","last_change":"2024-03-26T04:05:29.523046+0000","last_active":"2024-03-26T04:07:37.702987+0000","last_peered":"2024-03-26T04:07:37.702987+0000","last_clean":"2024-03-26T04:07:37.702987+0000","last_became_active":"2024-03-26T04:05:29.521835+0000","last_became_peered":"2024-03-26T04:05:29.521835+0000","last_unstale":"2024-03-26T04:07:37.702987+0000","last_undegraded":"2024-03-26T04:07:37.702987+0000","last_fullsized":"2024-03-26T04:07:37.702987+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:22.364212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.364212+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":354,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.584301+0000","last_change":"2024-03-26T04:03:50.413181+0000","last_active":"2024-03-26T04:07:35.584301+0000","last_peered":"2024-03-26T04:07:35.584301+0000","last_clean":"2024-03-26T04:07:35.584301+0000","last_became_active":"2024-03-26T04:02:52.511800+0000","last_became_peered":"2024-03-26T04:02:52.511800+0000","last_unstale":"2024-03-26T04:07:35.584301+0000","last_undegraded":"2024-03-26T04:07:35.584301+0000","last_fullsized":"2024-03-26T04:07:35.584301+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:50.413099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:03:50.413099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":880,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.585319+0000","last_change":"2024-03-26T04:04:06.282172+0000","last_active":"2024-03-26T04:07:35.585319+0000","last_peered":"2024-03-26T04:07:35.585319+0000","last_clean":"2024-03-26T04:07:35.585319+0000","last_became_active":"2024-03-26T04:02:52.513721+0000","last_became_peered":"2024-03-26T04:02:52.513721+0000","last_unstale":"2024-03-26T04:07:35.585319+0000","last_undegraded":"2024-03-26T04:07:35.585319+0000","last_fullsized":"2024-03-26T04:07:35.585319+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:04:06.281970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.281970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"421'21409","reported_seq":21660,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.061120+0000","last_change":"2024-03-26T04:05:07.668084+0000","last_active":"2024-03-26T04:07:36.061120+0000","last_peered":"2024-03-26T04:07:36.061120+0000","last_clean":"2024-03-26T04:07:36.061120+0000","last_became_active":"2024-03-26T04:03:47.643599+0000","last_became_peered":"2024-03-26T04:03:47.643599+0000","last_unstale":"2024-03-26T04:07:36.061120+0000","last_undegraded":"2024-03-26T04:07:36.061120+0000","last_fullsized":"2024-03-26T04:07:36.061120+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'8783","last_scrub_stamp":"2024-03-26T04:05:07.667791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.667791+0000","objects_scrubbed":8667,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196608,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21409,"num_write_kb":685184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":398,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833017+0000","last_change":"2024-03-26T04:06:00.896785+0000","last_active":"2024-03-26T04:07:37.833017+0000","last_peered":"2024-03-26T04:07:37.833017+0000","last_clean":"2024-03-26T04:07:37.833017+0000","last_became_active":"2024-03-26T04:02:52.506309+0000","last_became_peered":"2024-03-26T04:02:52.506309+0000","last_unstale":"2024-03-26T04:07:37.833017+0000","last_undegraded":"2024-03-26T04:07:37.833017+0000","last_fullsized":"2024-03-26T04:07:37.833017+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:00.896583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:00.896583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":505,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702939+0000","last_change":"2024-03-26T04:07:34.478534+0000","last_active":"2024-03-26T04:07:37.702939+0000","last_peered":"2024-03-26T04:07:37.702939+0000","last_clean":"2024-03-26T04:07:37.702939+0000","last_became_active":"2024-03-26T04:05:29.520457+0000","last_became_peered":"2024-03-26T04:05:29.520457+0000","last_unstale":"2024-03-26T04:07:37.702939+0000","last_undegraded":"2024-03-26T04:07:37.702939+0000","last_fullsized":"2024-03-26T04:07:37.702939+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.478427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.478427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:01.934278+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":772,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701104+0000","last_change":"2024-03-26T04:05:29.537130+0000","last_active":"2024-03-26T04:07:37.701104+0000","last_peered":"2024-03-26T04:07:37.701104+0000","last_clean":"2024-03-26T04:07:37.701104+0000","last_became_active":"2024-03-26T04:05:29.536797+0000","last_became_peered":"2024-03-26T04:05:29.536797+0000","last_unstale":"2024-03-26T04:07:37.701104+0000","last_undegraded":"2024-03-26T04:07:37.701104+0000","last_fullsized":"2024-03-26T04:07:37.701104+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:17.365625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:00:17.365625+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":907,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702545+0000","last_change":"2024-03-26T04:05:06.619430+0000","last_active":"2024-03-26T04:07:37.702545+0000","last_peered":"2024-03-26T04:07:37.702545+0000","last_clean":"2024-03-26T04:07:37.702545+0000","last_became_active":"2024-03-26T04:02:52.520710+0000","last_became_peered":"2024-03-26T04:02:52.520710+0000","last_unstale":"2024-03-26T04:07:37.702545+0000","last_undegraded":"2024-03-26T04:07:37.702545+0000","last_fullsized":"2024-03-26T04:07:37.702545+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T04:05:06.619261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:05:06.619261+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"421'21486","reported_seq":35515,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832874+0000","last_change":"2024-03-26T04:06:45.558381+0000","last_active":"2024-03-26T04:07:37.832874+0000","last_peered":"2024-03-26T04:07:37.832874+0000","last_clean":"2024-03-26T04:07:37.832874+0000","last_became_active":"2024-03-26T04:06:45.534824+0000","last_became_peered":"2024-03-26T04:06:45.534824+0000","last_unstale":"2024-03-26T04:07:37.832874+0000","last_undegraded":"2024-03-26T04:07:37.832874+0000","last_fullsized":"2024-03-26T04:07:37.832874+0000","mapping_epoch":399,"log_start":"400'17800","ondisk_log_start":"400'17800","created":349,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21486,"num_write_kb":687552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6933,"num_bytes_recovered":454361088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":492,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832395+0000","last_change":"2024-03-26T04:05:29.533929+0000","last_active":"2024-03-26T04:07:37.832395+0000","last_peered":"2024-03-26T04:07:37.832395+0000","last_clean":"2024-03-26T04:07:37.832395+0000","last_became_active":"2024-03-26T04:05:29.533470+0000","last_became_peered":"2024-03-26T04:05:29.533470+0000","last_unstale":"2024-03-26T04:07:37.832395+0000","last_undegraded":"2024-03-26T04:07:37.832395+0000","last_fullsized":"2024-03-26T04:07:37.832395+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:39.967821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:39.967821+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":702,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834586+0000","last_change":"2024-03-26T04:05:29.522502+0000","last_active":"2024-03-26T04:07:37.834586+0000","last_peered":"2024-03-26T04:07:37.834586+0000","last_clean":"2024-03-26T04:07:37.834586+0000","last_became_active":"2024-03-26T04:05:29.521089+0000","last_became_peered":"2024-03-26T04:05:29.521089+0000","last_unstale":"2024-03-26T04:07:37.834586+0000","last_undegraded":"2024-03-26T04:07:37.834586+0000","last_fullsized":"2024-03-26T04:07:37.834586+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_clean_scrub_stamp":"2024-03-26T04:02:49.166761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":767,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.586462+0000","last_change":"2024-03-26T04:04:49.184140+0000","last_active":"2024-03-26T04:07:35.586462+0000","last_peered":"2024-03-26T04:07:35.586462+0000","last_clean":"2024-03-26T04:07:35.586462+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:07:35.586462+0000","last_undegraded":"2024-03-26T04:07:35.586462+0000","last_fullsized":"2024-03-26T04:07:35.586462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:49.184018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:04:49.184018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":736,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701199+0000","last_change":"2024-03-26T04:05:08.666190+0000","last_active":"2024-03-26T04:07:37.701199+0000","last_peered":"2024-03-26T04:07:37.701199+0000","last_clean":"2024-03-26T04:07:37.701199+0000","last_became_active":"2024-03-26T04:02:52.505213+0000","last_became_peered":"2024-03-26T04:02:52.505213+0000","last_unstale":"2024-03-26T04:07:37.701199+0000","last_undegraded":"2024-03-26T04:07:37.701199+0000","last_fullsized":"2024-03-26T04:07:37.701199+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:08.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:08.666106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":825,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702937+0000","last_change":"2024-03-26T04:07:25.580036+0000","last_active":"2024-03-26T04:07:37.702937+0000","last_peered":"2024-03-26T04:07:37.702937+0000","last_clean":"2024-03-26T04:07:37.702937+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:07:37.702937+0000","last_undegraded":"2024-03-26T04:07:37.702937+0000","last_fullsized":"2024-03-26T04:07:37.702937+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:07:25.579888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.579888+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.986075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"421'21441","reported_seq":21686,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:36.061521+0000","last_change":"2024-03-26T04:05:10.785305+0000","last_active":"2024-03-26T04:07:36.061521+0000","last_peered":"2024-03-26T04:07:36.061521+0000","last_clean":"2024-03-26T04:07:36.061521+0000","last_became_active":"2024-03-26T04:03:47.643488+0000","last_became_peered":"2024-03-26T04:03:47.643488+0000","last_unstale":"2024-03-26T04:07:36.061521+0000","last_undegraded":"2024-03-26T04:07:36.061521+0000","last_fullsized":"2024-03-26T04:07:36.061521+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":8599,"log_size":3141,"log_dups_size":0,"ondisk_log_size":3141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65536,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21441,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1085,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832333+0000","last_change":"2024-03-26T04:05:38.828011+0000","last_active":"2024-03-26T04:07:37.832333+0000","last_peered":"2024-03-26T04:07:37.832333+0000","last_clean":"2024-03-26T04:07:37.832333+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:07:37.832333+0000","last_undegraded":"2024-03-26T04:07:37.832333+0000","last_fullsized":"2024-03-26T04:07:37.832333+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:05:38.827871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:05:38.827871+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":768,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702970+0000","last_change":"2024-03-26T04:07:27.607414+0000","last_active":"2024-03-26T04:07:37.702970+0000","last_peered":"2024-03-26T04:07:37.702970+0000","last_clean":"2024-03-26T04:07:37.702970+0000","last_became_active":"2024-03-26T04:05:29.520602+0000","last_became_peered":"2024-03-26T04:05:29.520602+0000","last_unstale":"2024-03-26T04:07:37.702970+0000","last_undegraded":"2024-03-26T04:07:37.702970+0000","last_fullsized":"2024-03-26T04:07:37.702970+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:27.607316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:07:27.607316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:35.557513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"379'13","reported_seq":932,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834806+0000","last_change":"2024-03-26T04:05:29.544439+0000","last_active":"2024-03-26T04:07:37.834806+0000","last_peered":"2024-03-26T04:07:37.834806+0000","last_clean":"2024-03-26T04:07:37.834806+0000","last_became_active":"2024-03-26T04:05:29.544028+0000","last_became_peered":"2024-03-26T04:05:29.544028+0000","last_unstale":"2024-03-26T04:07:37.834806+0000","last_undegraded":"2024-03-26T04:07:37.834806+0000","last_fullsized":"2024-03-26T04:07:37.834806+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:01:34.236967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.236967+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":354,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-03-26T04:07:35.562789+0000","last_change":"2024-03-26T04:04:06.376407+0000","last_active":"2024-03-26T04:07:35.562789+0000","last_peered":"2024-03-26T04:07:35.562789+0000","last_clean":"2024-03-26T04:07:35.562789+0000","last_became_active":"2024-03-26T04:02:52.491417+0000","last_became_peered":"2024-03-26T04:02:52.491417+0000","last_unstale":"2024-03-26T04:07:35.562789+0000","last_undegraded":"2024-03-26T04:07:35.562789+0000","last_fullsized":"2024-03-26T04:07:35.562789+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.376284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.376284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":869,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702014+0000","last_change":"2024-03-26T04:04:18.408633+0000","last_active":"2024-03-26T04:07:37.702014+0000","last_peered":"2024-03-26T04:07:37.702014+0000","last_clean":"2024-03-26T04:07:37.702014+0000","last_became_active":"2024-03-26T04:02:52.504477+0000","last_became_peered":"2024-03-26T04:02:52.504477+0000","last_unstale":"2024-03-26T04:07:37.702014+0000","last_undegraded":"2024-03-26T04:07:37.702014+0000","last_fullsized":"2024-03-26T04:07:37.702014+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:04:18.408372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.408372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":393216,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":344180,"num_write_kb":11013889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37712,"num_bytes_recovered":2471297024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":720896,"data_stored":720896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51979,"ondisk_log_size":51979,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1127,"num_read_kb":1122,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":7718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":15688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":942,"ondisk_log_size":942,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070695,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894404,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92850748,"statfs":{"total":95995035648,"available":95079165952,"internally_reserved":0,"allocated":2547712,"data_stored":1534036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":913308106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365037,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575109,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":859328,"kb_used_data":21556,"kb_used_omap":1,"kb_used_meta":837758,"kb_avail":92885824,"statfs":{"total":95995035648,"available":95115083776,"internally_reserved":0,"allocated":22073344,"data_stored":21050385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":857864645},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738634,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":958828,"kb_used_data":43296,"kb_used_omap":1,"kb_used_meta":915518,"kb_avail":92786324,"statfs":{"total":95995035648,"available":95013195776,"internally_reserved":0,"allocated":44335104,"data_stored":43332485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":937490885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637915,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318156,"kb_used_data":2372,"kb_used_omap":6,"kb_used_meta":315769,"kb_avail":93426996,"statfs":{"total":95995035648,"available":95669243904,"internally_reserved":0,"allocated":2428928,"data_stored":1459778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":323347749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953718,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954340,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92790812,"statfs":{"total":95995035648,"available":95017791488,"internally_reserved":0,"allocated":2772992,"data_stored":1794327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":720896,"data_stored":720896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:07:43.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:07:43.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:43.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:43.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:43.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:43.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:43.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:43.994 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:43 smithi086 ceph-mon[32965]: 6.12 deep-scrub starts 2024-03-26T04:07:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:43 smithi086 ceph-mon[32965]: 6.12 deep-scrub ok 2024-03-26T04:07:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:43 smithi086 ceph-mon[32965]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:43 smithi086 ceph-mon[32965]: pgmap v900: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 705 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T04:07:44.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[37501]: 6.12 deep-scrub starts 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[37501]: 6.12 deep-scrub ok 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[37501]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[37501]: pgmap v900: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 705 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[29446]: 6.12 deep-scrub starts 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[29446]: 6.12 deep-scrub ok 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[29446]: from='client.15732 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:44.142 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:43 smithi006 ceph-mon[29446]: pgmap v900: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 705 op/s; 1.5 MiB/s, 24 objects/s recovering 2024-03-26T04:07:44.146 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:44.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:44.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:44.396 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:44.488 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:44.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:44.522 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:44.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:44.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.648 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:44.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:44.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:44.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:44.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:44.901 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:44.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:44.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:45.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:45.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:45.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:45.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:45.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:45.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:45.347 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:45.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:45.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T04:07:45.385 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:45.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:45.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:45.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:45.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:45.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:45.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:45 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T04:07:45.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:45 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T04:07:45.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:45 smithi086 ceph-mon[32965]: pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 461 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T04:07:45.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T04:07:45.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T04:07:45.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[37501]: pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 461 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T04:07:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T04:07:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T04:07:45.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:45 smithi006 ceph-mon[29446]: pgmap v901: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 461 op/s; 1.3 MiB/s, 21 objects/s recovering 2024-03-26T04:07:45.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:45.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:45.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:45.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:45.793 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:45.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:45.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:45.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:45.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:46.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:46.048 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:46.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:46.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:46.197 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:07:46.217 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:07:46.236 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:07:46.256 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:07:46.276 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:07:46.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:46.296 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:07:46.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:46.318 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:07:46.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:07:46.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:07:46.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:46.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:46.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:46.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:46.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:46.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:46.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:46.818 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T04:07:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T04:07:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T04:07:46.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T04:07:46.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 4.16 deep-scrub starts 2024-03-26T04:07:46.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: 4.16 deep-scrub ok 2024-03-26T04:07:46.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/655119539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 4.16 deep-scrub starts 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: 4.16 deep-scrub ok 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/655119539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:46.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 4.16 deep-scrub starts 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: 4.16 deep-scrub ok 2024-03-26T04:07:46.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/655119539' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:07:46.919 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:46.945 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:46.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:47.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:07:47.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:47.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:47.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:47.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:47.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:47.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:47.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:47.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:47.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:47.433 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:47.458 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:47.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:47.559 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:47.583 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:47.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:47.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:47 smithi006 ceph-mon[37501]: pgmap v902: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 145 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:47.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:47 smithi006 ceph-mon[29446]: pgmap v902: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 145 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:47.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:47.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:47.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:47.813 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:47.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:47.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:47 smithi086 ceph-mon[32965]: pgmap v902: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 145 op/s; 1.2 MiB/s, 19 objects/s recovering 2024-03-26T04:07:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:47.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:47.965 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:47.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:47.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:48.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:48.093 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:48.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:48.220 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:48.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:48.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:48.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:48.476 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:48.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:48.604 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:48.706 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:48.732 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:48.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:48 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T04:07:48.743 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:48 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T04:07:48.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:48 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T04:07:48.743 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:48 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T04:07:48.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:48 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T04:07:48.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:48 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T04:07:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:48.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:48.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:48.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:48.991 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:49.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:49.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:49.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:49.246 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:49.348 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:49.374 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:49.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:49.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:49.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:49.634 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:49.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:49.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:49.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:49 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T04:07:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:49 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T04:07:49.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:49 smithi086 ceph-mon[32965]: pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 131 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:49.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T04:07:49.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T04:07:49.844 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[29446]: pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 131 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:49.844 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T04:07:49.844 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T04:07:49.844 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:49 smithi006 ceph-mon[37501]: pgmap v903: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 923 B/s rd, 0 B/s wr, 131 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:07:49.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:49.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:49.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:49.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:50.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:50.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:50.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.152 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:50.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.281 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:50.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:50.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:50.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:50.540 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:50.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:50.669 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:50.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 1.9 deep-scrub starts 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 1.9 deep-scrub ok 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 4.6 deep-scrub starts 2024-03-26T04:07:50.824 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[37501]: 4.6 deep-scrub ok 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 1.9 deep-scrub starts 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 1.9 deep-scrub ok 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 4.6 deep-scrub starts 2024-03-26T04:07:50.825 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:50 smithi006 ceph-mon[29446]: 4.6 deep-scrub ok 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 1.9 deep-scrub starts 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 1.9 deep-scrub ok 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 4.6 deep-scrub starts 2024-03-26T04:07:50.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:50 smithi086 ceph-mon[32965]: 4.6 deep-scrub ok 2024-03-26T04:07:50.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:50.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:50.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:51.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:51.055 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:51.067 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:07:51.070 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:07:51.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:51.182 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:51.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:51.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:51.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:51.434 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:51.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:51.476 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:51.510 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:51.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:51.560 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:51.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:51.607 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_3' removed 2024-03-26T04:07:51.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:51.695 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:51.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:51.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:07:51.724 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/995265759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: pgmap v904: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[37501]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/995265759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: pgmap v904: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-26T04:07:51.725 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:51 smithi006 ceph-mon[29446]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:51.797 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:51.824 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:51.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:51.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:07:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:07:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/995265759' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:07:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: pgmap v904: 158 pgs: 158 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-26T04:07:51.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:51 smithi086 ceph-mon[32965]: from='client.25670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:07:51.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:51.949 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:51.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:52.051 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:52.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:52.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:52.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:52.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:52.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:52.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:52.450 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:52.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:52.552 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:52.576 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:52.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:52.702 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:52.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:52.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:52.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:52.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:52 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T04:07:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:52 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T04:07:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:52 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:07:52.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:52 smithi086 ceph-mon[32965]: osdmap e423: 8 total, 8 up, 5 in 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[37501]: osdmap e423: 8 total, 8 up, 5 in 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:07:52.848 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:52 smithi006 ceph-mon[29446]: osdmap e423: 8 total, 8 up, 5 in 2024-03-26T04:07:52.875 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-26T04:07:52.876 INFO:tasks.radosbench:Beginning radosbench... 2024-03-26T04:07:52.876 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-26T04:07:52.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-26T04:07:52.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:52.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:53.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:53.079 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:53.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:53.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:53.205 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:53.305 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:07:53.307 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:53.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:53.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:53.457 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:53.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:53.583 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:53.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:53.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:53.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:53.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:53.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:53.835 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:53.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:53.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:53 smithi086 ceph-mon[32965]: pgmap v906: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:07:53.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:53 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:53.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:53 smithi006 ceph-mon[37501]: pgmap v906: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:07:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:53 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:53 smithi006 ceph-mon[29446]: pgmap v906: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:07:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:53 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:07:53.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:53.961 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:53.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:53.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:54.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:54.087 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:54.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:54.213 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:54.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:54.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:54.339 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:54.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:54.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:54.464 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:54.511 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:07:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:07:54.538 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":904,"stamp":"2024-03-26T04:07:50.784950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2561,"num_read_kb":2281,"num_write":345100,"num_write_kb":11013934,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38173,"num_bytes_recovered":2471309329,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54469,"ondisk_log_size":54469,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3912644,"kb_used_data":12508,"kb_used_omap":12,"kb_used_meta":3900083,"kb_avail":464813116,"statfs":{"total":479975178240,"available":475968630784,"internally_reserved":0,"allocated":12808192,"data_stored":7798216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13250,"internal_metadata":3993685054},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-15794200,"num_objects":-242,"num_object_clones":0,"num_object_copies":-726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-242,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":242,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003607"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":773,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701088+0000","last_change":"2024-03-26T04:05:03.632715+0000","last_active":"2024-03-26T04:07:37.701088+0000","last_peered":"2024-03-26T04:07:37.701088+0000","last_clean":"2024-03-26T04:07:37.701088+0000","last_became_active":"2024-03-26T04:01:35.797821+0000","last_became_peered":"2024-03-26T04:01:35.797821+0000","last_unstale":"2024-03-26T04:07:37.701088+0000","last_undegraded":"2024-03-26T04:07:37.701088+0000","last_fullsized":"2024-03-26T04:07:37.701088+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:03.632384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:03.632384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":776,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833232+0000","last_change":"2024-03-26T04:05:29.541099+0000","last_active":"2024-03-26T04:07:37.833232+0000","last_peered":"2024-03-26T04:07:37.833232+0000","last_clean":"2024-03-26T04:07:37.833232+0000","last_became_active":"2024-03-26T04:05:29.540706+0000","last_became_peered":"2024-03-26T04:05:29.540706+0000","last_unstale":"2024-03-26T04:07:37.833232+0000","last_undegraded":"2024-03-26T04:07:37.833232+0000","last_fullsized":"2024-03-26T04:07:37.833232+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:33.560099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:00:33.560099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":796,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698303+0000","last_change":"2024-03-26T04:05:29.539024+0000","last_active":"2024-03-26T04:07:37.698303+0000","last_peered":"2024-03-26T04:07:37.698303+0000","last_clean":"2024-03-26T04:07:37.698303+0000","last_became_active":"2024-03-26T04:05:29.538364+0000","last_became_peered":"2024-03-26T04:05:29.538364+0000","last_unstale":"2024-03-26T04:07:37.698303+0000","last_undegraded":"2024-03-26T04:07:37.698303+0000","last_fullsized":"2024-03-26T04:07:37.698303+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:52.117692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:52.117692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":876,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700839+0000","last_change":"2024-03-26T04:05:29.520520+0000","last_active":"2024-03-26T04:07:37.700839+0000","last_peered":"2024-03-26T04:07:37.700839+0000","last_clean":"2024-03-26T04:07:37.700839+0000","last_became_active":"2024-03-26T04:05:29.520084+0000","last_became_peered":"2024-03-26T04:05:29.520084+0000","last_unstale":"2024-03-26T04:07:37.700839+0000","last_undegraded":"2024-03-26T04:07:37.700839+0000","last_fullsized":"2024-03-26T04:07:37.700839+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:01:20.107206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.107206+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":435,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832594+0000","last_change":"2024-03-26T04:06:29.930571+0000","last_active":"2024-03-26T04:07:37.832594+0000","last_peered":"2024-03-26T04:07:37.832594+0000","last_clean":"2024-03-26T04:07:37.832594+0000","last_became_active":"2024-03-26T04:02:52.504637+0000","last_became_peered":"2024-03-26T04:02:52.504637+0000","last_unstale":"2024-03-26T04:07:37.832594+0000","last_undegraded":"2024-03-26T04:07:37.832594+0000","last_fullsized":"2024-03-26T04:07:37.832594+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:29.930462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:29.930462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:07:49.086851+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":769,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700751+0000","last_change":"2024-03-26T04:05:29.519865+0000","last_active":"2024-03-26T04:07:37.700751+0000","last_peered":"2024-03-26T04:07:37.700751+0000","last_clean":"2024-03-26T04:07:37.700751+0000","last_became_active":"2024-03-26T04:05:29.519486+0000","last_became_peered":"2024-03-26T04:05:29.519486+0000","last_unstale":"2024-03-26T04:07:37.700751+0000","last_undegraded":"2024-03-26T04:07:37.700751+0000","last_fullsized":"2024-03-26T04:07:37.700751+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.069804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.069804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":335,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835357+0000","last_change":"2024-03-26T04:07:30.526662+0000","last_active":"2024-03-26T04:07:37.835357+0000","last_peered":"2024-03-26T04:07:37.835357+0000","last_clean":"2024-03-26T04:07:37.835357+0000","last_became_active":"2024-03-26T04:05:29.536149+0000","last_became_peered":"2024-03-26T04:05:29.536149+0000","last_unstale":"2024-03-26T04:07:37.835357+0000","last_undegraded":"2024-03-26T04:07:37.835357+0000","last_fullsized":"2024-03-26T04:07:37.835357+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:44.136302+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":245,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.566623+0000","last_change":"2024-03-26T04:07:44.566663+0000","last_active":"2024-03-26T04:07:44.566623+0000","last_peered":"2024-03-26T04:07:44.566623+0000","last_clean":"2024-03-26T04:07:44.566623+0000","last_became_active":"2024-03-26T04:05:29.533552+0000","last_became_peered":"2024-03-26T04:05:29.533552+0000","last_unstale":"2024-03-26T04:07:44.566623+0000","last_undegraded":"2024-03-26T04:07:44.566623+0000","last_fullsized":"2024-03-26T04:07:44.566623+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:07:44.566551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:44.566551+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:09.048408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":758,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696889+0000","last_change":"2024-03-26T04:04:21.537611+0000","last_active":"2024-03-26T04:07:37.696889+0000","last_peered":"2024-03-26T04:07:37.696889+0000","last_clean":"2024-03-26T04:07:37.696889+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:07:37.696889+0000","last_undegraded":"2024-03-26T04:07:37.696889+0000","last_fullsized":"2024-03-26T04:07:37.696889+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:21.537431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:21.537431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":519,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697760+0000","last_change":"2024-03-26T04:03:53.342268+0000","last_active":"2024-03-26T04:07:37.697760+0000","last_peered":"2024-03-26T04:07:37.697760+0000","last_clean":"2024-03-26T04:07:37.697760+0000","last_became_active":"2024-03-26T04:01:35.802827+0000","last_became_peered":"2024-03-26T04:01:35.802827+0000","last_unstale":"2024-03-26T04:07:37.697760+0000","last_undegraded":"2024-03-26T04:07:37.697760+0000","last_fullsized":"2024-03-26T04:07:37.697760+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:53.342172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.342172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":475,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702535+0000","last_change":"2024-03-26T04:04:55.527963+0000","last_active":"2024-03-26T04:07:37.702535+0000","last_peered":"2024-03-26T04:07:37.702535+0000","last_clean":"2024-03-26T04:07:37.702535+0000","last_became_active":"2024-03-26T04:02:52.503960+0000","last_became_peered":"2024-03-26T04:02:52.503960+0000","last_unstale":"2024-03-26T04:07:37.702535+0000","last_undegraded":"2024-03-26T04:07:37.702535+0000","last_fullsized":"2024-03-26T04:07:37.702535+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:55.527836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.527836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":885,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701239+0000","last_change":"2024-03-26T04:07:26.622146+0000","last_active":"2024-03-26T04:07:37.701239+0000","last_peered":"2024-03-26T04:07:37.701239+0000","last_clean":"2024-03-26T04:07:37.701239+0000","last_became_active":"2024-03-26T04:05:29.521715+0000","last_became_peered":"2024-03-26T04:05:29.521715+0000","last_unstale":"2024-03-26T04:07:37.701239+0000","last_undegraded":"2024-03-26T04:07:37.701239+0000","last_fullsized":"2024-03-26T04:07:37.701239+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T04:07:26.622024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.622024+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:36.899154+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":578,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:38.459470+0000","last_change":"2024-03-26T04:07:38.459491+0000","last_active":"2024-03-26T04:07:38.459470+0000","last_peered":"2024-03-26T04:07:38.459470+0000","last_clean":"2024-03-26T04:07:38.459470+0000","last_became_active":"2024-03-26T04:05:29.532847+0000","last_became_peered":"2024-03-26T04:05:29.532847+0000","last_unstale":"2024-03-26T04:07:38.459470+0000","last_undegraded":"2024-03-26T04:07:38.459470+0000","last_fullsized":"2024-03-26T04:07:38.459470+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.459399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.459399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:54.713401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":792,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832771+0000","last_change":"2024-03-26T04:07:24.766554+0000","last_active":"2024-03-26T04:07:37.832771+0000","last_peered":"2024-03-26T04:07:37.832771+0000","last_clean":"2024-03-26T04:07:37.832771+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:07:37.832771+0000","last_undegraded":"2024-03-26T04:07:37.832771+0000","last_fullsized":"2024-03-26T04:07:37.832771+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.766403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.766403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:28.353556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":494,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701776+0000","last_change":"2024-03-26T04:04:01.292305+0000","last_active":"2024-03-26T04:07:37.701776+0000","last_peered":"2024-03-26T04:07:37.701776+0000","last_clean":"2024-03-26T04:07:37.701776+0000","last_became_active":"2024-03-26T04:01:35.797214+0000","last_became_peered":"2024-03-26T04:01:35.797214+0000","last_unstale":"2024-03-26T04:07:37.701776+0000","last_undegraded":"2024-03-26T04:07:37.701776+0000","last_fullsized":"2024-03-26T04:07:37.701776+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:01.292182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:01.292182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":837,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701758+0000","last_change":"2024-03-26T04:05:29.520607+0000","last_active":"2024-03-26T04:07:37.701758+0000","last_peered":"2024-03-26T04:07:37.701758+0000","last_clean":"2024-03-26T04:07:37.701758+0000","last_became_active":"2024-03-26T04:05:29.519855+0000","last_became_peered":"2024-03-26T04:05:29.519855+0000","last_unstale":"2024-03-26T04:07:37.701758+0000","last_undegraded":"2024-03-26T04:07:37.701758+0000","last_fullsized":"2024-03-26T04:07:37.701758+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:01:08.181298+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.181298+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":799,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702630+0000","last_change":"2024-03-26T04:05:02.671720+0000","last_active":"2024-03-26T04:07:37.702630+0000","last_peered":"2024-03-26T04:07:37.702630+0000","last_clean":"2024-03-26T04:07:37.702630+0000","last_became_active":"2024-03-26T04:02:52.500023+0000","last_became_peered":"2024-03-26T04:02:52.500023+0000","last_unstale":"2024-03-26T04:07:37.702630+0000","last_undegraded":"2024-03-26T04:07:37.702630+0000","last_fullsized":"2024-03-26T04:07:37.702630+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:02.671615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:02.671615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":768,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834516+0000","last_change":"2024-03-26T04:04:51.186722+0000","last_active":"2024-03-26T04:07:37.834516+0000","last_peered":"2024-03-26T04:07:37.834516+0000","last_clean":"2024-03-26T04:07:37.834516+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:07:37.834516+0000","last_undegraded":"2024-03-26T04:07:37.834516+0000","last_fullsized":"2024-03-26T04:07:37.834516+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.186605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.186605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":778,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697960+0000","last_change":"2024-03-26T04:03:54.321625+0000","last_active":"2024-03-26T04:07:37.697960+0000","last_peered":"2024-03-26T04:07:37.697960+0000","last_clean":"2024-03-26T04:07:37.697960+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:07:37.697960+0000","last_undegraded":"2024-03-26T04:07:37.697960+0000","last_fullsized":"2024-03-26T04:07:37.697960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:54.321548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:54.321548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":800,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833687+0000","last_change":"2024-03-26T04:05:29.533402+0000","last_active":"2024-03-26T04:07:37.833687+0000","last_peered":"2024-03-26T04:07:37.833687+0000","last_clean":"2024-03-26T04:07:37.833687+0000","last_became_active":"2024-03-26T04:05:29.525450+0000","last_became_peered":"2024-03-26T04:05:29.525450+0000","last_unstale":"2024-03-26T04:07:37.833687+0000","last_undegraded":"2024-03-26T04:07:37.833687+0000","last_fullsized":"2024-03-26T04:07:37.833687+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:02:45.014541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:45.014541+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":401,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834991+0000","last_change":"2024-03-26T04:05:29.533637+0000","last_active":"2024-03-26T04:07:37.834991+0000","last_peered":"2024-03-26T04:07:37.834991+0000","last_clean":"2024-03-26T04:07:37.834991+0000","last_became_active":"2024-03-26T04:05:29.533287+0000","last_became_peered":"2024-03-26T04:05:29.533287+0000","last_unstale":"2024-03-26T04:07:37.834991+0000","last_undegraded":"2024-03-26T04:07:37.834991+0000","last_fullsized":"2024-03-26T04:07:37.834991+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:22.809763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.809763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":789,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702022+0000","last_change":"2024-03-26T04:07:37.463272+0000","last_active":"2024-03-26T04:07:37.702022+0000","last_peered":"2024-03-26T04:07:37.702022+0000","last_clean":"2024-03-26T04:07:37.702022+0000","last_became_active":"2024-03-26T04:05:29.536639+0000","last_became_peered":"2024-03-26T04:05:29.536639+0000","last_unstale":"2024-03-26T04:07:37.702022+0000","last_undegraded":"2024-03-26T04:07:37.702022+0000","last_fullsized":"2024-03-26T04:07:37.702022+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.463158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.463158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.787883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":779,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701345+0000","last_change":"2024-03-26T04:07:24.624918+0000","last_active":"2024-03-26T04:07:37.701345+0000","last_peered":"2024-03-26T04:07:37.701345+0000","last_clean":"2024-03-26T04:07:37.701345+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:07:37.701345+0000","last_undegraded":"2024-03-26T04:07:37.701345+0000","last_fullsized":"2024-03-26T04:07:37.701345+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.624793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.624793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.269799+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":324,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834077+0000","last_change":"2024-03-26T04:07:28.446713+0000","last_active":"2024-03-26T04:07:37.834077+0000","last_peered":"2024-03-26T04:07:37.834077+0000","last_clean":"2024-03-26T04:07:37.834077+0000","last_became_active":"2024-03-26T04:05:29.521493+0000","last_became_peered":"2024-03-26T04:05:29.521493+0000","last_unstale":"2024-03-26T04:07:37.834077+0000","last_undegraded":"2024-03-26T04:07:37.834077+0000","last_fullsized":"2024-03-26T04:07:37.834077+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:07:28.446562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:28.446562+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:32.529078+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":238,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833941+0000","last_change":"2024-03-26T04:05:29.548371+0000","last_active":"2024-03-26T04:07:37.833941+0000","last_peered":"2024-03-26T04:07:37.833941+0000","last_clean":"2024-03-26T04:07:37.833941+0000","last_became_active":"2024-03-26T04:05:29.545095+0000","last_became_peered":"2024-03-26T04:05:29.545095+0000","last_unstale":"2024-03-26T04:07:37.833941+0000","last_undegraded":"2024-03-26T04:07:37.833941+0000","last_fullsized":"2024-03-26T04:07:37.833941+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:02:40.908630+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:02:40.908630+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"379'11","reported_seq":945,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.304775+0000","last_change":"2024-03-26T04:05:29.535165+0000","last_active":"2024-03-26T04:07:44.304775+0000","last_peered":"2024-03-26T04:07:44.304775+0000","last_clean":"2024-03-26T04:07:44.304775+0000","last_became_active":"2024-03-26T04:05:29.534648+0000","last_became_peered":"2024-03-26T04:05:29.534648+0000","last_unstale":"2024-03-26T04:07:44.304775+0000","last_undegraded":"2024-03-26T04:07:44.304775+0000","last_fullsized":"2024-03-26T04:07:44.304775+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:35.094793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.094793+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":808,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833187+0000","last_change":"2024-03-26T04:04:18.067687+0000","last_active":"2024-03-26T04:07:37.833187+0000","last_peered":"2024-03-26T04:07:37.833187+0000","last_clean":"2024-03-26T04:07:37.833187+0000","last_became_active":"2024-03-26T04:02:52.514953+0000","last_became_peered":"2024-03-26T04:02:52.514953+0000","last_unstale":"2024-03-26T04:07:37.833187+0000","last_undegraded":"2024-03-26T04:07:37.833187+0000","last_fullsized":"2024-03-26T04:07:37.833187+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:04:18.067462+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.067462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"356'32","reported_seq":876,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698038+0000","last_change":"2024-03-26T04:04:05.345019+0000","last_active":"2024-03-26T04:07:37.698038+0000","last_peered":"2024-03-26T04:07:37.698038+0000","last_clean":"2024-03-26T04:07:37.698038+0000","last_became_active":"2024-03-26T04:02:52.497638+0000","last_became_peered":"2024-03-26T04:02:52.497638+0000","last_unstale":"2024-03-26T04:07:37.698038+0000","last_undegraded":"2024-03-26T04:07:37.698038+0000","last_fullsized":"2024-03-26T04:07:37.698038+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'32","last_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:04:05.344849+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":772,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834143+0000","last_change":"2024-03-26T04:05:29.520660+0000","last_active":"2024-03-26T04:07:37.834143+0000","last_peered":"2024-03-26T04:07:37.834143+0000","last_clean":"2024-03-26T04:07:37.834143+0000","last_became_active":"2024-03-26T04:05:29.520352+0000","last_became_peered":"2024-03-26T04:05:29.520352+0000","last_unstale":"2024-03-26T04:07:37.834143+0000","last_undegraded":"2024-03-26T04:07:37.834143+0000","last_fullsized":"2024-03-26T04:07:37.834143+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:21.633920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.633920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":773,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698255+0000","last_change":"2024-03-26T04:07:29.534176+0000","last_active":"2024-03-26T04:07:37.698255+0000","last_peered":"2024-03-26T04:07:37.698255+0000","last_clean":"2024-03-26T04:07:37.698255+0000","last_became_active":"2024-03-26T04:05:29.536597+0000","last_became_peered":"2024-03-26T04:05:29.536597+0000","last_unstale":"2024-03-26T04:07:37.698255+0000","last_undegraded":"2024-03-26T04:07:37.698255+0000","last_fullsized":"2024-03-26T04:07:37.698255+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:29.533765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.533765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:31.454229+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":365,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696845+0000","last_change":"2024-03-26T04:07:31.484578+0000","last_active":"2024-03-26T04:07:37.696845+0000","last_peered":"2024-03-26T04:07:37.696845+0000","last_clean":"2024-03-26T04:07:37.696845+0000","last_became_active":"2024-03-26T04:05:29.540582+0000","last_became_peered":"2024-03-26T04:05:29.540582+0000","last_unstale":"2024-03-26T04:07:37.696845+0000","last_undegraded":"2024-03-26T04:07:37.696845+0000","last_fullsized":"2024-03-26T04:07:37.696845+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:31.484355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.484355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:56.172779+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":599,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835034+0000","last_change":"2024-03-26T04:05:29.537371+0000","last_active":"2024-03-26T04:07:37.835034+0000","last_peered":"2024-03-26T04:07:37.835034+0000","last_clean":"2024-03-26T04:07:37.835034+0000","last_became_active":"2024-03-26T04:05:29.536977+0000","last_became_peered":"2024-03-26T04:05:29.536977+0000","last_unstale":"2024-03-26T04:07:37.835034+0000","last_undegraded":"2024-03-26T04:07:37.835034+0000","last_fullsized":"2024-03-26T04:07:37.835034+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":790,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:42.011214+0000","last_change":"2024-03-26T04:07:42.011236+0000","last_active":"2024-03-26T04:07:42.011214+0000","last_peered":"2024-03-26T04:07:42.011214+0000","last_clean":"2024-03-26T04:07:42.011214+0000","last_became_active":"2024-03-26T04:05:29.533627+0000","last_became_peered":"2024-03-26T04:05:29.533627+0000","last_unstale":"2024-03-26T04:07:42.011214+0000","last_undegraded":"2024-03-26T04:07:42.011214+0000","last_fullsized":"2024-03-26T04:07:42.011214+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.011134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:43.732641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":701,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701036+0000","last_change":"2024-03-26T04:04:17.385877+0000","last_active":"2024-03-26T04:07:37.701036+0000","last_peered":"2024-03-26T04:07:37.701036+0000","last_clean":"2024-03-26T04:07:37.701036+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:07:37.701036+0000","last_undegraded":"2024-03-26T04:07:37.701036+0000","last_fullsized":"2024-03-26T04:07:37.701036+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:17.385765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.385765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":812,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698448+0000","last_change":"2024-03-26T04:07:37.490184+0000","last_active":"2024-03-26T04:07:37.698448+0000","last_peered":"2024-03-26T04:07:37.698448+0000","last_clean":"2024-03-26T04:07:37.698448+0000","last_became_active":"2024-03-26T04:05:29.538915+0000","last_became_peered":"2024-03-26T04:05:29.538915+0000","last_unstale":"2024-03-26T04:07:37.698448+0000","last_undegraded":"2024-03-26T04:07:37.698448+0000","last_fullsized":"2024-03-26T04:07:37.698448+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.490105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.490105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:05.057235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":298,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832905+0000","last_change":"2024-03-26T04:05:29.520954+0000","last_active":"2024-03-26T04:07:37.832905+0000","last_peered":"2024-03-26T04:07:37.832905+0000","last_clean":"2024-03-26T04:07:37.832905+0000","last_became_active":"2024-03-26T04:05:29.520584+0000","last_became_peered":"2024-03-26T04:05:29.520584+0000","last_unstale":"2024-03-26T04:07:37.832905+0000","last_undegraded":"2024-03-26T04:07:37.832905+0000","last_fullsized":"2024-03-26T04:07:37.832905+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:00:38.458453+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.458453+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":423,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697475+0000","last_change":"2024-03-26T04:05:29.537044+0000","last_active":"2024-03-26T04:07:37.697475+0000","last_peered":"2024-03-26T04:07:37.697475+0000","last_clean":"2024-03-26T04:07:37.697475+0000","last_became_active":"2024-03-26T04:05:29.536699+0000","last_became_peered":"2024-03-26T04:05:29.536699+0000","last_unstale":"2024-03-26T04:07:37.697475+0000","last_undegraded":"2024-03-26T04:07:37.697475+0000","last_fullsized":"2024-03-26T04:07:37.697475+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.635002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.635002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":791,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832589+0000","last_change":"2024-03-26T04:05:48.811855+0000","last_active":"2024-03-26T04:07:37.832589+0000","last_peered":"2024-03-26T04:07:37.832589+0000","last_clean":"2024-03-26T04:07:37.832589+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:07:37.832589+0000","last_undegraded":"2024-03-26T04:07:37.832589+0000","last_fullsized":"2024-03-26T04:07:37.832589+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:48.811706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:48.811706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":517,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.423642+0000","last_change":"2024-03-26T04:07:39.423667+0000","last_active":"2024-03-26T04:07:39.423642+0000","last_peered":"2024-03-26T04:07:39.423642+0000","last_clean":"2024-03-26T04:07:39.423642+0000","last_became_active":"2024-03-26T04:05:29.539760+0000","last_became_peered":"2024-03-26T04:05:29.539760+0000","last_unstale":"2024-03-26T04:07:39.423642+0000","last_undegraded":"2024-03-26T04:07:39.423642+0000","last_fullsized":"2024-03-26T04:07:39.423642+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:39.423550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.423550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:56.251695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":441,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696661+0000","last_change":"2024-03-26T04:05:29.545345+0000","last_active":"2024-03-26T04:07:37.696661+0000","last_peered":"2024-03-26T04:07:37.696661+0000","last_clean":"2024-03-26T04:07:37.696661+0000","last_became_active":"2024-03-26T04:05:29.542754+0000","last_became_peered":"2024-03-26T04:05:29.542754+0000","last_unstale":"2024-03-26T04:07:37.696661+0000","last_undegraded":"2024-03-26T04:07:37.696661+0000","last_fullsized":"2024-03-26T04:07:37.696661+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:01:19.700471+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.700471+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":780,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696476+0000","last_change":"2024-03-26T04:03:07.149962+0000","last_active":"2024-03-26T04:07:37.696476+0000","last_peered":"2024-03-26T04:07:37.696476+0000","last_clean":"2024-03-26T04:07:37.696476+0000","last_became_active":"2024-03-26T04:02:52.506113+0000","last_became_peered":"2024-03-26T04:02:52.506113+0000","last_unstale":"2024-03-26T04:07:37.696476+0000","last_undegraded":"2024-03-26T04:07:37.696476+0000","last_fullsized":"2024-03-26T04:07:37.696476+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:07.149881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.149881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":365,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697608+0000","last_change":"2024-03-26T04:04:51.740064+0000","last_active":"2024-03-26T04:07:37.697608+0000","last_peered":"2024-03-26T04:07:37.697608+0000","last_clean":"2024-03-26T04:07:37.697608+0000","last_became_active":"2024-03-26T04:02:52.489626+0000","last_became_peered":"2024-03-26T04:02:52.489626+0000","last_unstale":"2024-03-26T04:07:37.697608+0000","last_undegraded":"2024-03-26T04:07:37.697608+0000","last_fullsized":"2024-03-26T04:07:37.697608+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.739925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.739925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":810,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833197+0000","last_change":"2024-03-26T04:05:54.943954+0000","last_active":"2024-03-26T04:07:37.833197+0000","last_peered":"2024-03-26T04:07:37.833197+0000","last_clean":"2024-03-26T04:07:37.833197+0000","last_became_active":"2024-03-26T04:02:52.498923+0000","last_became_peered":"2024-03-26T04:02:52.498923+0000","last_unstale":"2024-03-26T04:07:37.833197+0000","last_undegraded":"2024-03-26T04:07:37.833197+0000","last_fullsized":"2024-03-26T04:07:37.833197+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:54.943879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:54.943879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":827,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833866+0000","last_change":"2024-03-26T04:05:29.535609+0000","last_active":"2024-03-26T04:07:37.833866+0000","last_peered":"2024-03-26T04:07:37.833866+0000","last_clean":"2024-03-26T04:07:37.833866+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:07:37.833866+0000","last_undegraded":"2024-03-26T04:07:37.833866+0000","last_fullsized":"2024-03-26T04:07:37.833866+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:01:36.382121+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.382121+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":401,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696974+0000","last_change":"2024-03-26T04:05:29.538705+0000","last_active":"2024-03-26T04:07:37.696974+0000","last_peered":"2024-03-26T04:07:37.696974+0000","last_clean":"2024-03-26T04:07:37.696974+0000","last_became_active":"2024-03-26T04:05:29.537974+0000","last_became_peered":"2024-03-26T04:05:29.537974+0000","last_unstale":"2024-03-26T04:07:37.696974+0000","last_undegraded":"2024-03-26T04:07:37.696974+0000","last_fullsized":"2024-03-26T04:07:37.696974+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.709457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.709457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":794,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833771+0000","last_change":"2024-03-26T04:03:57.362649+0000","last_active":"2024-03-26T04:07:37.833771+0000","last_peered":"2024-03-26T04:07:37.833771+0000","last_clean":"2024-03-26T04:07:37.833771+0000","last_became_active":"2024-03-26T04:02:52.512541+0000","last_became_peered":"2024-03-26T04:02:52.512541+0000","last_unstale":"2024-03-26T04:07:37.833771+0000","last_undegraded":"2024-03-26T04:07:37.833771+0000","last_fullsized":"2024-03-26T04:07:37.833771+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:57.362566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:57.362566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":786,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701895+0000","last_change":"2024-03-26T04:05:29.536775+0000","last_active":"2024-03-26T04:07:37.701895+0000","last_peered":"2024-03-26T04:07:37.701895+0000","last_clean":"2024-03-26T04:07:37.701895+0000","last_became_active":"2024-03-26T04:05:29.536500+0000","last_became_peered":"2024-03-26T04:05:29.536500+0000","last_unstale":"2024-03-26T04:07:37.701895+0000","last_undegraded":"2024-03-26T04:07:37.701895+0000","last_fullsized":"2024-03-26T04:07:37.701895+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:01:35.104690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":844,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701892+0000","last_change":"2024-03-26T04:05:29.521309+0000","last_active":"2024-03-26T04:07:37.701892+0000","last_peered":"2024-03-26T04:07:37.701892+0000","last_clean":"2024-03-26T04:07:37.701892+0000","last_became_active":"2024-03-26T04:05:29.520817+0000","last_became_peered":"2024-03-26T04:05:29.520817+0000","last_unstale":"2024-03-26T04:07:37.701892+0000","last_undegraded":"2024-03-26T04:07:37.701892+0000","last_fullsized":"2024-03-26T04:07:37.701892+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:02:51.525641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:51.525641+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":776,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697762+0000","last_change":"2024-03-26T04:04:58.704933+0000","last_active":"2024-03-26T04:07:37.697762+0000","last_peered":"2024-03-26T04:07:37.697762+0000","last_clean":"2024-03-26T04:07:37.697762+0000","last_became_active":"2024-03-26T04:02:52.497109+0000","last_became_peered":"2024-03-26T04:02:52.497109+0000","last_unstale":"2024-03-26T04:07:37.697762+0000","last_undegraded":"2024-03-26T04:07:37.697762+0000","last_fullsized":"2024-03-26T04:07:37.697762+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:04:58.704813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":428,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.303626+0000","last_change":"2024-03-26T04:03:53.367019+0000","last_active":"2024-03-26T04:07:39.303626+0000","last_peered":"2024-03-26T04:07:39.303626+0000","last_clean":"2024-03-26T04:07:39.303626+0000","last_became_active":"2024-03-26T04:02:52.511718+0000","last_became_peered":"2024-03-26T04:02:52.511718+0000","last_unstale":"2024-03-26T04:07:39.303626+0000","last_undegraded":"2024-03-26T04:07:39.303626+0000","last_fullsized":"2024-03-26T04:07:39.303626+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:53.366906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.366906+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":802,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832597+0000","last_change":"2024-03-26T04:07:32.845599+0000","last_active":"2024-03-26T04:07:37.832597+0000","last_peered":"2024-03-26T04:07:37.832597+0000","last_clean":"2024-03-26T04:07:37.832597+0000","last_became_active":"2024-03-26T04:05:29.533466+0000","last_became_peered":"2024-03-26T04:05:29.533466+0000","last_unstale":"2024-03-26T04:07:37.832597+0000","last_undegraded":"2024-03-26T04:07:37.832597+0000","last_fullsized":"2024-03-26T04:07:37.832597+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:32.845486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:32.845486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:54.200681+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"422'39","reported_seq":342,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:42.462265+0000","last_change":"2024-03-26T04:07:42.462298+0000","last_active":"2024-03-26T04:07:42.462265+0000","last_peered":"2024-03-26T04:07:42.462265+0000","last_clean":"2024-03-26T04:07:42.462265+0000","last_became_active":"2024-03-26T04:05:29.540006+0000","last_became_peered":"2024-03-26T04:05:29.540006+0000","last_unstale":"2024-03-26T04:07:42.462265+0000","last_undegraded":"2024-03-26T04:07:42.462265+0000","last_fullsized":"2024-03-26T04:07:42.462265+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'39","last_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.462212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:52.583235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":39,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":774,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.703023+0000","last_change":"2024-03-26T04:05:29.522233+0000","last_active":"2024-03-26T04:07:37.703023+0000","last_peered":"2024-03-26T04:07:37.703023+0000","last_clean":"2024-03-26T04:07:37.703023+0000","last_became_active":"2024-03-26T04:05:29.521886+0000","last_became_peered":"2024-03-26T04:05:29.521886+0000","last_unstale":"2024-03-26T04:07:37.703023+0000","last_undegraded":"2024-03-26T04:07:37.703023+0000","last_fullsized":"2024-03-26T04:07:37.703023+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.428232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.428232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"379'14","reported_seq":200,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.304479+0000","last_change":"2024-03-26T04:05:29.541223+0000","last_active":"2024-03-26T04:07:44.304479+0000","last_peered":"2024-03-26T04:07:44.304479+0000","last_clean":"2024-03-26T04:07:44.304479+0000","last_became_active":"2024-03-26T04:05:29.540855+0000","last_became_peered":"2024-03-26T04:05:29.540855+0000","last_unstale":"2024-03-26T04:07:44.304479+0000","last_undegraded":"2024-03-26T04:07:44.304479+0000","last_fullsized":"2024-03-26T04:07:44.304479+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'12","last_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_deep_scrub":"334'12","last_deep_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.907441+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":821,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697383+0000","last_change":"2024-03-26T04:04:17.541024+0000","last_active":"2024-03-26T04:07:37.697383+0000","last_peered":"2024-03-26T04:07:37.697383+0000","last_clean":"2024-03-26T04:07:37.697383+0000","last_became_active":"2024-03-26T04:02:52.492361+0000","last_became_peered":"2024-03-26T04:02:52.492361+0000","last_unstale":"2024-03-26T04:07:37.697383+0000","last_undegraded":"2024-03-26T04:07:37.697383+0000","last_fullsized":"2024-03-26T04:07:37.697383+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.538770+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":741,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833416+0000","last_change":"2024-03-26T04:06:21.728756+0000","last_active":"2024-03-26T04:07:37.833416+0000","last_peered":"2024-03-26T04:07:37.833416+0000","last_clean":"2024-03-26T04:07:37.833416+0000","last_became_active":"2024-03-26T04:02:52.505421+0000","last_became_peered":"2024-03-26T04:02:52.505421+0000","last_unstale":"2024-03-26T04:07:37.833416+0000","last_undegraded":"2024-03-26T04:07:37.833416+0000","last_fullsized":"2024-03-26T04:07:37.833416+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:06:21.728608+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:06:21.728608+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":688,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834261+0000","last_change":"2024-03-26T04:05:29.548434+0000","last_active":"2024-03-26T04:07:37.834261+0000","last_peered":"2024-03-26T04:07:37.834261+0000","last_clean":"2024-03-26T04:07:37.834261+0000","last_became_active":"2024-03-26T04:05:29.546602+0000","last_became_peered":"2024-03-26T04:05:29.546602+0000","last_unstale":"2024-03-26T04:07:37.834261+0000","last_undegraded":"2024-03-26T04:07:37.834261+0000","last_fullsized":"2024-03-26T04:07:37.834261+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:01:13.279325+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:01:13.279325+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":356,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834941+0000","last_change":"2024-03-26T04:03:58.337609+0000","last_active":"2024-03-26T04:07:37.834941+0000","last_peered":"2024-03-26T04:07:37.834941+0000","last_clean":"2024-03-26T04:07:37.834941+0000","last_became_active":"2024-03-26T04:02:52.505042+0000","last_became_peered":"2024-03-26T04:02:52.505042+0000","last_unstale":"2024-03-26T04:07:37.834941+0000","last_undegraded":"2024-03-26T04:07:37.834941+0000","last_fullsized":"2024-03-26T04:07:37.834941+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:58.337496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:58.337496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":479,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701201+0000","last_change":"2024-03-26T04:04:57.523173+0000","last_active":"2024-03-26T04:07:37.701201+0000","last_peered":"2024-03-26T04:07:37.701201+0000","last_clean":"2024-03-26T04:07:37.701201+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:07:37.701201+0000","last_undegraded":"2024-03-26T04:07:37.701201+0000","last_fullsized":"2024-03-26T04:07:37.701201+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:57.522794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:57.522794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"422'33","reported_seq":890,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:45.584193+0000","last_change":"2024-03-26T04:07:45.584228+0000","last_active":"2024-03-26T04:07:45.584193+0000","last_peered":"2024-03-26T04:07:45.584193+0000","last_clean":"2024-03-26T04:07:45.584193+0000","last_became_active":"2024-03-26T04:05:29.537550+0000","last_became_peered":"2024-03-26T04:05:29.537550+0000","last_unstale":"2024-03-26T04:07:45.584193+0000","last_undegraded":"2024-03-26T04:07:45.584193+0000","last_fullsized":"2024-03-26T04:07:45.584193+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'33","last_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.584130+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:57.820980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":794,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833187+0000","last_change":"2024-03-26T04:06:05.957492+0000","last_active":"2024-03-26T04:07:37.833187+0000","last_peered":"2024-03-26T04:07:37.833187+0000","last_clean":"2024-03-26T04:07:37.833187+0000","last_became_active":"2024-03-26T04:01:35.799877+0000","last_became_peered":"2024-03-26T04:01:35.799877+0000","last_unstale":"2024-03-26T04:07:37.833187+0000","last_undegraded":"2024-03-26T04:07:37.833187+0000","last_fullsized":"2024-03-26T04:07:37.833187+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:05.957415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:05.957415+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":510,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834854+0000","last_change":"2024-03-26T04:04:10.223664+0000","last_active":"2024-03-26T04:07:37.834854+0000","last_peered":"2024-03-26T04:07:37.834854+0000","last_clean":"2024-03-26T04:07:37.834854+0000","last_became_active":"2024-03-26T04:02:52.502225+0000","last_became_peered":"2024-03-26T04:02:52.502225+0000","last_unstale":"2024-03-26T04:07:37.834854+0000","last_undegraded":"2024-03-26T04:07:37.834854+0000","last_fullsized":"2024-03-26T04:07:37.834854+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:04:10.223350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":743,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696791+0000","last_change":"2024-03-26T04:05:29.540410+0000","last_active":"2024-03-26T04:07:37.696791+0000","last_peered":"2024-03-26T04:07:37.696791+0000","last_clean":"2024-03-26T04:07:37.696791+0000","last_became_active":"2024-03-26T04:05:29.540070+0000","last_became_peered":"2024-03-26T04:05:29.540070+0000","last_unstale":"2024-03-26T04:07:37.696791+0000","last_undegraded":"2024-03-26T04:07:37.696791+0000","last_fullsized":"2024-03-26T04:07:37.696791+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:48.062863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:02:48.062863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":228,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835048+0000","last_change":"2024-03-26T04:05:29.539905+0000","last_active":"2024-03-26T04:07:37.835048+0000","last_peered":"2024-03-26T04:07:37.835048+0000","last_clean":"2024-03-26T04:07:37.835048+0000","last_became_active":"2024-03-26T04:05:29.539596+0000","last_became_peered":"2024-03-26T04:05:29.539596+0000","last_unstale":"2024-03-26T04:07:37.835048+0000","last_undegraded":"2024-03-26T04:07:37.835048+0000","last_fullsized":"2024-03-26T04:07:37.835048+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:00:21.475271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:21.475271+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.6","version":"421'21382","reported_seq":21539,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697494+0000","last_change":"2024-03-26T04:03:47.643174+0000","last_active":"2024-03-26T04:07:37.697494+0000","last_peered":"2024-03-26T04:07:37.697494+0000","last_clean":"2024-03-26T04:07:37.697494+0000","last_became_active":"2024-03-26T04:03:47.642762+0000","last_became_peered":"2024-03-26T04:03:47.642762+0000","last_unstale":"2024-03-26T04:07:37.697494+0000","last_undegraded":"2024-03-26T04:07:37.697494+0000","last_fullsized":"2024-03-26T04:07:37.697494+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3082,"log_dups_size":0,"ondisk_log_size":3082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21382,"num_write_kb":684224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":495,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697547+0000","last_change":"2024-03-26T04:05:29.537569+0000","last_active":"2024-03-26T04:07:37.697547+0000","last_peered":"2024-03-26T04:07:37.697547+0000","last_clean":"2024-03-26T04:07:37.697547+0000","last_became_active":"2024-03-26T04:05:29.537211+0000","last_became_peered":"2024-03-26T04:05:29.537211+0000","last_unstale":"2024-03-26T04:07:37.697547+0000","last_undegraded":"2024-03-26T04:07:37.697547+0000","last_fullsized":"2024-03-26T04:07:37.697547+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:50.068294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:50.068294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":806,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833920+0000","last_change":"2024-03-26T04:06:27.796450+0000","last_active":"2024-03-26T04:07:37.833920+0000","last_peered":"2024-03-26T04:07:37.833920+0000","last_clean":"2024-03-26T04:07:37.833920+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:07:37.833920+0000","last_undegraded":"2024-03-26T04:07:37.833920+0000","last_fullsized":"2024-03-26T04:07:37.833920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T04:06:27.796333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:27.796333+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":773,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832466+0000","last_change":"2024-03-26T04:07:25.796444+0000","last_active":"2024-03-26T04:07:37.832466+0000","last_peered":"2024-03-26T04:07:37.832466+0000","last_clean":"2024-03-26T04:07:37.832466+0000","last_became_active":"2024-03-26T04:02:52.505008+0000","last_became_peered":"2024-03-26T04:02:52.505008+0000","last_unstale":"2024-03-26T04:07:37.832466+0000","last_undegraded":"2024-03-26T04:07:37.832466+0000","last_fullsized":"2024-03-26T04:07:37.832466+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:07:25.796341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.796341+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:27.516967+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":1033,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.304731+0000","last_change":"2024-03-26T04:05:45.805637+0000","last_active":"2024-03-26T04:07:44.304731+0000","last_peered":"2024-03-26T04:07:44.304731+0000","last_clean":"2024-03-26T04:07:44.304731+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:07:44.304731+0000","last_undegraded":"2024-03-26T04:07:44.304731+0000","last_fullsized":"2024-03-26T04:07:44.304731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:05:45.805513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:45.805513+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"421'21634","reported_seq":22203,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834437+0000","last_change":"2024-03-26T04:07:35.582310+0000","last_active":"2024-03-26T04:07:37.834437+0000","last_peered":"2024-03-26T04:07:37.834437+0000","last_clean":"2024-03-26T04:07:37.834437+0000","last_became_active":"2024-03-26T04:07:35.562368+0000","last_became_peered":"2024-03-26T04:07:35.562368+0000","last_unstale":"2024-03-26T04:07:37.834437+0000","last_undegraded":"2024-03-26T04:07:37.834437+0000","last_fullsized":"2024-03-26T04:07:37.834437+0000","mapping_epoch":420,"log_start":"404'18500","ondisk_log_start":"404'18500","created":349,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3134,"log_dups_size":10,"ondisk_log_size":3134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21634,"num_write_kb":692288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12976128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":371,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832732+0000","last_change":"2024-03-26T04:06:01.998434+0000","last_active":"2024-03-26T04:07:37.832732+0000","last_peered":"2024-03-26T04:07:37.832732+0000","last_clean":"2024-03-26T04:07:37.832732+0000","last_became_active":"2024-03-26T04:02:52.505076+0000","last_became_peered":"2024-03-26T04:02:52.505076+0000","last_unstale":"2024-03-26T04:07:37.832732+0000","last_undegraded":"2024-03-26T04:07:37.832732+0000","last_fullsized":"2024-03-26T04:07:37.832732+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:01.998326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:06:01.998326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":793,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833453+0000","last_change":"2024-03-26T04:06:06.861624+0000","last_active":"2024-03-26T04:07:37.833453+0000","last_peered":"2024-03-26T04:07:37.833453+0000","last_clean":"2024-03-26T04:07:37.833453+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:07:37.833453+0000","last_undegraded":"2024-03-26T04:07:37.833453+0000","last_fullsized":"2024-03-26T04:07:37.833453+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:06.861506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:06.861506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":889,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701278+0000","last_change":"2024-03-26T04:04:19.443761+0000","last_active":"2024-03-26T04:07:37.701278+0000","last_peered":"2024-03-26T04:07:37.701278+0000","last_clean":"2024-03-26T04:07:37.701278+0000","last_became_active":"2024-03-26T04:02:52.513309+0000","last_became_peered":"2024-03-26T04:02:52.513309+0000","last_unstale":"2024-03-26T04:07:37.701278+0000","last_undegraded":"2024-03-26T04:07:37.701278+0000","last_fullsized":"2024-03-26T04:07:37.701278+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:04:19.443608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.443608+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":690,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832649+0000","last_change":"2024-03-26T04:05:29.533866+0000","last_active":"2024-03-26T04:07:37.832649+0000","last_peered":"2024-03-26T04:07:37.832649+0000","last_clean":"2024-03-26T04:07:37.832649+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:07:37.832649+0000","last_undegraded":"2024-03-26T04:07:37.832649+0000","last_fullsized":"2024-03-26T04:07:37.832649+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:11.271561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:11.271561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"421'21466","reported_seq":21623,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697209+0000","last_change":"2024-03-26T04:03:47.643711+0000","last_active":"2024-03-26T04:07:37.697209+0000","last_peered":"2024-03-26T04:07:37.697209+0000","last_clean":"2024-03-26T04:07:37.697209+0000","last_became_active":"2024-03-26T04:03:47.643384+0000","last_became_peered":"2024-03-26T04:03:47.643384+0000","last_unstale":"2024-03-26T04:07:37.697209+0000","last_undegraded":"2024-03-26T04:07:37.697209+0000","last_fullsized":"2024-03-26T04:07:37.697209+0000","mapping_epoch":349,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3066,"log_dups_size":0,"ondisk_log_size":3066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21466,"num_write_kb":686912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":645,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832510+0000","last_change":"2024-03-26T04:05:29.543839+0000","last_active":"2024-03-26T04:07:37.832510+0000","last_peered":"2024-03-26T04:07:37.832510+0000","last_clean":"2024-03-26T04:07:37.832510+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:07:37.832510+0000","last_undegraded":"2024-03-26T04:07:37.832510+0000","last_fullsized":"2024-03-26T04:07:37.832510+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:37.891864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.891864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":691,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833223+0000","last_change":"2024-03-26T04:05:29.535670+0000","last_active":"2024-03-26T04:07:37.833223+0000","last_peered":"2024-03-26T04:07:37.833223+0000","last_clean":"2024-03-26T04:07:37.833223+0000","last_became_active":"2024-03-26T04:05:29.534559+0000","last_became_peered":"2024-03-26T04:05:29.534559+0000","last_unstale":"2024-03-26T04:07:37.833223+0000","last_undegraded":"2024-03-26T04:07:37.833223+0000","last_fullsized":"2024-03-26T04:07:37.833223+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.965300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":578,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701786+0000","last_change":"2024-03-26T04:04:14.382770+0000","last_active":"2024-03-26T04:07:37.701786+0000","last_peered":"2024-03-26T04:07:37.701786+0000","last_clean":"2024-03-26T04:07:37.701786+0000","last_became_active":"2024-03-26T04:02:52.500569+0000","last_became_peered":"2024-03-26T04:02:52.500569+0000","last_unstale":"2024-03-26T04:07:37.701786+0000","last_undegraded":"2024-03-26T04:07:37.701786+0000","last_fullsized":"2024-03-26T04:07:37.701786+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:14.382655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:14.382655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"356'51","reported_seq":863,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833711+0000","last_change":"2024-03-26T04:04:03.371506+0000","last_active":"2024-03-26T04:07:37.833711+0000","last_peered":"2024-03-26T04:07:37.833711+0000","last_clean":"2024-03-26T04:07:37.833711+0000","last_became_active":"2024-03-26T04:01:35.818705+0000","last_became_peered":"2024-03-26T04:01:35.818705+0000","last_unstale":"2024-03-26T04:07:37.833711+0000","last_undegraded":"2024-03-26T04:07:37.833711+0000","last_fullsized":"2024-03-26T04:07:37.833711+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'51","last_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.371335+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.5","version":"421'22006","reported_seq":29865,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834341+0000","last_change":"2024-03-26T04:07:04.562327+0000","last_active":"2024-03-26T04:07:37.834341+0000","last_peered":"2024-03-26T04:07:37.834341+0000","last_clean":"2024-03-26T04:07:37.834341+0000","last_became_active":"2024-03-26T04:07:04.537258+0000","last_became_peered":"2024-03-26T04:07:04.537258+0000","last_unstale":"2024-03-26T04:07:37.834341+0000","last_undegraded":"2024-03-26T04:07:37.834341+0000","last_fullsized":"2024-03-26T04:07:37.834341+0000","mapping_epoch":407,"log_start":"400'18300","ondisk_log_start":"400'18300","created":349,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3706,"log_dups_size":0,"ondisk_log_size":3706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22006,"num_write_kb":704192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3844,"num_bytes_recovered":251854848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":760,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697197+0000","last_change":"2024-03-26T04:05:29.543248+0000","last_active":"2024-03-26T04:07:37.697197+0000","last_peered":"2024-03-26T04:07:37.697197+0000","last_clean":"2024-03-26T04:07:37.697197+0000","last_became_active":"2024-03-26T04:05:29.541737+0000","last_became_peered":"2024-03-26T04:05:29.541737+0000","last_unstale":"2024-03-26T04:07:37.697197+0000","last_undegraded":"2024-03-26T04:07:37.697197+0000","last_fullsized":"2024-03-26T04:07:37.697197+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.410420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.410420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":707,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832687+0000","last_change":"2024-03-26T04:07:37.592031+0000","last_active":"2024-03-26T04:07:37.832687+0000","last_peered":"2024-03-26T04:07:37.832687+0000","last_clean":"2024-03-26T04:07:37.832687+0000","last_became_active":"2024-03-26T04:05:29.524644+0000","last_became_peered":"2024-03-26T04:05:29.524644+0000","last_unstale":"2024-03-26T04:07:37.832687+0000","last_undegraded":"2024-03-26T04:07:37.832687+0000","last_fullsized":"2024-03-26T04:07:37.832687+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.591946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.591946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.371327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":541,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833799+0000","last_change":"2024-03-26T04:07:34.405175+0000","last_active":"2024-03-26T04:07:37.833799+0000","last_peered":"2024-03-26T04:07:37.833799+0000","last_clean":"2024-03-26T04:07:37.833799+0000","last_became_active":"2024-03-26T04:05:29.531951+0000","last_became_peered":"2024-03-26T04:05:29.531951+0000","last_unstale":"2024-03-26T04:07:37.833799+0000","last_undegraded":"2024-03-26T04:07:37.833799+0000","last_fullsized":"2024-03-26T04:07:37.833799+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.404972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.404972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.885121+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":857,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698199+0000","last_change":"2024-03-26T04:04:55.743092+0000","last_active":"2024-03-26T04:07:37.698199+0000","last_peered":"2024-03-26T04:07:37.698199+0000","last_clean":"2024-03-26T04:07:37.698199+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:07:37.698199+0000","last_undegraded":"2024-03-26T04:07:37.698199+0000","last_fullsized":"2024-03-26T04:07:37.698199+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:04:55.742967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.742967+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"421'21544","reported_seq":30917,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697230+0000","last_change":"2024-03-26T04:07:03.053873+0000","last_active":"2024-03-26T04:07:37.697230+0000","last_peered":"2024-03-26T04:07:37.697230+0000","last_clean":"2024-03-26T04:07:37.697230+0000","last_became_active":"2024-03-26T04:07:03.030042+0000","last_became_peered":"2024-03-26T04:07:03.030042+0000","last_unstale":"2024-03-26T04:07:37.697230+0000","last_undegraded":"2024-03-26T04:07:37.697230+0000","last_fullsized":"2024-03-26T04:07:37.697230+0000","mapping_epoch":405,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21544,"num_write_kb":689408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4603,"num_bytes_recovered":301662208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":582,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696751+0000","last_change":"2024-03-26T04:05:29.538976+0000","last_active":"2024-03-26T04:07:37.696751+0000","last_peered":"2024-03-26T04:07:37.696751+0000","last_clean":"2024-03-26T04:07:37.696751+0000","last_became_active":"2024-03-26T04:05:29.538433+0000","last_became_peered":"2024-03-26T04:05:29.538433+0000","last_unstale":"2024-03-26T04:07:37.696751+0000","last_undegraded":"2024-03-26T04:07:37.696751+0000","last_fullsized":"2024-03-26T04:07:37.696751+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:37.939498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.939498+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":366,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702339+0000","last_change":"2024-03-26T04:04:03.264229+0000","last_active":"2024-03-26T04:07:37.702339+0000","last_peered":"2024-03-26T04:07:37.702339+0000","last_clean":"2024-03-26T04:07:37.702339+0000","last_became_active":"2024-03-26T04:02:52.520574+0000","last_became_peered":"2024-03-26T04:02:52.520574+0000","last_unstale":"2024-03-26T04:07:37.702339+0000","last_undegraded":"2024-03-26T04:07:37.702339+0000","last_fullsized":"2024-03-26T04:07:37.702339+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:03.264133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.264133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":665,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833582+0000","last_change":"2024-03-26T04:04:17.110459+0000","last_active":"2024-03-26T04:07:37.833582+0000","last_peered":"2024-03-26T04:07:37.833582+0000","last_clean":"2024-03-26T04:07:37.833582+0000","last_became_active":"2024-03-26T04:02:52.495493+0000","last_became_peered":"2024-03-26T04:02:52.495493+0000","last_unstale":"2024-03-26T04:07:37.833582+0000","last_undegraded":"2024-03-26T04:07:37.833582+0000","last_fullsized":"2024-03-26T04:07:37.833582+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:04:17.110315+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.110315+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":778,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701683+0000","last_change":"2024-03-26T04:04:52.483535+0000","last_active":"2024-03-26T04:07:37.701683+0000","last_peered":"2024-03-26T04:07:37.701683+0000","last_clean":"2024-03-26T04:07:37.701683+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:07:37.701683+0000","last_undegraded":"2024-03-26T04:07:37.701683+0000","last_fullsized":"2024-03-26T04:07:37.701683+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:52.483429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:04:52.483429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"379'51","reported_seq":1174,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.303844+0000","last_change":"2024-03-26T04:05:29.540308+0000","last_active":"2024-03-26T04:07:39.303844+0000","last_peered":"2024-03-26T04:07:39.303844+0000","last_clean":"2024-03-26T04:07:39.303844+0000","last_became_active":"2024-03-26T04:05:29.539914+0000","last_became_peered":"2024-03-26T04:05:29.539914+0000","last_unstale":"2024-03-26T04:07:39.303844+0000","last_undegraded":"2024-03-26T04:07:39.303844+0000","last_fullsized":"2024-03-26T04:07:39.303844+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T04:01:23.451122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.451122+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":772,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697797+0000","last_change":"2024-03-26T04:05:04.831962+0000","last_active":"2024-03-26T04:07:37.697797+0000","last_peered":"2024-03-26T04:07:37.697797+0000","last_clean":"2024-03-26T04:07:37.697797+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:07:37.697797+0000","last_undegraded":"2024-03-26T04:07:37.697797+0000","last_fullsized":"2024-03-26T04:07:37.697797+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:04.831457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:04.831457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"421'21530","reported_seq":38483,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832979+0000","last_change":"2024-03-26T04:06:19.310400+0000","last_active":"2024-03-26T04:07:37.832979+0000","last_peered":"2024-03-26T04:07:37.832979+0000","last_clean":"2024-03-26T04:07:37.832979+0000","last_became_active":"2024-03-26T04:06:19.282014+0000","last_became_peered":"2024-03-26T04:06:19.282014+0000","last_unstale":"2024-03-26T04:07:37.832979+0000","last_undegraded":"2024-03-26T04:07:37.832979+0000","last_fullsized":"2024-03-26T04:07:37.832979+0000","mapping_epoch":392,"log_start":"400'17775","ondisk_log_start":"400'17775","created":349,"last_epoch_clean":393,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3755,"log_dups_size":1247,"ondisk_log_size":3755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21530,"num_write_kb":688960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8394,"num_bytes_recovered":550109184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":614,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697003+0000","last_change":"2024-03-26T04:04:11.458855+0000","last_active":"2024-03-26T04:07:37.697003+0000","last_peered":"2024-03-26T04:07:37.697003+0000","last_clean":"2024-03-26T04:07:37.697003+0000","last_became_active":"2024-03-26T04:00:32.425416+0000","last_became_peered":"2024-03-26T04:00:32.425416+0000","last_unstale":"2024-03-26T04:07:37.697003+0000","last_undegraded":"2024-03-26T04:07:37.697003+0000","last_fullsized":"2024-03-26T04:07:37.697003+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:11.458734+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:04:11.458734+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":816,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:38.023167+0000","last_change":"2024-03-26T04:07:38.023167+0000","last_active":"2024-03-26T04:07:38.023167+0000","last_peered":"2024-03-26T04:07:38.023167+0000","last_clean":"2024-03-26T04:07:38.023167+0000","last_became_active":"2024-03-26T04:05:29.542557+0000","last_became_peered":"2024-03-26T04:05:29.542557+0000","last_unstale":"2024-03-26T04:07:38.023167+0000","last_undegraded":"2024-03-26T04:07:38.023167+0000","last_fullsized":"2024-03-26T04:07:38.023167+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.023066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.026959+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":793,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833432+0000","last_change":"2024-03-26T04:05:29.535985+0000","last_active":"2024-03-26T04:07:37.833432+0000","last_peered":"2024-03-26T04:07:37.833432+0000","last_clean":"2024-03-26T04:07:37.833432+0000","last_became_active":"2024-03-26T04:05:29.533776+0000","last_became_peered":"2024-03-26T04:05:29.533776+0000","last_unstale":"2024-03-26T04:07:37.833432+0000","last_undegraded":"2024-03-26T04:07:37.833432+0000","last_fullsized":"2024-03-26T04:07:37.833432+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.324845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.324845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":716,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832889+0000","last_change":"2024-03-26T04:05:29.544757+0000","last_active":"2024-03-26T04:07:37.832889+0000","last_peered":"2024-03-26T04:07:37.832889+0000","last_clean":"2024-03-26T04:07:37.832889+0000","last_became_active":"2024-03-26T04:05:29.543967+0000","last_became_peered":"2024-03-26T04:05:29.543967+0000","last_unstale":"2024-03-26T04:07:37.832889+0000","last_undegraded":"2024-03-26T04:07:37.832889+0000","last_fullsized":"2024-03-26T04:07:37.832889+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:27.325285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:01:27.325285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"371'29","reported_seq":853,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834809+0000","last_change":"2024-03-26T04:04:59.217984+0000","last_active":"2024-03-26T04:07:37.834809+0000","last_peered":"2024-03-26T04:07:37.834809+0000","last_clean":"2024-03-26T04:07:37.834809+0000","last_became_active":"2024-03-26T04:01:35.795965+0000","last_became_peered":"2024-03-26T04:01:35.795965+0000","last_unstale":"2024-03-26T04:07:37.834809+0000","last_undegraded":"2024-03-26T04:07:37.834809+0000","last_fullsized":"2024-03-26T04:07:37.834809+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'29","last_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:04:59.217868+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":710,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835079+0000","last_change":"2024-03-26T04:05:29.536479+0000","last_active":"2024-03-26T04:07:37.835079+0000","last_peered":"2024-03-26T04:07:37.835079+0000","last_clean":"2024-03-26T04:07:37.835079+0000","last_became_active":"2024-03-26T04:05:29.536165+0000","last_became_peered":"2024-03-26T04:05:29.536165+0000","last_unstale":"2024-03-26T04:07:37.835079+0000","last_undegraded":"2024-03-26T04:07:37.835079+0000","last_fullsized":"2024-03-26T04:07:37.835079+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.247532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.247532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"421'21544","reported_seq":21701,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833499+0000","last_change":"2024-03-26T04:03:47.644681+0000","last_active":"2024-03-26T04:07:37.833499+0000","last_peered":"2024-03-26T04:07:37.833499+0000","last_clean":"2024-03-26T04:07:37.833499+0000","last_became_active":"2024-03-26T04:03:47.644384+0000","last_became_peered":"2024-03-26T04:03:47.644384+0000","last_unstale":"2024-03-26T04:07:37.833499+0000","last_undegraded":"2024-03-26T04:07:37.833499+0000","last_fullsized":"2024-03-26T04:07:37.833499+0000","mapping_epoch":349,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3144,"log_dups_size":0,"ondisk_log_size":3144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21544,"num_write_kb":689408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":506,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702360+0000","last_change":"2024-03-26T04:04:02.403906+0000","last_active":"2024-03-26T04:07:37.702360+0000","last_peered":"2024-03-26T04:07:37.702360+0000","last_clean":"2024-03-26T04:07:37.702360+0000","last_became_active":"2024-03-26T04:02:52.504066+0000","last_became_peered":"2024-03-26T04:02:52.504066+0000","last_unstale":"2024-03-26T04:07:37.702360+0000","last_undegraded":"2024-03-26T04:07:37.702360+0000","last_fullsized":"2024-03-26T04:07:37.702360+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:04:02.403784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1068,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832325+0000","last_change":"2024-03-26T04:06:03.860318+0000","last_active":"2024-03-26T04:07:37.832325+0000","last_peered":"2024-03-26T04:07:37.832325+0000","last_clean":"2024-03-26T04:07:37.832325+0000","last_became_active":"2024-03-26T04:00:27.442487+0000","last_became_peered":"2024-03-26T04:00:27.442487+0000","last_unstale":"2024-03-26T04:07:37.832325+0000","last_undegraded":"2024-03-26T04:07:37.832325+0000","last_fullsized":"2024-03-26T04:07:37.832325+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:06:03.860237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:06:03.860237+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":400,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700854+0000","last_change":"2024-03-26T04:04:56.545589+0000","last_active":"2024-03-26T04:07:37.700854+0000","last_peered":"2024-03-26T04:07:37.700854+0000","last_clean":"2024-03-26T04:07:37.700854+0000","last_became_active":"2024-03-26T04:02:52.499614+0000","last_became_peered":"2024-03-26T04:02:52.499614+0000","last_unstale":"2024-03-26T04:07:37.700854+0000","last_undegraded":"2024-03-26T04:07:37.700854+0000","last_fullsized":"2024-03-26T04:07:37.700854+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:56.545464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:56.545464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":986,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.304825+0000","last_change":"2024-03-26T04:04:20.432746+0000","last_active":"2024-03-26T04:07:44.304825+0000","last_peered":"2024-03-26T04:07:44.304825+0000","last_clean":"2024-03-26T04:07:44.304825+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:07:44.304825+0000","last_undegraded":"2024-03-26T04:07:44.304825+0000","last_fullsized":"2024-03-26T04:07:44.304825+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:04:20.432551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:20.432551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":592,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834087+0000","last_change":"2024-03-26T04:05:29.547970+0000","last_active":"2024-03-26T04:07:37.834087+0000","last_peered":"2024-03-26T04:07:37.834087+0000","last_clean":"2024-03-26T04:07:37.834087+0000","last_became_active":"2024-03-26T04:05:29.544566+0000","last_became_peered":"2024-03-26T04:05:29.544566+0000","last_unstale":"2024-03-26T04:07:37.834087+0000","last_undegraded":"2024-03-26T04:07:37.834087+0000","last_fullsized":"2024-03-26T04:07:37.834087+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.381181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.381181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.8","version":"421'21530","reported_seq":26673,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696654+0000","last_change":"2024-03-26T04:07:19.157035+0000","last_active":"2024-03-26T04:07:37.696654+0000","last_peered":"2024-03-26T04:07:37.696654+0000","last_clean":"2024-03-26T04:07:37.696654+0000","last_became_active":"2024-03-26T04:07:19.140559+0000","last_became_peered":"2024-03-26T04:07:19.140559+0000","last_unstale":"2024-03-26T04:07:37.696654+0000","last_undegraded":"2024-03-26T04:07:37.696654+0000","last_fullsized":"2024-03-26T04:07:37.696654+0000","mapping_epoch":414,"log_start":"404'18400","ondisk_log_start":"404'18400","created":349,"last_epoch_clean":415,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3130,"log_dups_size":0,"ondisk_log_size":3130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21530,"num_write_kb":688960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2487,"num_bytes_recovered":162856960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":668,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833686+0000","last_change":"2024-03-26T04:04:07.285956+0000","last_active":"2024-03-26T04:07:37.833686+0000","last_peered":"2024-03-26T04:07:37.833686+0000","last_clean":"2024-03-26T04:07:37.833686+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:07:37.833686+0000","last_undegraded":"2024-03-26T04:07:37.833686+0000","last_fullsized":"2024-03-26T04:07:37.833686+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:07.285700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:07.285700+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":792,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833921+0000","last_change":"2024-03-26T04:05:29.522174+0000","last_active":"2024-03-26T04:07:37.833921+0000","last_peered":"2024-03-26T04:07:37.833921+0000","last_clean":"2024-03-26T04:07:37.833921+0000","last_became_active":"2024-03-26T04:05:29.521425+0000","last_became_peered":"2024-03-26T04:05:29.521425+0000","last_unstale":"2024-03-26T04:07:37.833921+0000","last_undegraded":"2024-03-26T04:07:37.833921+0000","last_fullsized":"2024-03-26T04:07:37.833921+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.552445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.552445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":807,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696917+0000","last_change":"2024-03-26T04:03:56.329590+0000","last_active":"2024-03-26T04:07:37.696917+0000","last_peered":"2024-03-26T04:07:37.696917+0000","last_clean":"2024-03-26T04:07:37.696917+0000","last_became_active":"2024-03-26T04:02:52.508294+0000","last_became_peered":"2024-03-26T04:02:52.508294+0000","last_unstale":"2024-03-26T04:07:37.696917+0000","last_undegraded":"2024-03-26T04:07:37.696917+0000","last_fullsized":"2024-03-26T04:07:37.696917+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:56.329511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:56.329511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1951,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:48.306616+0000","last_change":"2024-03-26T04:07:31.850944+0000","last_active":"2024-03-26T04:07:48.306616+0000","last_peered":"2024-03-26T04:07:48.306616+0000","last_clean":"2024-03-26T04:07:48.306616+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:07:48.306616+0000","last_undegraded":"2024-03-26T04:07:48.306616+0000","last_fullsized":"2024-03-26T04:07:48.306616+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:07:31.850790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.850790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.959779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1137,"num_read_kb":1132,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"241'24","reported_seq":804,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697917+0000","last_change":"2024-03-26T04:05:29.542004+0000","last_active":"2024-03-26T04:07:37.697917+0000","last_peered":"2024-03-26T04:07:37.697917+0000","last_clean":"2024-03-26T04:07:37.697917+0000","last_became_active":"2024-03-26T04:05:29.538986+0000","last_became_peered":"2024-03-26T04:05:29.538986+0000","last_unstale":"2024-03-26T04:07:37.697917+0000","last_undegraded":"2024-03-26T04:07:37.697917+0000","last_fullsized":"2024-03-26T04:07:37.697917+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:00:18.345305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:00:18.345305+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":484,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.835102+0000","last_change":"2024-03-26T04:05:01.160620+0000","last_active":"2024-03-26T04:07:37.835102+0000","last_peered":"2024-03-26T04:07:37.835102+0000","last_clean":"2024-03-26T04:07:37.835102+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:07:37.835102+0000","last_undegraded":"2024-03-26T04:07:37.835102+0000","last_fullsized":"2024-03-26T04:07:37.835102+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:05:01.160427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"421'24449","reported_seq":24809,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702239+0000","last_change":"2024-03-26T04:05:23.910568+0000","last_active":"2024-03-26T04:07:37.702239+0000","last_peered":"2024-03-26T04:07:37.702239+0000","last_clean":"2024-03-26T04:07:37.702239+0000","last_became_active":"2024-03-26T04:03:47.643956+0000","last_became_peered":"2024-03-26T04:03:47.643956+0000","last_unstale":"2024-03-26T04:07:37.702239+0000","last_undegraded":"2024-03-26T04:07:37.702239+0000","last_fullsized":"2024-03-26T04:07:37.702239+0000","mapping_epoch":349,"log_start":"404'21403","ondisk_log_start":"404'21403","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":4910,"log_size":3046,"log_dups_size":3000,"ondisk_log_size":3046,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":21172,"num_write_kb":677441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":623,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697659+0000","last_change":"2024-03-26T04:04:53.785962+0000","last_active":"2024-03-26T04:07:37.697659+0000","last_peered":"2024-03-26T04:07:37.697659+0000","last_clean":"2024-03-26T04:07:37.697659+0000","last_became_active":"2024-03-26T04:02:55.535259+0000","last_became_peered":"2024-03-26T04:02:55.535259+0000","last_unstale":"2024-03-26T04:07:37.697659+0000","last_undegraded":"2024-03-26T04:07:37.697659+0000","last_fullsized":"2024-03-26T04:07:37.697659+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:53.785762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:53.785762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":781,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:41.426677+0000","last_change":"2024-03-26T04:07:41.426704+0000","last_active":"2024-03-26T04:07:41.426677+0000","last_peered":"2024-03-26T04:07:41.426677+0000","last_clean":"2024-03-26T04:07:41.426677+0000","last_became_active":"2024-03-26T04:05:29.540971+0000","last_became_peered":"2024-03-26T04:05:29.540971+0000","last_unstale":"2024-03-26T04:07:41.426677+0000","last_undegraded":"2024-03-26T04:07:41.426677+0000","last_fullsized":"2024-03-26T04:07:41.426677+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:41.426604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:07:41.426604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:57.590188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":802,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:45.465243+0000","last_change":"2024-03-26T04:07:45.465265+0000","last_active":"2024-03-26T04:07:45.465243+0000","last_peered":"2024-03-26T04:07:45.465243+0000","last_clean":"2024-03-26T04:07:45.465243+0000","last_became_active":"2024-03-26T04:05:29.532029+0000","last_became_peered":"2024-03-26T04:05:29.532029+0000","last_unstale":"2024-03-26T04:07:45.465243+0000","last_undegraded":"2024-03-26T04:07:45.465243+0000","last_fullsized":"2024-03-26T04:07:45.465243+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.465179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.465179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:00.714731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"358'44","reported_seq":883,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697980+0000","last_change":"2024-03-26T04:04:19.574250+0000","last_active":"2024-03-26T04:07:37.697980+0000","last_peered":"2024-03-26T04:07:37.697980+0000","last_clean":"2024-03-26T04:07:37.697980+0000","last_became_active":"2024-03-26T04:01:35.824185+0000","last_became_peered":"2024-03-26T04:01:35.824185+0000","last_unstale":"2024-03-26T04:07:37.697980+0000","last_undegraded":"2024-03-26T04:07:37.697980+0000","last_fullsized":"2024-03-26T04:07:37.697980+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.571532+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":726,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697291+0000","last_change":"2024-03-26T04:05:29.537174+0000","last_active":"2024-03-26T04:07:37.697291+0000","last_peered":"2024-03-26T04:07:37.697291+0000","last_clean":"2024-03-26T04:07:37.697291+0000","last_became_active":"2024-03-26T04:05:29.536647+0000","last_became_peered":"2024-03-26T04:05:29.536647+0000","last_unstale":"2024-03-26T04:07:37.697291+0000","last_undegraded":"2024-03-26T04:07:37.697291+0000","last_fullsized":"2024-03-26T04:07:37.697291+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.058039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.058039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.c","version":"421'21270","reported_seq":21597,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833110+0000","last_change":"2024-03-26T04:06:09.371173+0000","last_active":"2024-03-26T04:07:37.833110+0000","last_peered":"2024-03-26T04:07:37.833110+0000","last_clean":"2024-03-26T04:07:37.833110+0000","last_became_active":"2024-03-26T04:03:47.646236+0000","last_became_peered":"2024-03-26T04:03:47.646236+0000","last_unstale":"2024-03-26T04:07:37.833110+0000","last_undegraded":"2024-03-26T04:07:37.833110+0000","last_fullsized":"2024-03-26T04:07:37.833110+0000","mapping_epoch":349,"log_start":"404'18100","ondisk_log_start":"404'18100","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"389'14177","last_scrub_stamp":"2024-03-26T04:06:09.370906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:06:09.370906+0000","objects_scrubbed":7160,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1528,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21270,"num_write_kb":680640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":469,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834907+0000","last_change":"2024-03-26T04:07:29.526841+0000","last_active":"2024-03-26T04:07:37.834907+0000","last_peered":"2024-03-26T04:07:37.834907+0000","last_clean":"2024-03-26T04:07:37.834907+0000","last_became_active":"2024-03-26T04:05:29.536837+0000","last_became_peered":"2024-03-26T04:05:29.536837+0000","last_unstale":"2024-03-26T04:07:37.834907+0000","last_undegraded":"2024-03-26T04:07:37.834907+0000","last_fullsized":"2024-03-26T04:07:37.834907+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:29.526591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.526591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:46.626775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":789,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.696507+0000","last_change":"2024-03-26T04:05:29.539255+0000","last_active":"2024-03-26T04:07:37.696507+0000","last_peered":"2024-03-26T04:07:37.696507+0000","last_clean":"2024-03-26T04:07:37.696507+0000","last_became_active":"2024-03-26T04:05:29.538900+0000","last_became_peered":"2024-03-26T04:05:29.538900+0000","last_unstale":"2024-03-26T04:07:37.696507+0000","last_undegraded":"2024-03-26T04:07:37.696507+0000","last_fullsized":"2024-03-26T04:07:37.696507+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:10.378037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:10.378037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":431,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834653+0000","last_change":"2024-03-26T04:05:29.544209+0000","last_active":"2024-03-26T04:07:37.834653+0000","last_peered":"2024-03-26T04:07:37.834653+0000","last_clean":"2024-03-26T04:07:37.834653+0000","last_became_active":"2024-03-26T04:05:29.542761+0000","last_became_peered":"2024-03-26T04:05:29.542761+0000","last_unstale":"2024-03-26T04:07:37.834653+0000","last_undegraded":"2024-03-26T04:07:37.834653+0000","last_fullsized":"2024-03-26T04:07:37.834653+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:03.918739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:03.918739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":845,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833832+0000","last_change":"2024-03-26T04:04:12.249958+0000","last_active":"2024-03-26T04:07:37.833832+0000","last_peered":"2024-03-26T04:07:37.833832+0000","last_clean":"2024-03-26T04:07:37.833832+0000","last_became_active":"2024-03-26T04:01:35.819144+0000","last_became_peered":"2024-03-26T04:01:35.819144+0000","last_unstale":"2024-03-26T04:07:37.833832+0000","last_undegraded":"2024-03-26T04:07:37.833832+0000","last_fullsized":"2024-03-26T04:07:37.833832+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:04:12.249693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:12.249693+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":1055,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.303624+0000","last_change":"2024-03-26T04:02:52.536740+0000","last_active":"2024-03-26T04:07:39.303624+0000","last_peered":"2024-03-26T04:07:39.303624+0000","last_clean":"2024-03-26T04:07:39.303624+0000","last_became_active":"2024-03-26T04:02:52.495925+0000","last_became_peered":"2024-03-26T04:02:52.495925+0000","last_unstale":"2024-03-26T04:07:39.303624+0000","last_undegraded":"2024-03-26T04:07:39.303624+0000","last_fullsized":"2024-03-26T04:07:39.303624+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-03-26T04:02:46.076593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:46.076593+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"421'21498","reported_seq":37029,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832197+0000","last_change":"2024-03-26T04:06:33.714049+0000","last_active":"2024-03-26T04:07:37.832197+0000","last_peered":"2024-03-26T04:07:37.832197+0000","last_clean":"2024-03-26T04:07:37.832197+0000","last_became_active":"2024-03-26T04:06:33.689489+0000","last_became_peered":"2024-03-26T04:06:33.689489+0000","last_unstale":"2024-03-26T04:07:37.832197+0000","last_undegraded":"2024-03-26T04:07:37.832197+0000","last_fullsized":"2024-03-26T04:07:37.832197+0000","mapping_epoch":397,"log_start":"400'17800","ondisk_log_start":"400'17800","created":349,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3698,"log_dups_size":0,"ondisk_log_size":3698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21498,"num_write_kb":687936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7684,"num_bytes_recovered":503578624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":977,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832986+0000","last_change":"2024-03-26T04:07:26.838873+0000","last_active":"2024-03-26T04:07:37.832986+0000","last_peered":"2024-03-26T04:07:37.832986+0000","last_clean":"2024-03-26T04:07:37.832986+0000","last_became_active":"2024-03-26T04:02:55.533229+0000","last_became_peered":"2024-03-26T04:02:55.533229+0000","last_unstale":"2024-03-26T04:07:37.832986+0000","last_undegraded":"2024-03-26T04:07:37.832986+0000","last_fullsized":"2024-03-26T04:07:37.832986+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:26.838741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.838741+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.279852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":797,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:40.043010+0000","last_change":"2024-03-26T04:07:40.043033+0000","last_active":"2024-03-26T04:07:40.043010+0000","last_peered":"2024-03-26T04:07:40.043010+0000","last_clean":"2024-03-26T04:07:40.043010+0000","last_became_active":"2024-03-26T04:05:29.542416+0000","last_became_peered":"2024-03-26T04:05:29.542416+0000","last_unstale":"2024-03-26T04:07:40.043010+0000","last_undegraded":"2024-03-26T04:07:40.043010+0000","last_fullsized":"2024-03-26T04:07:40.043010+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:40.042936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:07:40.042936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.886296+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":781,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.697960+0000","last_change":"2024-03-26T04:05:07.862785+0000","last_active":"2024-03-26T04:07:37.697960+0000","last_peered":"2024-03-26T04:07:37.697960+0000","last_clean":"2024-03-26T04:07:37.697960+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:07:37.697960+0000","last_undegraded":"2024-03-26T04:07:37.697960+0000","last_fullsized":"2024-03-26T04:07:37.697960+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:05:07.862659+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.862659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":879,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701258+0000","last_change":"2024-03-26T04:04:16.357551+0000","last_active":"2024-03-26T04:07:37.701258+0000","last_peered":"2024-03-26T04:07:37.701258+0000","last_clean":"2024-03-26T04:07:37.701258+0000","last_became_active":"2024-03-26T04:01:35.799996+0000","last_became_peered":"2024-03-26T04:01:35.799996+0000","last_unstale":"2024-03-26T04:07:37.701258+0000","last_undegraded":"2024-03-26T04:07:37.701258+0000","last_fullsized":"2024-03-26T04:07:37.701258+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:04:16.357311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:16.357311+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":489,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.303740+0000","last_change":"2024-03-26T04:03:51.382658+0000","last_active":"2024-03-26T04:07:39.303740+0000","last_peered":"2024-03-26T04:07:39.303740+0000","last_clean":"2024-03-26T04:07:39.303740+0000","last_became_active":"2024-03-26T04:02:52.515390+0000","last_became_peered":"2024-03-26T04:02:52.515390+0000","last_unstale":"2024-03-26T04:07:39.303740+0000","last_undegraded":"2024-03-26T04:07:39.303740+0000","last_fullsized":"2024-03-26T04:07:39.303740+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:51.382563+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:03:51.382563+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.e","version":"421'21472","reported_seq":28780,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702169+0000","last_change":"2024-03-26T04:07:04.052128+0000","last_active":"2024-03-26T04:07:37.702169+0000","last_peered":"2024-03-26T04:07:37.702169+0000","last_clean":"2024-03-26T04:07:37.702169+0000","last_became_active":"2024-03-26T04:07:04.035244+0000","last_became_peered":"2024-03-26T04:07:04.035244+0000","last_unstale":"2024-03-26T04:07:37.702169+0000","last_undegraded":"2024-03-26T04:07:37.702169+0000","last_fullsized":"2024-03-26T04:07:37.702169+0000","mapping_epoch":406,"log_start":"404'18500","ondisk_log_start":"404'18500","created":349,"last_epoch_clean":407,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":2972,"log_dups_size":3000,"ondisk_log_size":2972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21472,"num_write_kb":687104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3569,"num_bytes_recovered":233897984,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":627,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:39.474278+0000","last_change":"2024-03-26T04:07:39.474303+0000","last_active":"2024-03-26T04:07:39.474278+0000","last_peered":"2024-03-26T04:07:39.474278+0000","last_clean":"2024-03-26T04:07:39.474278+0000","last_became_active":"2024-03-26T04:05:29.521761+0000","last_became_peered":"2024-03-26T04:05:29.521761+0000","last_unstale":"2024-03-26T04:07:39.474278+0000","last_undegraded":"2024-03-26T04:07:39.474278+0000","last_fullsized":"2024-03-26T04:07:39.474278+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:39.474219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.474219+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:43.579453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":779,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701129+0000","last_change":"2024-03-26T04:07:30.569585+0000","last_active":"2024-03-26T04:07:37.701129+0000","last_peered":"2024-03-26T04:07:37.701129+0000","last_clean":"2024-03-26T04:07:37.701129+0000","last_became_active":"2024-03-26T04:05:29.532854+0000","last_became_peered":"2024-03-26T04:05:29.532854+0000","last_unstale":"2024-03-26T04:07:37.701129+0000","last_undegraded":"2024-03-26T04:07:37.701129+0000","last_fullsized":"2024-03-26T04:07:37.701129+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.569498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.569498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.566912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":284,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698375+0000","last_change":"2024-03-26T04:03:06.122009+0000","last_active":"2024-03-26T04:07:37.698375+0000","last_peered":"2024-03-26T04:07:37.698375+0000","last_clean":"2024-03-26T04:07:37.698375+0000","last_became_active":"2024-03-26T04:02:52.496389+0000","last_became_peered":"2024-03-26T04:02:52.496389+0000","last_unstale":"2024-03-26T04:07:37.698375+0000","last_undegraded":"2024-03-26T04:07:37.698375+0000","last_fullsized":"2024-03-26T04:07:37.698375+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:06.121917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:06.121917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":355,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832626+0000","last_change":"2024-03-26T04:07:35.968314+0000","last_active":"2024-03-26T04:07:37.832626+0000","last_peered":"2024-03-26T04:07:37.832626+0000","last_clean":"2024-03-26T04:07:37.832626+0000","last_became_active":"2024-03-26T04:02:52.511833+0000","last_became_peered":"2024-03-26T04:02:52.511833+0000","last_unstale":"2024-03-26T04:07:37.832626+0000","last_undegraded":"2024-03-26T04:07:37.832626+0000","last_fullsized":"2024-03-26T04:07:37.832626+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:07:35.968179+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:07:35.968179+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:00.483729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":782,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701055+0000","last_change":"2024-03-26T04:04:06.297839+0000","last_active":"2024-03-26T04:07:37.701055+0000","last_peered":"2024-03-26T04:07:37.701055+0000","last_clean":"2024-03-26T04:07:37.701055+0000","last_became_active":"2024-03-26T04:02:52.498870+0000","last_became_peered":"2024-03-26T04:02:52.498870+0000","last_unstale":"2024-03-26T04:07:37.701055+0000","last_undegraded":"2024-03-26T04:07:37.701055+0000","last_fullsized":"2024-03-26T04:07:37.701055+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.297701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"421'21798","reported_seq":21955,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.700718+0000","last_change":"2024-03-26T04:03:47.644772+0000","last_active":"2024-03-26T04:07:37.700718+0000","last_peered":"2024-03-26T04:07:37.700718+0000","last_clean":"2024-03-26T04:07:37.700718+0000","last_became_active":"2024-03-26T04:03:47.644103+0000","last_became_peered":"2024-03-26T04:03:47.644103+0000","last_unstale":"2024-03-26T04:07:37.700718+0000","last_undegraded":"2024-03-26T04:07:37.700718+0000","last_fullsized":"2024-03-26T04:07:37.700718+0000","mapping_epoch":349,"log_start":"404'18800","ondisk_log_start":"404'18800","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":2998,"log_dups_size":3000,"ondisk_log_size":2998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21798,"num_write_kb":697536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":464,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833151+0000","last_change":"2024-03-26T04:07:22.530369+0000","last_active":"2024-03-26T04:07:37.833151+0000","last_peered":"2024-03-26T04:07:37.833151+0000","last_clean":"2024-03-26T04:07:37.833151+0000","last_became_active":"2024-03-26T04:05:29.524691+0000","last_became_peered":"2024-03-26T04:05:29.524691+0000","last_unstale":"2024-03-26T04:07:37.833151+0000","last_undegraded":"2024-03-26T04:07:37.833151+0000","last_fullsized":"2024-03-26T04:07:37.833151+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:22.530143+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:07:22.530143+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:22.608960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":455,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834245+0000","last_change":"2024-03-26T04:05:29.532951+0000","last_active":"2024-03-26T04:07:37.834245+0000","last_peered":"2024-03-26T04:07:37.834245+0000","last_clean":"2024-03-26T04:07:37.834245+0000","last_became_active":"2024-03-26T04:05:29.532692+0000","last_became_peered":"2024-03-26T04:05:29.532692+0000","last_unstale":"2024-03-26T04:07:37.834245+0000","last_undegraded":"2024-03-26T04:07:37.834245+0000","last_fullsized":"2024-03-26T04:07:37.834245+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:25.365447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.365447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"379'13","reported_seq":1026,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:44.304779+0000","last_change":"2024-03-26T04:05:29.523046+0000","last_active":"2024-03-26T04:07:44.304779+0000","last_peered":"2024-03-26T04:07:44.304779+0000","last_clean":"2024-03-26T04:07:44.304779+0000","last_became_active":"2024-03-26T04:05:29.521835+0000","last_became_peered":"2024-03-26T04:05:29.521835+0000","last_unstale":"2024-03-26T04:07:44.304779+0000","last_undegraded":"2024-03-26T04:07:44.304779+0000","last_fullsized":"2024-03-26T04:07:44.304779+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:00:22.364212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.364212+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":356,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832856+0000","last_change":"2024-03-26T04:03:50.413181+0000","last_active":"2024-03-26T04:07:37.832856+0000","last_peered":"2024-03-26T04:07:37.832856+0000","last_clean":"2024-03-26T04:07:37.832856+0000","last_became_active":"2024-03-26T04:02:52.511800+0000","last_became_peered":"2024-03-26T04:02:52.511800+0000","last_unstale":"2024-03-26T04:07:37.832856+0000","last_undegraded":"2024-03-26T04:07:37.832856+0000","last_fullsized":"2024-03-26T04:07:37.832856+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:50.413099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:03:50.413099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":882,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834836+0000","last_change":"2024-03-26T04:04:06.282172+0000","last_active":"2024-03-26T04:07:37.834836+0000","last_peered":"2024-03-26T04:07:37.834836+0000","last_clean":"2024-03-26T04:07:37.834836+0000","last_became_active":"2024-03-26T04:02:52.513721+0000","last_became_peered":"2024-03-26T04:02:52.513721+0000","last_unstale":"2024-03-26T04:07:37.834836+0000","last_undegraded":"2024-03-26T04:07:37.834836+0000","last_fullsized":"2024-03-26T04:07:37.834836+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:04:06.281970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.281970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"421'21412","reported_seq":21665,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832313+0000","last_change":"2024-03-26T04:05:07.668084+0000","last_active":"2024-03-26T04:07:37.832313+0000","last_peered":"2024-03-26T04:07:37.832313+0000","last_clean":"2024-03-26T04:07:37.832313+0000","last_became_active":"2024-03-26T04:03:47.643599+0000","last_became_peered":"2024-03-26T04:03:47.643599+0000","last_unstale":"2024-03-26T04:07:37.832313+0000","last_undegraded":"2024-03-26T04:07:37.832313+0000","last_fullsized":"2024-03-26T04:07:37.832313+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'8783","last_scrub_stamp":"2024-03-26T04:05:07.667791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.667791+0000","objects_scrubbed":8667,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21412,"num_write_kb":685184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":398,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833017+0000","last_change":"2024-03-26T04:06:00.896785+0000","last_active":"2024-03-26T04:07:37.833017+0000","last_peered":"2024-03-26T04:07:37.833017+0000","last_clean":"2024-03-26T04:07:37.833017+0000","last_became_active":"2024-03-26T04:02:52.506309+0000","last_became_peered":"2024-03-26T04:02:52.506309+0000","last_unstale":"2024-03-26T04:07:37.833017+0000","last_undegraded":"2024-03-26T04:07:37.833017+0000","last_fullsized":"2024-03-26T04:07:37.833017+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:00.896583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:00.896583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":505,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702939+0000","last_change":"2024-03-26T04:07:34.478534+0000","last_active":"2024-03-26T04:07:37.702939+0000","last_peered":"2024-03-26T04:07:37.702939+0000","last_clean":"2024-03-26T04:07:37.702939+0000","last_became_active":"2024-03-26T04:05:29.520457+0000","last_became_peered":"2024-03-26T04:05:29.520457+0000","last_unstale":"2024-03-26T04:07:37.702939+0000","last_undegraded":"2024-03-26T04:07:37.702939+0000","last_fullsized":"2024-03-26T04:07:37.702939+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.478427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.478427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:01.934278+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":780,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:45.504593+0000","last_change":"2024-03-26T04:07:45.504621+0000","last_active":"2024-03-26T04:07:45.504593+0000","last_peered":"2024-03-26T04:07:45.504593+0000","last_clean":"2024-03-26T04:07:45.504593+0000","last_became_active":"2024-03-26T04:05:29.536797+0000","last_became_peered":"2024-03-26T04:05:29.536797+0000","last_unstale":"2024-03-26T04:07:45.504593+0000","last_undegraded":"2024-03-26T04:07:45.504593+0000","last_fullsized":"2024-03-26T04:07:45.504593+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.504517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.504517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:07.030754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":907,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702545+0000","last_change":"2024-03-26T04:05:06.619430+0000","last_active":"2024-03-26T04:07:37.702545+0000","last_peered":"2024-03-26T04:07:37.702545+0000","last_clean":"2024-03-26T04:07:37.702545+0000","last_became_active":"2024-03-26T04:02:52.520710+0000","last_became_peered":"2024-03-26T04:02:52.520710+0000","last_unstale":"2024-03-26T04:07:37.702545+0000","last_undegraded":"2024-03-26T04:07:37.702545+0000","last_fullsized":"2024-03-26T04:07:37.702545+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T04:05:06.619261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:05:06.619261+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"421'21486","reported_seq":35515,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832874+0000","last_change":"2024-03-26T04:06:45.558381+0000","last_active":"2024-03-26T04:07:37.832874+0000","last_peered":"2024-03-26T04:07:37.832874+0000","last_clean":"2024-03-26T04:07:37.832874+0000","last_became_active":"2024-03-26T04:06:45.534824+0000","last_became_peered":"2024-03-26T04:06:45.534824+0000","last_unstale":"2024-03-26T04:07:37.832874+0000","last_undegraded":"2024-03-26T04:07:37.832874+0000","last_fullsized":"2024-03-26T04:07:37.832874+0000","mapping_epoch":399,"log_start":"400'17800","ondisk_log_start":"400'17800","created":349,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":0,"log_size":3686,"log_dups_size":0,"ondisk_log_size":3686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21486,"num_write_kb":687552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6933,"num_bytes_recovered":454361088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":492,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.832395+0000","last_change":"2024-03-26T04:05:29.533929+0000","last_active":"2024-03-26T04:07:37.832395+0000","last_peered":"2024-03-26T04:07:37.832395+0000","last_clean":"2024-03-26T04:07:37.832395+0000","last_became_active":"2024-03-26T04:05:29.533470+0000","last_became_peered":"2024-03-26T04:05:29.533470+0000","last_unstale":"2024-03-26T04:07:37.832395+0000","last_undegraded":"2024-03-26T04:07:37.832395+0000","last_fullsized":"2024-03-26T04:07:37.832395+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:39.967821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:39.967821+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":703,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834628+0000","last_change":"2024-03-26T04:05:29.522502+0000","last_active":"2024-03-26T04:07:37.834628+0000","last_peered":"2024-03-26T04:07:37.834628+0000","last_clean":"2024-03-26T04:07:37.834628+0000","last_became_active":"2024-03-26T04:05:29.521089+0000","last_became_peered":"2024-03-26T04:05:29.521089+0000","last_unstale":"2024-03-26T04:07:37.834628+0000","last_undegraded":"2024-03-26T04:07:37.834628+0000","last_fullsized":"2024-03-26T04:07:37.834628+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_clean_scrub_stamp":"2024-03-26T04:02:49.166761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":769,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.834724+0000","last_change":"2024-03-26T04:04:49.184140+0000","last_active":"2024-03-26T04:07:37.834724+0000","last_peered":"2024-03-26T04:07:37.834724+0000","last_clean":"2024-03-26T04:07:37.834724+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:07:37.834724+0000","last_undegraded":"2024-03-26T04:07:37.834724+0000","last_fullsized":"2024-03-26T04:07:37.834724+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:49.184018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:04:49.184018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":736,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.701199+0000","last_change":"2024-03-26T04:05:08.666190+0000","last_active":"2024-03-26T04:07:37.701199+0000","last_peered":"2024-03-26T04:07:37.701199+0000","last_clean":"2024-03-26T04:07:37.701199+0000","last_became_active":"2024-03-26T04:02:52.505213+0000","last_became_peered":"2024-03-26T04:02:52.505213+0000","last_unstale":"2024-03-26T04:07:37.701199+0000","last_undegraded":"2024-03-26T04:07:37.701199+0000","last_fullsized":"2024-03-26T04:07:37.701199+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:08.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:08.666106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":825,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702937+0000","last_change":"2024-03-26T04:07:25.580036+0000","last_active":"2024-03-26T04:07:37.702937+0000","last_peered":"2024-03-26T04:07:37.702937+0000","last_clean":"2024-03-26T04:07:37.702937+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:07:37.702937+0000","last_undegraded":"2024-03-26T04:07:37.702937+0000","last_fullsized":"2024-03-26T04:07:37.702937+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:07:25.579888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.579888+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.986075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"421'21442","reported_seq":21689,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.833321+0000","last_change":"2024-03-26T04:05:10.785305+0000","last_active":"2024-03-26T04:07:37.833321+0000","last_peered":"2024-03-26T04:07:37.833321+0000","last_clean":"2024-03-26T04:07:37.833321+0000","last_became_active":"2024-03-26T04:03:47.643488+0000","last_became_peered":"2024-03-26T04:03:47.643488+0000","last_unstale":"2024-03-26T04:07:37.833321+0000","last_undegraded":"2024-03-26T04:07:37.833321+0000","last_fullsized":"2024-03-26T04:07:37.833321+0000","mapping_epoch":349,"log_start":"404'18300","ondisk_log_start":"404'18300","created":349,"last_epoch_clean":350,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:03:46.622304+0000","last_clean_scrub_stamp":"2024-03-26T04:03:46.622304+0000","objects_scrubbed":8599,"log_size":3142,"log_dups_size":0,"ondisk_log_size":3142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21442,"num_write_kb":686144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1093,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:48.931637+0000","last_change":"2024-03-26T04:07:48.931663+0000","last_active":"2024-03-26T04:07:48.931637+0000","last_peered":"2024-03-26T04:07:48.931637+0000","last_clean":"2024-03-26T04:07:48.931637+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:07:48.931637+0000","last_undegraded":"2024-03-26T04:07:48.931637+0000","last_fullsized":"2024-03-26T04:07:48.931637+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.931565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.287206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":768,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702970+0000","last_change":"2024-03-26T04:07:27.607414+0000","last_active":"2024-03-26T04:07:37.702970+0000","last_peered":"2024-03-26T04:07:37.702970+0000","last_clean":"2024-03-26T04:07:37.702970+0000","last_became_active":"2024-03-26T04:05:29.520602+0000","last_became_peered":"2024-03-26T04:05:29.520602+0000","last_unstale":"2024-03-26T04:07:37.702970+0000","last_undegraded":"2024-03-26T04:07:37.702970+0000","last_fullsized":"2024-03-26T04:07:37.702970+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:27.607316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:07:27.607316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:35.557513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"379'13","reported_seq":935,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:43.094822+0000","last_change":"2024-03-26T04:05:29.544439+0000","last_active":"2024-03-26T04:07:43.094822+0000","last_peered":"2024-03-26T04:07:43.094822+0000","last_clean":"2024-03-26T04:07:43.094822+0000","last_became_active":"2024-03-26T04:05:29.544028+0000","last_became_peered":"2024-03-26T04:05:29.544028+0000","last_unstale":"2024-03-26T04:07:43.094822+0000","last_undegraded":"2024-03-26T04:07:43.094822+0000","last_fullsized":"2024-03-26T04:07:43.094822+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:01:34.236967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.236967+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":356,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.698104+0000","last_change":"2024-03-26T04:04:06.376407+0000","last_active":"2024-03-26T04:07:37.698104+0000","last_peered":"2024-03-26T04:07:37.698104+0000","last_clean":"2024-03-26T04:07:37.698104+0000","last_became_active":"2024-03-26T04:02:52.491417+0000","last_became_peered":"2024-03-26T04:02:52.491417+0000","last_unstale":"2024-03-26T04:07:37.698104+0000","last_undegraded":"2024-03-26T04:07:37.698104+0000","last_fullsized":"2024-03-26T04:07:37.698104+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.376284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.376284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":869,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-03-26T04:07:37.702014+0000","last_change":"2024-03-26T04:04:18.408633+0000","last_active":"2024-03-26T04:07:37.702014+0000","last_peered":"2024-03-26T04:07:37.702014+0000","last_clean":"2024-03-26T04:07:37.702014+0000","last_became_active":"2024-03-26T04:02:52.504477+0000","last_became_peered":"2024-03-26T04:02:52.504477+0000","last_unstale":"2024-03-26T04:07:37.702014+0000","last_undegraded":"2024-03-26T04:07:37.702014+0000","last_fullsized":"2024-03-26T04:07:37.702014+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:04:18.408372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.408372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":344186,"num_write_kb":11013889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37712,"num_bytes_recovered":2471297024,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51985,"ondisk_log_size":51985,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1137,"num_read_kb":1132,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":7718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":15688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":955,"ondisk_log_size":955,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070697,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":894400,"kb_used_data":2488,"kb_used_omap":1,"kb_used_meta":891902,"kb_avail":92850752,"statfs":{"total":95995035648,"available":95079170048,"internally_reserved":0,"allocated":2547712,"data_stored":1534036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":913308106},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575111,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":823636,"kb_used_data":2508,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92921516,"statfs":{"total":95995035648,"available":95151632384,"internally_reserved":0,"allocated":2568192,"data_stored":1534443,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738636,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":922108,"kb_used_data":2420,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":92823044,"statfs":{"total":95995035648,"available":95050797056,"internally_reserved":0,"allocated":2478080,"data_stored":1468743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637917,"num_pgs":82,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318168,"kb_used_data":2384,"kb_used_omap":6,"kb_used_meta":315769,"kb_avail":93426984,"statfs":{"total":95995035648,"available":95669231616,"internally_reserved":0,"allocated":2441216,"data_stored":1466667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":323347749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953720,"num_pgs":101,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":954332,"kb_used_data":2708,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92790820,"statfs":{"total":95995035648,"available":95017799680,"internally_reserved":0,"allocated":2772992,"data_stored":1794327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:07:54.544 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T04:07:54.544 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:07:54.544 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-26T04:07:54.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-26T04:07:54.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:54.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:54.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:54.724 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:54.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:54.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:54.854 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[37501]: osdmap e424: 8 total, 8 up, 5 in 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:54.867 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:54 smithi006 ceph-mon[29446]: osdmap e424: 8 total, 8 up, 5 in 2024-03-26T04:07:54.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:54.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:54.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:54.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:55.083 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:07:55.094 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:54 smithi086 ceph-mon[32965]: osdmap e424: 8 total, 8 up, 5 in 2024-03-26T04:07:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:55.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:55.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:55.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:55.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:55.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:55.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:55.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:55.502 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:55.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:55.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:07:55.598 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:07:55.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:55.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:55.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:55.711 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:07:55.711 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:07:55.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:55.755 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:55.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:55.880 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:55.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:55.881 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: pgmap v907: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 427 B/s rd, 0 op/s 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3294604912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:55.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: pgmap v907: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 427 B/s rd, 0 op/s 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3294604912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:55.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:56.006 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:56.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: pgmap v907: 142 pgs: 142 active+clean; 582 KiB data, 3.7 GiB used, 443 GiB / 447 GiB avail; 427 B/s rd, 0 op/s 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3294604912' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:07:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:07:56.108 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:56.131 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:56.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:56.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:56.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:56.317 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:56.318 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:07:56.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:56.383 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:56.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:56.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:56.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:56.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:56.610 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:56.634 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:56.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:56.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.761 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:56.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:56.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:56.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:56.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:56.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: 5.19 scrub starts 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: 5.19 scrub ok 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[37501]: osdmap e425: 8 total, 8 up, 5 in 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:56.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: 5.19 scrub starts 2024-03-26T04:07:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: 5.19 scrub ok 2024-03-26T04:07:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:07:56.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:56 smithi006 ceph-mon[29446]: osdmap e425: 8 total, 8 up, 5 in 2024-03-26T04:07:56.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T04:07:56.954 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: 5.19 scrub starts 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: 5.19 scrub ok 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:07:56.955 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:56 smithi086 ceph-mon[32965]: osdmap e425: 8 total, 8 up, 5 in 2024-03-26T04:07:56.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.016 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:57.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:57.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:57.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:57.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:57.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:57.266 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:57.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:57.315 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:07:57.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:57.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:57.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:57.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:57.517 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:57.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:57.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:57.644 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:57.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:57.770 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:57.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:57.789 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:57 smithi086 ceph-mon[32965]: pgmap v910: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 734 B/s rd, 0 op/s 2024-03-26T04:07:57.789 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:57 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T04:07:57.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:57.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:57 smithi006 ceph-mon[37501]: pgmap v910: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 734 B/s rd, 0 op/s 2024-03-26T04:07:57.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:57 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T04:07:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:57 smithi006 ceph-mon[29446]: pgmap v910: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 734 B/s rd, 0 op/s 2024-03-26T04:07:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:57 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T04:07:57.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:57.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:57.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:58.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.025 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:58.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:58.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:58.150 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:58.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:58.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:58.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:58.309 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:07:58.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:58.401 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:58.503 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:58.526 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:58.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:58.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:58.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:07:58.776 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:58.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:07:58.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:58.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:58.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:58.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:07:59.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:59.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:59.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:59.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:07:59.154 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:59.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:07:59.180 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:07:59.180 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:07:59.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:07:59.279 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:59.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:07:59.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:07:59.405 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:07:59.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:07:59.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:59.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:07:59.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:07:59.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:07:59.758 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:07:59.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T04:07:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:07:59.783 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:59 smithi086 ceph-mon[32965]: pgmap v911: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:07:59.783 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:07:59 smithi086 ceph-mon[32965]: osdmap e426: 8 total, 8 up, 5 in 2024-03-26T04:07:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:07:59.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:07:59.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:59 smithi006 ceph-mon[29446]: pgmap v911: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:07:59.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:07:59 smithi006 ceph-mon[29446]: osdmap e426: 8 total, 8 up, 5 in 2024-03-26T04:07:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:59 smithi006 ceph-mon[37501]: pgmap v911: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:07:59.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:07:59 smithi006 ceph-mon[37501]: osdmap e426: 8 total, 8 up, 5 in 2024-03-26T04:07:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:07:59.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:07:59.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:00.004 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:00.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:00.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:00.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:00.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:00.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:00.286 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:00.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:00.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:00.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:00.536 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:00.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:00.661 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:00.760 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:08:00.763 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:00.780 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:08:00.787 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:00.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:00.800 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:08:00.819 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:08:00.840 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:08:00.861 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:08:00.882 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:08:00.889 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:00.902 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:08:00.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:08:00.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:08:00.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:08:00.904 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:08:00.905 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:08:00.905 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:08:00.905 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:08:00.905 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:08:00.913 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:00.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:00.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:01.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:01.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:01.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:01.141 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:01.142 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:01.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:01.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:01.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.293 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:01.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:01.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:01.417 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:01.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:01.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:01.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:01.624 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_4' created 2024-03-26T04:08:01.635 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:01.635 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:08:01.635 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:08:01.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:08:01.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:01.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:01.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:01.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:01.793 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:01 smithi086 ceph-mon[32965]: pgmap v913: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:08:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4226320231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/550675174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:01 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[37501]: pgmap v913: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:08:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4226320231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:01.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/550675174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[29446]: pgmap v913: 142 pgs: 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:08:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4226320231' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/550675174' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:01 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-26T04:08:01.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:01.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:01.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:02.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:02.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:02.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:02.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:02.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:02.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-26T04:08:02.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:02.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T04:08:02.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:02.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:02.419 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:02.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:02.534 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:02.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:02.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:02.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:02.670 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:02.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:02.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:02.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[37501]: osdmap e427: 8 total, 8 up, 5 in 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T04:08:02.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:02 smithi006 ceph-mon[29446]: osdmap e427: 8 total, 8 up, 5 in 2024-03-26T04:08:02.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:02.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:02 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:02.909 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:02 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:02.910 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:02 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-26T04:08:02.910 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:02 smithi086 ceph-mon[32965]: osdmap e427: 8 total, 8 up, 5 in 2024-03-26T04:08:02.922 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:02.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:02.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "epoch": 428, 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "fsid": "de980954-eb22-11ee-95d0-87774f69a715", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "created": "2024-03-26T03:43:14.534401+0000", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "modified": "2024-03-26T04:08:02.613672+0000", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "last_up_change": "2024-03-26T04:05:28.913716+0000", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "last_in_change": "2024-03-26T04:02:51.443642+0000", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_num": 5801984, 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_set": [ 2024-03-26T04:08:02.971 INFO:teuthology.orchestra.run.smithi006.stdout: "noscrub", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "pglog_hardlimit", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "purged_snapdirs", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "recovery_deletes", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "sortbitwise" 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_version": 20, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "full_ratio": 0.94999998807907104, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_snapshot": "", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_max": 11, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "max_osd": 8, 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "require_min_compat_client": "luminous", 2024-03-26T04:08:02.972 INFO:teuthology.orchestra.run.smithi006.stdout: "min_compat_client": "luminous", 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "require_osd_release": "squid", 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "allow_crimson": false, 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "pools": [ 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 1, 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".mgr", 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:46:30.183658+0000", 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:02.973 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 11, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 11, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 11, 2024-03-26T04:08:02.974 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 11, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 11, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "289", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "271", 2024-03-26T04:08:02.975 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:02.976 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:02.977 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_max": 32, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_min": 1 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.978 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "mgr": {} 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.1600000858306885, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.1600000858306885, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.3600000143051147, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.3600000143051147, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:02.979 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 2, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "datapool", 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:41.622497+0000", 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 8193, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:02.980 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 3, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 3, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 3, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 3, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 3, 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:02.981 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 2, 2024-03-26T04:08:02.982 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 69, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:02.983 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:02.984 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "rbd": {} 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.6600000858306885, 2024-03-26T04:08:02.985 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.6600000858306885, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.5, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.3300000429153442, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.3300000429153442, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 3, 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": ".rgw.root", 2024-03-26T04:08:02.986 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:42.191818+0000", 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T04:08:02.987 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.988 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "67", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:02.989 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:02.990 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:02.991 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4800000190734863, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4800000190734863, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.559999942779541, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.559999942779541, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:02.992 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 4, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.log", 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:44.079106+0000", 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:02.993 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T04:08:02.994 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "69", 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:02.995 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:02.996 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:02.997 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T04:08:02.998 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4800000190734863, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4800000190734863, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.559999942779541, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.559999942779541, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 5, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.control", 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:46.035261+0000", 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:03.000 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:03.001 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "71", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:03.002 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:03.003 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:03.004 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:03.005 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.2400000095367432, 2024-03-26T04:08:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.2400000095367432, 2024-03-26T04:08:03.006 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.4099999666213989, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.4099999666213989, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.007 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 6, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "default.rgw.meta", 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T03:48:48.086810+0000", 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 1, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool", 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:03.008 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 32, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 32, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 32, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 32, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 32, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:03.009 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "73", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:03.010 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:03.011 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:03.012 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "options": { 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_bias": 4 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": { 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "rgw": {} 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 1.9800000190734863, 2024-03-26T04:08:03.013 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 1.9800000190734863, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.25, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.25, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "pool": 11, 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_name": "unique_pool_4", 2024-03-26T04:08:03.014 INFO:teuthology.orchestra.run.smithi006.stdout: "create_time": "2024-03-26T04:08:01.187203+0000", 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flags": 32769, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flags_names": "hashpspool,creating", 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "type": 1, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 3, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "min_size": 2, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_rule": 0, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_count": 0, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_target": 0, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_barrier": 0, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "object_hash": 2, 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_autoscale_mode": "off", 2024-03-26T04:08:03.015 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num": 16, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num": 16, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_placement_num_target": 16, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_target": 16, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_num_pending": 16, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "last_pg_merge_meta": { 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "source_pgid": "0.0", 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "ready_epoch": 0, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_started": 0, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "last_epoch_clean": 0, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "source_version": "0'0", 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "target_version": "0'0" 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.016 INFO:teuthology.orchestra.run.smithi006.stdout: "last_change": "427", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend": "0", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "last_force_op_resend_preluminous": "0", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "auid": 0, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_mode": "selfmanaged", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_seq": 0, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "snap_epoch": 0, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "pool_snaps": [], 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps": "[]", 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_bytes": 0, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "quota_max_objects": 0, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "tiers": [], 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "tier_of": -1, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "read_tier": -1, 2024-03-26T04:08:03.017 INFO:teuthology.orchestra.run.smithi006.stdout: "write_tier": -1, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_mode": "none", 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_bytes": 0, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "target_max_objects": 0, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_flush_age": 0, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "cache_min_evict_age": 0, 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profile": "", 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_params": { 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "none" 2024-03-26T04:08:03.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_period": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_count": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "use_gmt_hitset": true, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "min_read_recency_for_promote": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "min_write_recency_for_promote": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_grade_decay_rate": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "hit_set_search_last_n": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "grade_table": [], 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "stripe_width": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "expected_num_objects": 0, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "fast_read": false, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "options": {}, 2024-03-26T04:08:03.019 INFO:teuthology.orchestra.run.smithi006.stdout: "application_metadata": {}, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "read_balance": { 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "score_type": "Fair distribution", 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "score_acting": 2.4800000190734863, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "score_stable": 2.4800000190734863, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "optimal_score": 0.62999999523162842, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_acting": 1.559999942779541, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "raw_score_stable": 1.559999942779541, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity_weighted": 0.96983146667480469, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity": 0.97000002861022949, 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "average_primary_affinity_weighted": 0.97000002861022949 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T04:08:03.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "1d1c84cf-5fa7-437e-8004-3b7aca831d6e", 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 8, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 427, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.021 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6801", 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.022 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6802", 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6804", 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6803", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1246039834 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6801/1246039834", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6802/1246039834", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6804/1246039834", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6803/1246039834", 2024-03-26T04:08:03.024 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "753f7459-62ec-447b-a0da-64d06e813f16", 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.025 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 13, 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 427, 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6805", 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.026 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6806", 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6808", 2024-03-26T04:08:03.027 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6807", 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 260311411 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.028 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6805/260311411", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6806/260311411", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6808/260311411", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6807/260311411", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "b7d7409f-341a-4711-8570-92fe9082a528", 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T04:08:03.029 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 18, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 335, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 378, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 427, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 336, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6809", 2024-03-26T04:08:03.030 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2571159281 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6810", 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2571159281 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.031 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6812", 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2571159281 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6811", 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2571159281 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6809/2571159281", 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6810/2571159281", 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6812/2571159281", 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6811/2571159281", 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.033 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "a91459a0-dee4-420c-ae2f-940756c985a6", 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 25, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 322, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.034 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6813", 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6814", 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6816", 2024-03-26T04:08:03.036 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.6:6815", 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 2795883342 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.6:6813/2795883342", 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.6:6814/2795883342", 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.6:6816/2795883342", 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.6:6815/2795883342", 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.038 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "087da212-7bf7-4982-b7cc-11f3f66c7ff8", 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 32, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 427, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.039 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6800", 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.040 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6801", 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.041 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6803", 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6802", 2024-03-26T04:08:03.042 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 931538411 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6800/931538411", 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6801/931538411", 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6803/931538411", 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6802/931538411", 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.043 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "7a688745-b213-4bbc-a735-42ffd2a5bfdc", 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 41, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 167, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 180, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 276, 2024-03-26T04:08:03.044 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 168, 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6804", 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.045 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6805", 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.046 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6807", 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6806", 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 909197223 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6804/909197223", 2024-03-26T04:08:03.047 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6805/909197223", 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6807/909197223", 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6806/909197223", 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "58b58555-59a3-4620-a15f-024f7041713f", 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 0, 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 0, 2024-03-26T04:08:03.048 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 1, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 50, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 95, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6808", 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T04:08:03.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6809", 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.050 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6811", 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.051 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6810", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 1580133777 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6808/1580133777", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6809/1580133777", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6811/1580133777", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6810/1580133777", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "uuid": "49b1833d-0a24-4168-a195-47b8ac1d9bc7", 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "up": 1, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "in": 1, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "weight": 1, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_affinity": 0.7586517333984375, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_begin": 0, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "last_clean_end": 0, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "up_from": 59, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "up_thru": 427, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "down_at": 0, 2024-03-26T04:08:03.053 INFO:teuthology.orchestra.run.smithi006.stdout: "lost_at": 0, 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addrs": { 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6812", 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addrs": { 2024-03-26T04:08:03.054 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6813", 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addrs": { 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6815", 2024-03-26T04:08:03.055 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addrs": { 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "addrvec": [ 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "type": "v2", 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "addr": "172.21.15.86:6814", 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "nonce": 3905426306 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "public_addr": "172.21.15.86:6812/3905426306", 2024-03-26T04:08:03.056 INFO:teuthology.orchestra.run.smithi006.stdout: "cluster_addr": "172.21.15.86:6813/3905426306", 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_back_addr": "172.21.15.86:6815/3905426306", 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "heartbeat_front_addr": "172.21.15.86:6814/3905426306", 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "state": [ 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "exists", 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "up" 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osd_xinfo": [ 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 0, 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.057 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:07:11.059988+0000", 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 1, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.058 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:07:12.426144+0000", 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 2, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T04:08:03.059 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:06:55.921779+0000", 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 335 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 3, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:06:54.762616+0000", 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 4, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:07:09.154256+0000", 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 5, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.061 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:06:33.013988+0000", 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 167 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 6, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 65536, 2024-03-26T04:08:03.062 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:07:08.073716+0000", 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "osd": 7, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "down_stamp": "0.000000", 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_probability": 0, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "laggy_interval": 0, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "features": 4540701547738038271, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "old_weight": 0, 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "last_purged_snaps_scrub": "2024-03-26T04:08:01.450627+0000", 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "dead_epoch": 0 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap": [ 2024-03-26T04:08:03.063 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "5.2", 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "osds": [ 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: 7, 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: 4, 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: 6 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_items": [ 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "3.1f", 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.064 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 0, 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.17", 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.065 INFO:teuthology.orchestra.run.smithi006.stdout: "pgid": "4.1f", 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "mappings": [ 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "from": 1, 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "to": 2 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_upmap_primaries": [], 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "pg_temp": [], 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "primary_temp": [], 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "blocklist": { 2024-03-26T04:08:03.066 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1978459317": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1771670410": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/3964053112": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/14180211": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/2547319348": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/601470125": "2024-03-27T03:44:16.095969+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/3870917014": "2024-03-27T03:44:16.095969+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1892511221": "2024-03-27T03:43:49.288943+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/435737617": "2024-03-27T03:43:49.288943+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/93567168": "2024-03-27T03:49:23.135700+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/2150730999": "2024-03-27T03:43:49.288943+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:6800/1166209788": "2024-03-27T03:43:49.288943+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1248481299": "2024-03-27T03:44:16.095969+0000", 2024-03-26T04:08:03.067 INFO:teuthology.orchestra.run.smithi006.stdout: "172.21.15.6:0/1832156537": "2024-03-27T03:44:16.095969+0000" 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "range_blocklist": {}, 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "erasure_code_profiles": { 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "default": { 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "crush-failure-domain": "osd", 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "k": "2", 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "m": "1", 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "plugin": "jerasure", 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "technique": "reed_sol_van" 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "removed_snaps_queue": [], 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "new_removed_snaps": [], 2024-03-26T04:08:03.068 INFO:teuthology.orchestra.run.smithi006.stdout: "new_purged_snaps": [], 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "crush_node_flags": {}, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "device_class_flags": {}, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode": { 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_enabled": false, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_bucket_count": 0, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "degraded_stretch_mode": 0, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "recovering_stretch_mode": 0, 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: "stretch_mode_bucket": 0 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:08:03.069 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:08:03.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:03.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:03.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:03.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:03.219 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:03.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:03.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:03.344 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:03.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:03.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:03.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:03.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:03.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:03.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:03.719 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:03.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:03.747 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:08:03.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:03.844 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:03.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:03 smithi086 ceph-mon[32965]: pgmap v915: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:08:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:03 smithi086 ceph-mon[32965]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:08:03.869 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:03 smithi086 ceph-mon[32965]: osdmap e428: 8 total, 8 up, 5 in 2024-03-26T04:08:03.870 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3880793203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T04:08:03.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[37501]: pgmap v915: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:08:03.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[37501]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:08:03.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[37501]: osdmap e428: 8 total, 8 up, 5 in 2024-03-26T04:08:03.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3880793203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T04:08:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[29446]: pgmap v915: 158 pgs: 16 unknown, 142 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 755 B/s rd, 0 op/s 2024-03-26T04:08:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[29446]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:08:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[29446]: osdmap e428: 8 total, 8 up, 5 in 2024-03-26T04:08:03.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3880793203' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-26T04:08:03.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:03.969 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:03.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:03.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:04.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:04.093 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:04.119 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246577 9.c 9:3c22429d:::benchmark_data_smithi006_132277_object114120:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.64369587800002, 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0190602820000001, 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.120 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246577 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474004+0000", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474005+0000", 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.392e-06 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474011+0000", 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6559999999999997e-06 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474028+0000", 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6803000000000002e-05 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474036+0000", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2830000000000001e-06 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474180+0000", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143886 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.474200+0000", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0151000000000001e-05 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487746+0000", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0135455710000001 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487828+0000", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2149000000000002e-05 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.487907+0000", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9178999999999995e-05 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492885+0000", 2024-03-26T04:08:04.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0049778419999999997 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492929+0000", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4484000000000003e-05 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.492960+0000", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1207999999999997e-05 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493012+0000", 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1739e-05 2024-03-26T04:08:04.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493064+0000", 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1938999999999999e-05 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246589 9.c 9:39119fb8:::benchmark_data_smithi006_132277_object114132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.62894429200003, 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0130038640000001, 2024-03-26T04:08:04.126 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246589 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488755+0000", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488756+0000", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0669999999999999e-06 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488763+0000", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9990000000000002e-06 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488773+0000", 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9250000000000007e-06 2024-03-26T04:08:04.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488779+0000", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7579999999999998e-06 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488826+0000", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7361999999999998e-05 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.488834+0000", 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2060000000000001e-06 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.489766+0000", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.000932545 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.489892+0000", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000126031 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.490007+0000", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011425399999999999 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.130 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493161+0000", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0031543069999999999 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493214+0000", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3563000000000002e-05 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501677+0000", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084626419999999994 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501713+0000", 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5618999999999999e-05 2024-03-26T04:08:04.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501759+0000", 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6585999999999997e-05 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246591 9.c 9:3623bec9:::benchmark_data_smithi006_132277_object114134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.61655090400001, 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0007254729999999, 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.133 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246591 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501149+0000", 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501151+0000", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8729999999999999e-06 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501160+0000", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2569999999999996e-06 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501186+0000", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5873000000000001e-05 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501201+0000", 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5702999999999999e-05 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501323+0000", 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000121037 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.501350+0000", 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7759999999999998e-05 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491627+0000", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99027648999999995 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491716+0000", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9291999999999998e-05 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.491794+0000", 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8319999999999996e-05 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493276+0000", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001481254 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493292+0000", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6690000000000001e-05 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501788+0000", 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0084954339999999996 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501830+0000", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2565000000000003e-05 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501874+0000", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3924999999999997e-05 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246599 9.c 9:3b75a9f3:::benchmark_data_smithi006_132277_object114142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:08:04.139 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.608333319, 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.99391891300000001, 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246599 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509366+0000", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509368+0000", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2049999999999999e-06 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509372+0000", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9150000000000004e-06 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509382+0000", 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0016e-05 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509390+0000", 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9869999999999999e-06 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509442+0000", 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1116999999999999e-05 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.509451+0000", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0990000000000005e-06 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493330+0000", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98387935100000001 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493444+0000", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114162 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.493519+0000", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5093999999999996e-05 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498353+0000", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048332870000000003 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498420+0000", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7359000000000003e-05 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503230+0000", 2024-03-26T04:08:04.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0048096390000000001 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503261+0000", 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1149999999999998e-05 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503285+0000", 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4532000000000002e-05 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.145 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246608 9.c 9:3b4033d5:::benchmark_data_smithi006_132277_object114151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.59937512300002, 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.98504782099999999, 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246608 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.146 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518325+0000", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518326+0000", 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.578e-06 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518331+0000", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0690000000000001e-06 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518341+0000", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5929999999999995e-06 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518348+0000", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6550000000000004e-06 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518403+0000", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5022000000000003e-05 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.518413+0000", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6120000000000005e-06 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.494934+0000", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.97652113500000004 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.495065+0000", 2024-03-26T04:08:04.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013094800000000001 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.495183+0000", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118294 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498409+0000", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.003225983 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498423+0000", 2024-03-26T04:08:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3837e-05 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503319+0000", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.004895885 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503349+0000", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9784000000000001e-05 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.503372+0000", 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3425999999999999e-05 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.151 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246642 9.c 9:342d0ba9:::benchmark_data_smithi006_132277_object114185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.56214405100002, 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94969048599999994, 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246642 2024-03-26T04:08:04.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555556+0000", 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555557+0000", 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.057e-06 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555561+0000", 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1509999999999997e-06 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555571+0000", 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.893e-06 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555578+0000", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0040000000000002e-06 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555623+0000", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5130000000000003e-05 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.555629+0000", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2659999999999998e-06 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496611+0000", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94098225300000005 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496742+0000", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013053199999999999 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.496862+0000", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000120195 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501516+0000", 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0046534380000000002 2024-03-26T04:08:04.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501570+0000", 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4661999999999999e-05 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505186+0000", 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0036161320000000002 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505220+0000", 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3342e-05 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.505246+0000", 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6431000000000002e-05 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15312.0:246653 9.c 9:309ed127:::benchmark_data_smithi006_132277_object114196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e332)", 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:08:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 298.54841123, 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93844572599999998, 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15312", 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/2977875296", 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 246653 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569288+0000", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569289+0000", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9099999999999991e-07 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569294+0000", 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.16e-06 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569304+0000", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0180000000000001e-05 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569310+0000", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1689999999999997e-06 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569354+0000", 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4168999999999997e-05 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.161 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:05.569361+0000", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.99e-06 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498462+0000", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92910126800000004 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498588+0000", 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125926 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 0,7", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.498728+0000", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013995300000000001 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501622+0000", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028934070000000002 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.501633+0000", 2024-03-26T04:08:04.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1182000000000001e-05 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507630+0000", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0059974529999999998 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507690+0000", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9312999999999999e-05 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:03:06.507734+0000", 2024-03-26T04:08:04.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4564999999999998e-05 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:173267 10.e 10:797b5757:::benchmark_data_smithi006_159841_object1172:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e377)", 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:28.930335+0000", 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 155.18736482599999, 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.645992664, 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.165 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 173267 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930335+0000", 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930335+0000", 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.166 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930335+0000", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4300000000000003e-07 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930339+0000", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9360000000000001e-06 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930347+0000", 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1600000000000001e-06 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.167 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.930353+0000", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5400000000000001e-06 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.931428+0000", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010745900000000001 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.931547+0000", 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119547 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.168 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 7", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.931662+0000", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011490200000000001 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.933759+0000", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020966549999999998 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951153+0000", 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.017394542999999998 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.169 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951166+0000", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3123e-05 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506715+0000", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.55554879300000004 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506730+0000", 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4793999999999999e-05 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.170 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.576273+0000", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0695429459999999 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.576328+0000", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4491999999999999e-05 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:173276 10.e 10:7d36b4cc:::benchmark_data_smithi006_159841_object1181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:28.951655+0000", 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 155.16604507400001, 2024-03-26T04:08:04.171 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.632613984, 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 173276 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951655+0000", 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.172 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951655+0000", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951656+0000", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4869999999999999e-06 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951662+0000", 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0769999999999996e-06 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951674+0000", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1430000000000001e-05 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.951685+0000", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1345000000000001e-05 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.952001+0000", 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031607400000000001 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.952011+0000", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8300000000000008e-06 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506782+0000", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.55477071899999997 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506794+0000", 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2686e-05 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.576508+0000", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0697141619999999 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.576595+0000", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6954999999999997e-05 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.576687+0000", 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1457000000000002e-05 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581262+0000", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0045753410000000001 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581345+0000", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2562e-05 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.584169+0000", 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002824558 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.584206+0000", 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6897999999999998e-05 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.584269+0000", 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2403000000000005e-05 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:173282 10.e 10:7a333229:::benchmark_data_smithi006_159841_object1187:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-26T04:08:04.178 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:28.994786+0000", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 155.12291360500001, 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5957221270000002, 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 173282 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994786+0000", 2024-03-26T04:08:04.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994786+0000", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994788+0000", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.587e-06 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994793+0000", 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1900000000000003e-06 2024-03-26T04:08:04.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994803+0000", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0396e-05 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994811+0000", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1370000000000002e-06 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994856+0000", 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.464e-05 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.994863+0000", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.165e-06 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506828+0000", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.51196498800000001 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506834+0000", 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.835e-06 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.182 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.183 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.578165+0000", 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0713312699999999 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.578296+0000", 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000130228 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.184 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.578440+0000", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000144003 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581328+0000", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0028882930000000001 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581365+0000", 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7487999999999998e-05 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.185 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590305+0000", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0089393219999999995 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590381+0000", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6582000000000006e-05 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590508+0000", 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012700299999999999 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.186 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:173285 10.e 10:75d4f64b:::benchmark_data_smithi006_159841_object1190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:28.998275+0000", 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 155.11942498299999, 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5924787060000001, 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 173285 2024-03-26T04:08:04.187 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998275+0000", 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998275+0000", 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.188 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998275+0000", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8999999999999996e-07 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998278+0000", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0199999999999999e-06 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998285+0000", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6059999999999999e-06 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998292+0000", 2024-03-26T04:08:04.189 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7610000000000002e-06 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998329+0000", 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7332000000000001e-05 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:28.998335+0000", 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8259999999999998e-06 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.190 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506874+0000", 2024-03-26T04:08:04.193 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.50853966500000003 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506884+0000", 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2990000000000002e-06 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.579528+0000", 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.072644216 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.194 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.579672+0000", 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000143518 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.579823+0000", 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00015190700000000001 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581376+0000", 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0015527099999999999 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.195 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.582516+0000", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011396100000000001 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590573+0000", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008057247 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590639+0000", 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5510000000000001e-05 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.196 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590753+0000", 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000114889 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:173287 10.e 10:79b59f23:::benchmark_data_smithi006_159841_object1192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e378)", 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:05:29.000775+0000", 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 155.11692443600001, 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5901856300000001, 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.197 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 173287 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000775+0000", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000775+0000", 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.198 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000776+0000", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2900000000000003e-07 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000779+0000", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9960000000000002e-06 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000786+0000", 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3540000000000002e-06 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.199 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000791+0000", 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9200000000000003e-06 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.200 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000825+0000", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3571000000000001e-05 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.000831+0000", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5640000000000002e-06 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506901+0000", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.506069926 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:29.506906+0000", 2024-03-26T04:08:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8749999999999999e-06 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581412+0000", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.074506323 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581496+0000", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3725000000000002e-05 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.581586+0000", 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0134000000000001e-05 2024-03-26T04:08:04.202 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.582611+0000", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010249650000000001 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.582727+0000", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116185 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590803+0000", 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.008075156 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.203 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590869+0000", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6148000000000004e-05 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:05:30.590961+0000", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2158999999999996e-05 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:297970 10.e 10:708bde39:::benchmark_data_smithi006_159841_object125875:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.091922+0000", 2024-03-26T04:08:04.204 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.025777370999997, 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.96234198199999998, 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 297970 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091922+0000", 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.205 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091922+0000", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091924+0000", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7010000000000001e-06 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091930+0000", 2024-03-26T04:08:04.206 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.144e-06 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091944+0000", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3896999999999999e-05 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.091952+0000", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0819999999999999e-06 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.092019+0000", 2024-03-26T04:08:04.207 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6450000000000002e-05 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.092029+0000", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0536e-05 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.050495+0000", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95846624599999997 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.050618+0000", 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012277100000000001 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.208 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.050705+0000", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7280999999999993e-05 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054138+0000", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0034330340000000002 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054175+0000", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6808999999999999e-05 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.209 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054208+0000", 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3210000000000002e-05 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054239+0000", 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0219999999999999e-05 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054264+0000", 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5601000000000001e-05 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.210 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:297988 10.e 10:745754ff:::benchmark_data_smithi006_159841_object125893:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.102280+0000", 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.015419420999997, 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95388772399999999, 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 297988 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.211 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102280+0000", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102280+0000", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102281+0000", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999997e-07 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.212 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102286+0000", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2320000000000001e-06 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102296+0000", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6050000000000004e-06 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102304+0000", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5719999999999997e-06 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102353+0000", 2024-03-26T04:08:04.213 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9026000000000001e-05 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.102362+0000", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1099999999999992e-06 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.052253+0000", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94989173000000005 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.052327+0000", 2024-03-26T04:08:04.214 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3734999999999998e-05 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.052413+0000", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5945000000000005e-05 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054292+0000", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00187896 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054322+0000", 2024-03-26T04:08:04.215 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9609000000000001e-05 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056110+0000", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017885570000000001 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056141+0000", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1072000000000003e-05 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056168+0000", 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6661000000000001e-05 2024-03-26T04:08:04.216 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:297993 10.e 10:7bfeb05d:::benchmark_data_smithi006_159841_object125898:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.104835+0000", 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.012864362999998, 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.95145781799999996, 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 297993 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.217 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104835+0000", 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104835+0000", 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.218 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104836+0000", 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.02e-06 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104847+0000", 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.084e-05 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104864+0000", 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7152000000000001e-05 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.219 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104873+0000", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8650000000000007e-06 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104918+0000", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4913000000000002e-05 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.104924+0000", 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8989999999999998e-06 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.220 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.053288+0000", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.94836350400000002 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.053356+0000", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8073999999999999e-05 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.053427+0000", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0963000000000005e-05 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.221 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054331+0000", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090452800000000004 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056205+0000", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0018741039999999999 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056245+0000", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0062999999999999e-05 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.222 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056272+0000", 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7158999999999999e-05 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056293+0000", 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0733999999999998e-05 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:298030 10.e 10:7c5879f7:::benchmark_data_smithi006_159841_object125935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.126901+0000", 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.990799170000003, 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.93151971899999997, 2024-03-26T04:08:04.223 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 298030 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126901+0000", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126901+0000", 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126901+0000", 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9800000000000002e-07 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126906+0000", 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5950000000000001e-06 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.225 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126915+0000", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5799999999999992e-06 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126922+0000", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.379e-06 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126982+0000", 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9568999999999998e-05 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.226 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.126991+0000", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8609999999999993e-06 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054355+0000", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92736317000000001 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054425+0000", 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0865999999999995e-05 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.227 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.054510+0000", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4863000000000004e-05 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056215+0000", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0017050050000000001 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056338+0000", 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000122612 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.228 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058322+0000", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0019844609999999999 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058374+0000", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1369000000000002e-05 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058420+0000", 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6490999999999999e-05 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.229 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:298045 10.e 10:71694644:::benchmark_data_smithi006_159841_object125950:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.135847+0000", 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.981852367999998, 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.92268361799999998, 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 298045 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.230 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135847+0000", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135847+0000", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135848+0000", 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.08e-06 2024-03-26T04:08:04.231 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135853+0000", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0830000000000003e-06 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135864+0000", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0818999999999999e-05 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135871+0000", 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.3790000000000003e-06 2024-03-26T04:08:04.232 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135923+0000", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1959999999999997e-05 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.135930+0000", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1110000000000002e-06 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.055248+0000", 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.91931807099999996 2024-03-26T04:08:04.233 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.055317+0000", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9673999999999997e-05 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.055388+0000", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0158000000000002e-05 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056350+0000", 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00096207300000000003 2024-03-26T04:08:04.234 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058274+0000", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001924615 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058473+0000", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019890000000000001 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058507+0000", 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4116999999999999e-05 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.235 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058531+0000", 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3578e-05 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:298092 10.e 10:7e7e39f1:::benchmark_data_smithi006_159841_object125997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.217853+0000", 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.899847084000001, 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.84293515500000005, 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.236 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 298092 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217853+0000", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217853+0000", 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.237 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217853+0000", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6199999999999996e-07 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217858+0000", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6160000000000004e-06 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217867+0000", 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3170000000000008e-06 2024-03-26T04:08:04.238 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217875+0000", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4560000000000003e-06 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217928+0000", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2688000000000001e-05 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.217934+0000", 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1600000000000003e-06 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.239 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056373+0000", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.83843970099999998 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056436+0000", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2601000000000006e-05 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.056535+0000", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9382999999999994e-05 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.240 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058540+0000", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0020047419999999999 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.058594+0000", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4064000000000003e-05 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.060732+0000", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021381730000000002 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.060763+0000", 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0321e-05 2024-03-26T04:08:04.241 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.060788+0000", 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5071000000000001e-05 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:298118 10.e 10:75b35ad5:::benchmark_data_smithi006_159841_object126023:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.269880+0000", 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.847819886000003, 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.79593710500000003, 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.242 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 298118 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269880+0000", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269880+0000", 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269881+0000", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.062e-06 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269886+0000", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9830000000000004e-06 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269898+0000", 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1958e-05 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269905+0000", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6909999999999997e-06 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269961+0000", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5516999999999997e-05 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.269970+0000", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6659999999999997e-06 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.059711+0000", 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.789741472 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.059801+0000", 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9634000000000003e-05 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.059881+0000", 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0710999999999993e-05 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.062042+0000", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0021604369999999999 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.063317+0000", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012753040000000001 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.065715+0000", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0023979029999999998 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.065772+0000", 2024-03-26T04:08:04.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6461e-05 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.065817+0000", 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5306000000000003e-05 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_op(client.15540.0:298120 10.e 10:7bd266c7:::benchmark_data_smithi006_159841_object126025:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e406)", 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-03-26T04:07:03.270778+0000", 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 60.846921188000003, 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.79796330599999998, 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-03-26T04:08:04.248 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "client_info": { 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "client": "client.15540", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "tid": 298120 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270778+0000", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270778+0000", 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-03-26T04:08:04.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270779+0000", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1299999999999998e-07 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270784+0000", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8810000000000004e-06 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270795+0000", 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0526000000000001e-05 2024-03-26T04:08:04.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270802+0000", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.588e-06 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270853+0000", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1088000000000003e-05 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for peered", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:03.270861+0000", 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2370000000000003e-06 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.061052+0000", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.79019091900000005 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.061116+0000", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4480999999999995e-05 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "waiting for subops from 2,7", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.061196+0000", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9932999999999995e-05 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op_commit", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.063472+0000", 2024-03-26T04:08:04.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002275689 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.063519+0000", 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6896e-05 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.068640+0000", 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0051210980000000001 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.068696+0000", 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6332999999999997e-05 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-03-26T04:07:04.068742+0000", 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5723999999999998e-05 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-03-26T04:08:04.254 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:04.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:04.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:04.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:04.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:04.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:04.404 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:04.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:04.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:04.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:04.631 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:04.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[37501]: osdmap e429: 8 total, 8 up, 5 in 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[37501]: osdmap e430: 8 total, 8 up, 5 in 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:04.702 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:04.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[29446]: osdmap e429: 8 total, 8 up, 5 in 2024-03-26T04:08:04.703 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:04 smithi006 ceph-mon[29446]: osdmap e430: 8 total, 8 up, 5 in 2024-03-26T04:08:04.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:04.781 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:04.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T04:08:04.883 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:04.888 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:04 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:04.888 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:04 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:04.889 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:04 smithi086 ceph-mon[32965]: osdmap e429: 8 total, 8 up, 5 in 2024-03-26T04:08:04.889 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:04 smithi086 ceph-mon[32965]: osdmap e430: 8 total, 8 up, 5 in 2024-03-26T04:08:04.906 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:04.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:04.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:05.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:05.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:05.107 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:05.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:05.160 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:05.256 INFO:tasks.thrashosds.thrasher:j is {'epoch': 428, 'fsid': 'de980954-eb22-11ee-95d0-87774f69a715', 'created': '2024-03-26T03:43:14.534401+0000', 'modified': '2024-03-26T04:08:02.613672+0000', 'last_up_change': '2024-03-26T04:05:28.913716+0000', 'last_in_change': '2024-03-26T04:02:51.443642+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T03:46:30.183658+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '289', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '271', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.1600000858306885, 'score_stable': 2.1600000858306885, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.3600000143051147, 'raw_score_stable': 1.3600000143051147, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-26T03:48:41.622497+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-26T03:48:42.191818+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-26T03:48:44.079106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-26T03:48:46.035261+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.409999966621399, 'raw_score_stable': 1.409999966621399, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-26T03:48:48.086810+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9800000190734863, 'score_stable': 1.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-26T04:08:01.187203+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '427', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.9698314666748047, 'average_primary_affinity': 0.9700000286102295, 'average_primary_affinity_weighted': 0.9700000286102295}}], 'osds': [{'osd': 0, 'uuid': '1d1c84cf-5fa7-437e-8004-3b7aca831d6e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6801', 'nonce': 1246039834}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6802', 'nonce': 1246039834}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6804', 'nonce': 1246039834}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6803', 'nonce': 1246039834}]}, 'public_addr': '172.21.15.6:6801/1246039834', 'cluster_addr': '172.21.15.6:6802/1246039834', 'heartbeat_back_addr': '172.21.15.6:6804/1246039834', 'heartbeat_front_addr': '172.21.15.6:6803/1246039834', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '753f7459-62ec-447b-a0da-64d06e813f16', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6805', 'nonce': 260311411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6806', 'nonce': 260311411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6808', 'nonce': 260311411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6807', 'nonce': 260311411}]}, 'public_addr': '172.21.15.6:6805/260311411', 'cluster_addr': '172.21.15.6:6806/260311411', 'heartbeat_back_addr': '172.21.15.6:6808/260311411', 'heartbeat_front_addr': '172.21.15.6:6807/260311411', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'b7d7409f-341a-4711-8570-92fe9082a528', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 335, 'up_from': 378, 'up_thru': 427, 'down_at': 336, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6809', 'nonce': 2571159281}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6810', 'nonce': 2571159281}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6812', 'nonce': 2571159281}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6811', 'nonce': 2571159281}]}, 'public_addr': '172.21.15.6:6809/2571159281', 'cluster_addr': '172.21.15.6:6810/2571159281', 'heartbeat_back_addr': '172.21.15.6:6812/2571159281', 'heartbeat_front_addr': '172.21.15.6:6811/2571159281', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a91459a0-dee4-420c-ae2f-940756c985a6', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 322, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6813', 'nonce': 2795883342}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6814', 'nonce': 2795883342}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6816', 'nonce': 2795883342}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.6:6815', 'nonce': 2795883342}]}, 'public_addr': '172.21.15.6:6813/2795883342', 'cluster_addr': '172.21.15.6:6814/2795883342', 'heartbeat_back_addr': '172.21.15.6:6816/2795883342', 'heartbeat_front_addr': '172.21.15.6:6815/2795883342', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '087da212-7bf7-4982-b7cc-11f3f66c7ff8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6800', 'nonce': 931538411}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6801', 'nonce': 931538411}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6803', 'nonce': 931538411}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6802', 'nonce': 931538411}]}, 'public_addr': '172.21.15.86:6800/931538411', 'cluster_addr': '172.21.15.86:6801/931538411', 'heartbeat_back_addr': '172.21.15.86:6803/931538411', 'heartbeat_front_addr': '172.21.15.86:6802/931538411', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7a688745-b213-4bbc-a735-42ffd2a5bfdc', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 41, 'last_clean_end': 167, 'up_from': 180, 'up_thru': 276, 'down_at': 168, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6804', 'nonce': 909197223}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6805', 'nonce': 909197223}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6807', 'nonce': 909197223}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6806', 'nonce': 909197223}]}, 'public_addr': '172.21.15.86:6804/909197223', 'cluster_addr': '172.21.15.86:6805/909197223', 'heartbeat_back_addr': '172.21.15.86:6807/909197223', 'heartbeat_front_addr': '172.21.15.86:6806/909197223', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '58b58555-59a3-4620-a15f-024f7041713f', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6808', 'nonce': 1580133777}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6809', 'nonce': 1580133777}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6811', 'nonce': 1580133777}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6810', 'nonce': 1580133777}]}, 'public_addr': '172.21.15.86:6808/1580133777', 'cluster_addr': '172.21.15.86:6809/1580133777', 'heartbeat_back_addr': '172.21.15.86:6811/1580133777', 'heartbeat_front_addr': '172.21.15.86:6810/1580133777', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '49b1833d-0a24-4168-a195-47b8ac1d9bc7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.7586517333984375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 427, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6812', 'nonce': 3905426306}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6813', 'nonce': 3905426306}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6815', 'nonce': 3905426306}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.86:6814', 'nonce': 3905426306}]}, 'public_addr': '172.21.15.86:6812/3905426306', 'cluster_addr': '172.21.15.86:6813/3905426306', 'heartbeat_back_addr': '172.21.15.86:6815/3905426306', 'heartbeat_front_addr': '172.21.15.86:6814/3905426306', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T04:07:11.059988+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T04:07:12.426144+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T04:06:55.921779+0000', 'dead_epoch': 335}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T04:06:54.762616+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T04:07:09.154256+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T04:06:33.013988+0000', 'dead_epoch': 167}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-26T04:07:08.073716+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-26T04:08:01.450627+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.2', 'osds': [7, 4, 6]}], 'pg_upmap_items': [{'pgid': '3.1f', 'mappings': [{'from': 0, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.6:0/1978459317': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/1771670410': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/3964053112': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/14180211': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:6800/2547319348': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/601470125': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:6800/3870917014': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1892511221': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/435737617': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/93567168': '2024-03-27T03:49:23.135700+0000', '172.21.15.6:0/2150730999': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:6800/1166209788': '2024-03-27T03:43:49.288943+0000', '172.21.15.6:0/1248481299': '2024-03-27T03:44:16.095969+0000', '172.21.15.6:0/1832156537': '2024-03-27T03:44:16.095969+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-26T04:08:05.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:08:05.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:05.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:05.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:05.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:05.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:05.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:05.536 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:05.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:05.567 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:05.638 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:05.657 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:08:05.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:05.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:05.789 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:05.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[37501]: pgmap v918: 158 pgs: 14 unknown, 144 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3143615568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[29446]: pgmap v918: 158 pgs: 14 unknown, 144 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3143615568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.837 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:05 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:05 smithi086 ceph-mon[32965]: pgmap v918: 158 pgs: 14 unknown, 144 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:08:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:05 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3143615568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:05 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:05.891 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:05.915 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:05.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:05.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:06.016 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:06.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.040 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:06.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:06.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:06.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:06.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:06.293 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:06.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:06.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:06.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:06.521 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:06.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:06.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:06.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:06.666 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:08:06.667 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-03-26T04:08:06.667 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:06.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:06.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.799 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:06.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:06.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:06 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:06 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:06.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:06 smithi086 ceph-mon[32965]: osdmap e431: 8 total, 8 up, 5 in 2024-03-26T04:08:06.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:06.923 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:06.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:06.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:07.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:07.049 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:07.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:07.084 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[29446]: osdmap e431: 8 total, 8 up, 5 in 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:07.085 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:06 smithi006 ceph-mon[37501]: osdmap e431: 8 total, 8 up, 5 in 2024-03-26T04:08:07.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:07.174 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:07.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:07.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.301 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:07.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:07.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T04:08:07.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:07.426 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:07.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:07.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:07.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:07.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:07.591 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:07.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:07.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:07.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:07.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:07.802 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:07.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:07.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:07 smithi006 ceph-mon[37501]: pgmap v921: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:08:07.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:07 smithi006 ceph-mon[29446]: pgmap v921: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:08:07.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:07.915 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:07 smithi086 ceph-mon[32965]: pgmap v921: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:08:07.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:08.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:08.054 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:08.155 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:08.179 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:08.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:08.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.305 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:08.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:08.429 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:08.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:08.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:08.601 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:08.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:08.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:08.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:08.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:08.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:08.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:08.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:08.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[37501]: osdmap e432: 8 total, 8 up, 5 in 2024-03-26T04:08:08.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:08.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:08.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:08 smithi006 ceph-mon[29446]: osdmap e432: 8 total, 8 up, 5 in 2024-03-26T04:08:08.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:08.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:08 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:08.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:08 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:08.934 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:08 smithi086 ceph-mon[32965]: osdmap e432: 8 total, 8 up, 5 in 2024-03-26T04:08:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:08.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:08.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:09.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:09.061 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:09.066 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:08:09.085 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:08:09.105 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:08:09.124 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:08:09.146 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:08:09.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:09.166 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:08:09.183 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:09.183 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:08:09.183 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:08:09.183 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:08:09.183 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:09.187 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:08:09.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:09.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:09.208 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:08:09.209 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:08:09.210 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:08:09.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:09.314 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:09.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:09.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:09.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:09.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:09.562 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-26T04:08:09.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:09.667 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:09.692 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:09.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:08:09.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:09.816 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:09.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:09.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T04:08:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:09.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: pgmap v923: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: osdmap e433: 8 total, 8 up, 5 in 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/720906024' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4079817803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[37501]: osdmap e434: 8 total, 8 up, 5 in 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: pgmap v923: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: osdmap e433: 8 total, 8 up, 5 in 2024-03-26T04:08:09.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/720906024' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:09.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:09.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4079817803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:09.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:08:09.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:09 smithi006 ceph-mon[29446]: osdmap e434: 8 total, 8 up, 5 in 2024-03-26T04:08:09.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:09.959 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:09.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:10.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: pgmap v923: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: osdmap e433: 8 total, 8 up, 5 in 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/720906024' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:08:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4079817803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:08:10.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:09 smithi086 ceph-mon[32965]: osdmap e434: 8 total, 8 up, 5 in 2024-03-26T04:08:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:10.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:10.134 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:10.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:10.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:10.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:10.340 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:10.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:10.466 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:10.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:10.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:10.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:10.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:10.842 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:10.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:10.968 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:10.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:10.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:11.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:11.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.096 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:11.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:11.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.221 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:11.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:11.285 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:11 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:11.285 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:11 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:11.285 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:11 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:11.285 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:11 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:11.322 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.350 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:11.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:11.369 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:11.370 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:11.370 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:11 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:11.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:11.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:11.577 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:11.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:11.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:11.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:11.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:11.846 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:11.846 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:11.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:11.863 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:08:11.866 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:08:11.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:11.979 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:11.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:11.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:12.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:12.104 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:12.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.231 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:12.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:12.296 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:12.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:12 smithi086 ceph-mon[32965]: pgmap v926: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:12 smithi086 ceph-mon[32965]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:08:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:12 smithi086 ceph-mon[32965]: osdmap e435: 8 total, 8 up, 5 in 2024-03-26T04:08:12.357 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[37501]: pgmap v926: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[37501]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[37501]: osdmap e435: 8 total, 8 up, 5 in 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[29446]: pgmap v926: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[29446]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:08:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:12 smithi006 ceph-mon[29446]: osdmap e435: 8 total, 8 up, 5 in 2024-03-26T04:08:12.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:12.484 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:12.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T04:08:12.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:12.609 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:12.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:12.735 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:12.831 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":927,"stamp":"2024-03-26T04:08:11.039230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":2580,"num_read_kb":2300,"num_write":914,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":461,"num_bytes_recovered":12305,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2490,"ondisk_log_size":2490,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":3932704,"kb_used_data":13068,"kb_used_omap":12,"kb_used_meta":3919539,"kb_avail":464793056,"statfs":{"total":479975178240,"available":475948089344,"internally_reserved":0,"allocated":13381632,"data_stored":8106756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13250,"internal_metadata":4013607998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.674058"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":797,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553173+0000","last_change":"2024-03-26T04:05:03.632715+0000","last_active":"2024-03-26T04:08:09.553173+0000","last_peered":"2024-03-26T04:08:09.553173+0000","last_clean":"2024-03-26T04:08:09.553173+0000","last_became_active":"2024-03-26T04:01:35.797821+0000","last_became_peered":"2024-03-26T04:01:35.797821+0000","last_unstale":"2024-03-26T04:08:09.553173+0000","last_undegraded":"2024-03-26T04:08:09.553173+0000","last_fullsized":"2024-03-26T04:08:09.553173+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:03.632384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:03.632384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394593+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":800,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589587+0000","last_change":"2024-03-26T04:07:55.522053+0000","last_active":"2024-03-26T04:08:04.589587+0000","last_peered":"2024-03-26T04:08:04.589587+0000","last_clean":"2024-03-26T04:08:04.589587+0000","last_became_active":"2024-03-26T04:05:29.540706+0000","last_became_peered":"2024-03-26T04:05:29.540706+0000","last_unstale":"2024-03-26T04:08:04.589587+0000","last_undegraded":"2024-03-26T04:08:04.589587+0000","last_fullsized":"2024-03-26T04:08:04.589587+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:55.521958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:07:55.521958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:07.082309+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":812,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555397+0000","last_change":"2024-03-26T04:05:29.539024+0000","last_active":"2024-03-26T04:08:04.555397+0000","last_peered":"2024-03-26T04:08:04.555397+0000","last_clean":"2024-03-26T04:08:04.555397+0000","last_became_active":"2024-03-26T04:05:29.538364+0000","last_became_peered":"2024-03-26T04:05:29.538364+0000","last_unstale":"2024-03-26T04:08:04.555397+0000","last_undegraded":"2024-03-26T04:08:04.555397+0000","last_fullsized":"2024-03-26T04:08:04.555397+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:52.117692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:52.117692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"241'21","reported_seq":901,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.552832+0000","last_change":"2024-03-26T04:05:29.520520+0000","last_active":"2024-03-26T04:08:09.552832+0000","last_peered":"2024-03-26T04:08:09.552832+0000","last_clean":"2024-03-26T04:08:09.552832+0000","last_became_active":"2024-03-26T04:05:29.520084+0000","last_became_peered":"2024-03-26T04:05:29.520084+0000","last_unstale":"2024-03-26T04:08:09.552832+0000","last_undegraded":"2024-03-26T04:08:09.552832+0000","last_fullsized":"2024-03-26T04:08:09.552832+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:01:20.107206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:20.107206+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394197+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":458,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106487+0000","last_change":"2024-03-26T04:06:29.930571+0000","last_active":"2024-03-26T04:08:09.106487+0000","last_peered":"2024-03-26T04:08:09.106487+0000","last_clean":"2024-03-26T04:08:09.106487+0000","last_became_active":"2024-03-26T04:02:52.504637+0000","last_became_peered":"2024-03-26T04:02:52.504637+0000","last_unstale":"2024-03-26T04:08:09.106487+0000","last_undegraded":"2024-03-26T04:08:09.106487+0000","last_fullsized":"2024-03-26T04:08:09.106487+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:29.930462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:29.930462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":793,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.552675+0000","last_change":"2024-03-26T04:05:29.519865+0000","last_active":"2024-03-26T04:08:09.552675+0000","last_peered":"2024-03-26T04:08:09.552675+0000","last_clean":"2024-03-26T04:08:09.552675+0000","last_became_active":"2024-03-26T04:05:29.519486+0000","last_became_peered":"2024-03-26T04:05:29.519486+0000","last_unstale":"2024-03-26T04:08:09.552675+0000","last_undegraded":"2024-03-26T04:08:09.552675+0000","last_fullsized":"2024-03-26T04:08:09.552675+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.069804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.069804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394231+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":355,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661756+0000","last_change":"2024-03-26T04:07:30.526662+0000","last_active":"2024-03-26T04:08:07.661756+0000","last_peered":"2024-03-26T04:08:07.661756+0000","last_clean":"2024-03-26T04:08:07.661756+0000","last_became_active":"2024-03-26T04:05:29.536149+0000","last_became_peered":"2024-03-26T04:05:29.536149+0000","last_unstale":"2024-03-26T04:08:07.661756+0000","last_undegraded":"2024-03-26T04:08:07.661756+0000","last_fullsized":"2024-03-26T04:08:07.661756+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:44.136302+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"241'51","reported_seq":266,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660752+0000","last_change":"2024-03-26T04:07:44.566663+0000","last_active":"2024-03-26T04:08:07.660752+0000","last_peered":"2024-03-26T04:08:07.660752+0000","last_clean":"2024-03-26T04:08:07.660752+0000","last_became_active":"2024-03-26T04:05:29.533552+0000","last_became_peered":"2024-03-26T04:05:29.533552+0000","last_unstale":"2024-03-26T04:08:07.660752+0000","last_undegraded":"2024-03-26T04:08:07.660752+0000","last_fullsized":"2024-03-26T04:08:07.660752+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:07:44.566551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:44.566551+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:09.048408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":78,"num_read_kb":65,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":774,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554377+0000","last_change":"2024-03-26T04:04:21.537611+0000","last_active":"2024-03-26T04:08:04.554377+0000","last_peered":"2024-03-26T04:08:04.554377+0000","last_clean":"2024-03-26T04:08:04.554377+0000","last_became_active":"2024-03-26T03:48:50.037145+0000","last_became_peered":"2024-03-26T03:48:50.037145+0000","last_unstale":"2024-03-26T04:08:04.554377+0000","last_undegraded":"2024-03-26T04:08:04.554377+0000","last_fullsized":"2024-03-26T04:08:04.554377+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:21.537431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:21.537431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":535,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554831+0000","last_change":"2024-03-26T04:03:53.342268+0000","last_active":"2024-03-26T04:08:04.554831+0000","last_peered":"2024-03-26T04:08:04.554831+0000","last_clean":"2024-03-26T04:08:04.554831+0000","last_became_active":"2024-03-26T04:01:35.802827+0000","last_became_peered":"2024-03-26T04:01:35.802827+0000","last_unstale":"2024-03-26T04:08:04.554831+0000","last_undegraded":"2024-03-26T04:08:04.554831+0000","last_fullsized":"2024-03-26T04:08:04.554831+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:53.342172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.342172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":499,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554402+0000","last_change":"2024-03-26T04:04:55.527963+0000","last_active":"2024-03-26T04:08:09.554402+0000","last_peered":"2024-03-26T04:08:09.554402+0000","last_clean":"2024-03-26T04:08:09.554402+0000","last_became_active":"2024-03-26T04:02:52.503960+0000","last_became_peered":"2024-03-26T04:02:52.503960+0000","last_unstale":"2024-03-26T04:08:09.554402+0000","last_undegraded":"2024-03-26T04:08:09.554402+0000","last_fullsized":"2024-03-26T04:08:09.554402+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:55.527836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.527836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394461+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"241'28","reported_seq":909,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553449+0000","last_change":"2024-03-26T04:07:26.622146+0000","last_active":"2024-03-26T04:08:09.553449+0000","last_peered":"2024-03-26T04:08:09.553449+0000","last_clean":"2024-03-26T04:08:09.553449+0000","last_became_active":"2024-03-26T04:05:29.521715+0000","last_became_peered":"2024-03-26T04:05:29.521715+0000","last_unstale":"2024-03-26T04:08:09.553449+0000","last_undegraded":"2024-03-26T04:08:09.553449+0000","last_fullsized":"2024-03-26T04:08:09.553449+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T04:07:26.622024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.622024+0000","objects_scrubbed":6,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:36.899154+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":594,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589694+0000","last_change":"2024-03-26T04:07:38.459491+0000","last_active":"2024-03-26T04:08:04.589694+0000","last_peered":"2024-03-26T04:08:04.589694+0000","last_clean":"2024-03-26T04:08:04.589694+0000","last_became_active":"2024-03-26T04:05:29.532847+0000","last_became_peered":"2024-03-26T04:05:29.532847+0000","last_unstale":"2024-03-26T04:08:04.589694+0000","last_undegraded":"2024-03-26T04:08:04.589694+0000","last_fullsized":"2024-03-26T04:08:04.589694+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.459399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.459399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:54.713401+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":814,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.105888+0000","last_change":"2024-03-26T04:07:24.766554+0000","last_active":"2024-03-26T04:08:09.105888+0000","last_peered":"2024-03-26T04:08:09.105888+0000","last_clean":"2024-03-26T04:08:09.105888+0000","last_became_active":"2024-03-26T03:48:48.031671+0000","last_became_peered":"2024-03-26T03:48:48.031671+0000","last_unstale":"2024-03-26T04:08:09.105888+0000","last_undegraded":"2024-03-26T04:08:09.105888+0000","last_fullsized":"2024-03-26T04:08:09.105888+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.766403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.766403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:28.353556+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":518,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553260+0000","last_change":"2024-03-26T04:04:01.292305+0000","last_active":"2024-03-26T04:08:09.553260+0000","last_peered":"2024-03-26T04:08:09.553260+0000","last_clean":"2024-03-26T04:08:09.553260+0000","last_became_active":"2024-03-26T04:01:35.797214+0000","last_became_peered":"2024-03-26T04:01:35.797214+0000","last_unstale":"2024-03-26T04:08:09.553260+0000","last_undegraded":"2024-03-26T04:08:09.553260+0000","last_fullsized":"2024-03-26T04:08:09.553260+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:01.292182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:01.292182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394283+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"241'16","reported_seq":861,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553731+0000","last_change":"2024-03-26T04:05:29.520607+0000","last_active":"2024-03-26T04:08:09.553731+0000","last_peered":"2024-03-26T04:08:09.553731+0000","last_clean":"2024-03-26T04:08:09.553731+0000","last_became_active":"2024-03-26T04:05:29.519855+0000","last_became_peered":"2024-03-26T04:05:29.519855+0000","last_unstale":"2024-03-26T04:08:09.553731+0000","last_undegraded":"2024-03-26T04:08:09.553731+0000","last_fullsized":"2024-03-26T04:08:09.553731+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:01:08.181298+0000","last_deep_scrub":"131'14","last_deep_scrub_stamp":"2024-03-26T03:52:38.915559+0000","last_clean_scrub_stamp":"2024-03-26T04:01:08.181298+0000","objects_scrubbed":3,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394215+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":23,"num_read_kb":18,"num_write":14,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":823,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554821+0000","last_change":"2024-03-26T04:05:02.671720+0000","last_active":"2024-03-26T04:08:09.554821+0000","last_peered":"2024-03-26T04:08:09.554821+0000","last_clean":"2024-03-26T04:08:09.554821+0000","last_became_active":"2024-03-26T04:02:52.500023+0000","last_became_peered":"2024-03-26T04:02:52.500023+0000","last_unstale":"2024-03-26T04:08:09.554821+0000","last_undegraded":"2024-03-26T04:08:09.554821+0000","last_fullsized":"2024-03-26T04:08:09.554821+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:02.671615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:02.671615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394504+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":784,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590625+0000","last_change":"2024-03-26T04:04:51.186722+0000","last_active":"2024-03-26T04:08:04.590625+0000","last_peered":"2024-03-26T04:08:04.590625+0000","last_clean":"2024-03-26T04:08:04.590625+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:08:04.590625+0000","last_undegraded":"2024-03-26T04:08:04.590625+0000","last_fullsized":"2024-03-26T04:08:04.590625+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.186605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.186605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588638+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":794,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555018+0000","last_change":"2024-03-26T04:03:54.321625+0000","last_active":"2024-03-26T04:08:04.555018+0000","last_peered":"2024-03-26T04:08:04.555018+0000","last_clean":"2024-03-26T04:08:04.555018+0000","last_became_active":"2024-03-26T03:48:44.014711+0000","last_became_peered":"2024-03-26T03:48:44.014711+0000","last_unstale":"2024-03-26T04:08:04.555018+0000","last_undegraded":"2024-03-26T04:08:04.555018+0000","last_fullsized":"2024-03-26T04:08:04.555018+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:54.321548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:54.321548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"241'27","reported_seq":821,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661826+0000","last_change":"2024-03-26T04:05:29.533402+0000","last_active":"2024-03-26T04:08:07.661826+0000","last_peered":"2024-03-26T04:08:07.661826+0000","last_clean":"2024-03-26T04:08:07.661826+0000","last_became_active":"2024-03-26T04:05:29.525450+0000","last_became_peered":"2024-03-26T04:05:29.525450+0000","last_unstale":"2024-03-26T04:08:07.661826+0000","last_undegraded":"2024-03-26T04:08:07.661826+0000","last_fullsized":"2024-03-26T04:08:07.661826+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:02:45.014541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:45.014541+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682154+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":417,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.591093+0000","last_change":"2024-03-26T04:05:29.533637+0000","last_active":"2024-03-26T04:08:04.591093+0000","last_peered":"2024-03-26T04:08:04.591093+0000","last_clean":"2024-03-26T04:08:04.591093+0000","last_became_active":"2024-03-26T04:05:29.533287+0000","last_became_peered":"2024-03-26T04:05:29.533287+0000","last_unstale":"2024-03-26T04:08:04.591093+0000","last_undegraded":"2024-03-26T04:08:04.591093+0000","last_fullsized":"2024-03-26T04:08:04.591093+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:22.809763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:01:22.809763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588348+0000","scrub_duration":144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":813,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553443+0000","last_change":"2024-03-26T04:07:37.463272+0000","last_active":"2024-03-26T04:08:09.553443+0000","last_peered":"2024-03-26T04:08:09.553443+0000","last_clean":"2024-03-26T04:08:09.553443+0000","last_became_active":"2024-03-26T04:05:29.536639+0000","last_became_peered":"2024-03-26T04:05:29.536639+0000","last_unstale":"2024-03-26T04:08:09.553443+0000","last_undegraded":"2024-03-26T04:08:09.553443+0000","last_fullsized":"2024-03-26T04:08:09.553443+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.463158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.463158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.787883+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":803,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553546+0000","last_change":"2024-03-26T04:07:24.624918+0000","last_active":"2024-03-26T04:08:09.553546+0000","last_peered":"2024-03-26T04:08:09.553546+0000","last_clean":"2024-03-26T04:08:09.553546+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:08:09.553546+0000","last_undegraded":"2024-03-26T04:08:09.553546+0000","last_fullsized":"2024-03-26T04:08:09.553546+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.624793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.624793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.269799+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"241'24","reported_seq":341,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590454+0000","last_change":"2024-03-26T04:07:28.446713+0000","last_active":"2024-03-26T04:08:04.590454+0000","last_peered":"2024-03-26T04:08:04.590454+0000","last_clean":"2024-03-26T04:08:04.590454+0000","last_became_active":"2024-03-26T04:05:29.521493+0000","last_became_peered":"2024-03-26T04:05:29.521493+0000","last_unstale":"2024-03-26T04:08:04.590454+0000","last_undegraded":"2024-03-26T04:08:04.590454+0000","last_fullsized":"2024-03-26T04:08:04.590454+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:07:28.446562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:28.446562+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:32.529078+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":31,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":259,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.662021+0000","last_change":"2024-03-26T04:05:29.548371+0000","last_active":"2024-03-26T04:08:07.662021+0000","last_peered":"2024-03-26T04:08:07.662021+0000","last_clean":"2024-03-26T04:08:07.662021+0000","last_became_active":"2024-03-26T04:05:29.545095+0000","last_became_peered":"2024-03-26T04:05:29.545095+0000","last_unstale":"2024-03-26T04:08:07.662021+0000","last_undegraded":"2024-03-26T04:08:07.662021+0000","last_fullsized":"2024-03-26T04:08:07.662021+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:02:40.908630+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:02:40.908630+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682135+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"379'11","reported_seq":973,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554918+0000","last_change":"2024-03-26T04:05:29.535165+0000","last_active":"2024-03-26T04:08:09.554918+0000","last_peered":"2024-03-26T04:08:09.554918+0000","last_clean":"2024-03-26T04:08:09.554918+0000","last_became_active":"2024-03-26T04:05:29.534648+0000","last_became_peered":"2024-03-26T04:05:29.534648+0000","last_unstale":"2024-03-26T04:08:09.554918+0000","last_undegraded":"2024-03-26T04:08:09.554918+0000","last_fullsized":"2024-03-26T04:08:09.554918+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:00:35.094793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:00:35.094793+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":824,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589545+0000","last_change":"2024-03-26T04:04:18.067687+0000","last_active":"2024-03-26T04:08:04.589545+0000","last_peered":"2024-03-26T04:08:04.589545+0000","last_clean":"2024-03-26T04:08:04.589545+0000","last_became_active":"2024-03-26T04:02:52.514953+0000","last_became_peered":"2024-03-26T04:02:52.514953+0000","last_unstale":"2024-03-26T04:08:04.589545+0000","last_undegraded":"2024-03-26T04:08:04.589545+0000","last_fullsized":"2024-03-26T04:08:04.589545+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:04:18.067462+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.067462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588607+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"356'32","reported_seq":892,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555503+0000","last_change":"2024-03-26T04:04:05.345019+0000","last_active":"2024-03-26T04:08:04.555503+0000","last_peered":"2024-03-26T04:08:04.555503+0000","last_clean":"2024-03-26T04:08:04.555503+0000","last_became_active":"2024-03-26T04:02:52.497638+0000","last_became_peered":"2024-03-26T04:02:52.497638+0000","last_unstale":"2024-03-26T04:08:04.555503+0000","last_undegraded":"2024-03-26T04:08:04.555503+0000","last_fullsized":"2024-03-26T04:08:04.555503+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'32","last_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:04:05.344849+0000","objects_scrubbed":6,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":796,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590342+0000","last_change":"2024-03-26T04:07:47.538830+0000","last_active":"2024-03-26T04:08:04.590342+0000","last_peered":"2024-03-26T04:08:04.590342+0000","last_clean":"2024-03-26T04:08:04.590342+0000","last_became_active":"2024-03-26T04:05:29.520352+0000","last_became_peered":"2024-03-26T04:05:29.520352+0000","last_unstale":"2024-03-26T04:08:04.590342+0000","last_undegraded":"2024-03-26T04:08:04.590342+0000","last_fullsized":"2024-03-26T04:08:04.590342+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:47.538728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:47.538728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:04.686218+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":789,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555747+0000","last_change":"2024-03-26T04:07:29.534176+0000","last_active":"2024-03-26T04:08:04.555747+0000","last_peered":"2024-03-26T04:08:04.555747+0000","last_clean":"2024-03-26T04:08:04.555747+0000","last_became_active":"2024-03-26T04:05:29.536597+0000","last_became_peered":"2024-03-26T04:05:29.536597+0000","last_unstale":"2024-03-26T04:08:04.555747+0000","last_undegraded":"2024-03-26T04:08:04.555747+0000","last_fullsized":"2024-03-26T04:08:04.555747+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:29.533765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.533765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:31.454229+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":381,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554321+0000","last_change":"2024-03-26T04:07:31.484578+0000","last_active":"2024-03-26T04:08:04.554321+0000","last_peered":"2024-03-26T04:08:04.554321+0000","last_clean":"2024-03-26T04:08:04.554321+0000","last_became_active":"2024-03-26T04:05:29.540582+0000","last_became_peered":"2024-03-26T04:05:29.540582+0000","last_unstale":"2024-03-26T04:08:04.554321+0000","last_undegraded":"2024-03-26T04:08:04.554321+0000","last_fullsized":"2024-03-26T04:08:04.554321+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:31.484355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.484355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:56.172779+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"241'27","reported_seq":619,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.662274+0000","last_change":"2024-03-26T04:05:29.537371+0000","last_active":"2024-03-26T04:08:07.662274+0000","last_peered":"2024-03-26T04:08:07.662274+0000","last_clean":"2024-03-26T04:08:07.662274+0000","last_became_active":"2024-03-26T04:05:29.536977+0000","last_became_peered":"2024-03-26T04:05:29.536977+0000","last_unstale":"2024-03-26T04:08:07.662274+0000","last_undegraded":"2024-03-26T04:08:07.662274+0000","last_fullsized":"2024-03-26T04:08:07.662274+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":31,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":812,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106621+0000","last_change":"2024-03-26T04:07:42.011236+0000","last_active":"2024-03-26T04:08:09.106621+0000","last_peered":"2024-03-26T04:08:09.106621+0000","last_clean":"2024-03-26T04:08:09.106621+0000","last_became_active":"2024-03-26T04:05:29.533627+0000","last_became_peered":"2024-03-26T04:05:29.533627+0000","last_unstale":"2024-03-26T04:08:09.106621+0000","last_undegraded":"2024-03-26T04:08:09.106621+0000","last_fullsized":"2024-03-26T04:08:09.106621+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.011134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:43.732641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":725,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.552812+0000","last_change":"2024-03-26T04:04:17.385877+0000","last_active":"2024-03-26T04:08:09.552812+0000","last_peered":"2024-03-26T04:08:09.552812+0000","last_clean":"2024-03-26T04:08:09.552812+0000","last_became_active":"2024-03-26T03:50:45.592971+0000","last_became_peered":"2024-03-26T03:50:45.592971+0000","last_unstale":"2024-03-26T04:08:09.552812+0000","last_undegraded":"2024-03-26T04:08:09.552812+0000","last_fullsized":"2024-03-26T04:08:09.552812+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:17.385765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.385765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394407+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":829,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555695+0000","last_change":"2024-03-26T04:07:37.490184+0000","last_active":"2024-03-26T04:08:04.555695+0000","last_peered":"2024-03-26T04:08:04.555695+0000","last_clean":"2024-03-26T04:08:04.555695+0000","last_became_active":"2024-03-26T04:05:29.538915+0000","last_became_peered":"2024-03-26T04:05:29.538915+0000","last_unstale":"2024-03-26T04:08:04.555695+0000","last_undegraded":"2024-03-26T04:08:04.555695+0000","last_fullsized":"2024-03-26T04:08:04.555695+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.490105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.490105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:05.057235+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"241'10","reported_seq":317,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589374+0000","last_change":"2024-03-26T04:07:59.722722+0000","last_active":"2024-03-26T04:08:04.589374+0000","last_peered":"2024-03-26T04:08:04.589374+0000","last_clean":"2024-03-26T04:08:04.589374+0000","last_became_active":"2024-03-26T04:05:29.520584+0000","last_became_peered":"2024-03-26T04:05:29.520584+0000","last_unstale":"2024-03-26T04:08:04.589374+0000","last_undegraded":"2024-03-26T04:08:04.589374+0000","last_fullsized":"2024-03-26T04:08:04.589374+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:00:38.458453+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:00:38.458453+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588312+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":9,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":439,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554669+0000","last_change":"2024-03-26T04:05:29.537044+0000","last_active":"2024-03-26T04:08:04.554669+0000","last_peered":"2024-03-26T04:08:04.554669+0000","last_clean":"2024-03-26T04:08:04.554669+0000","last_became_active":"2024-03-26T04:05:29.536699+0000","last_became_peered":"2024-03-26T04:05:29.536699+0000","last_unstale":"2024-03-26T04:08:04.554669+0000","last_undegraded":"2024-03-26T04:08:04.554669+0000","last_fullsized":"2024-03-26T04:08:04.554669+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:29.635002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:00:29.635002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":813,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106333+0000","last_change":"2024-03-26T04:05:48.811855+0000","last_active":"2024-03-26T04:08:09.106333+0000","last_peered":"2024-03-26T04:08:09.106333+0000","last_clean":"2024-03-26T04:08:09.106333+0000","last_became_active":"2024-03-26T03:50:45.593234+0000","last_became_peered":"2024-03-26T03:50:45.593234+0000","last_unstale":"2024-03-26T04:08:09.106333+0000","last_undegraded":"2024-03-26T04:08:09.106333+0000","last_fullsized":"2024-03-26T04:08:09.106333+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:48.811706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:48.811706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":533,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590500+0000","last_change":"2024-03-26T04:07:39.423667+0000","last_active":"2024-03-26T04:08:04.590500+0000","last_peered":"2024-03-26T04:08:04.590500+0000","last_clean":"2024-03-26T04:08:04.590500+0000","last_became_active":"2024-03-26T04:05:29.539760+0000","last_became_peered":"2024-03-26T04:05:29.539760+0000","last_unstale":"2024-03-26T04:08:04.590500+0000","last_undegraded":"2024-03-26T04:08:04.590500+0000","last_fullsized":"2024-03-26T04:08:04.590500+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:39.423550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.423550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:56.251695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"241'31","reported_seq":457,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554170+0000","last_change":"2024-03-26T04:05:29.545345+0000","last_active":"2024-03-26T04:08:04.554170+0000","last_peered":"2024-03-26T04:08:04.554170+0000","last_clean":"2024-03-26T04:08:04.554170+0000","last_became_active":"2024-03-26T04:05:29.542754+0000","last_became_peered":"2024-03-26T04:05:29.542754+0000","last_unstale":"2024-03-26T04:08:04.554170+0000","last_undegraded":"2024-03-26T04:08:04.554170+0000","last_fullsized":"2024-03-26T04:08:04.554170+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:01:19.700471+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:01:19.700471+0000","objects_scrubbed":6,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":796,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554087+0000","last_change":"2024-03-26T04:03:07.149962+0000","last_active":"2024-03-26T04:08:04.554087+0000","last_peered":"2024-03-26T04:08:04.554087+0000","last_clean":"2024-03-26T04:08:04.554087+0000","last_became_active":"2024-03-26T04:02:52.506113+0000","last_became_peered":"2024-03-26T04:02:52.506113+0000","last_unstale":"2024-03-26T04:08:04.554087+0000","last_undegraded":"2024-03-26T04:08:04.554087+0000","last_fullsized":"2024-03-26T04:08:04.554087+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:07.149881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.149881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":381,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554779+0000","last_change":"2024-03-26T04:04:51.740064+0000","last_active":"2024-03-26T04:08:04.554779+0000","last_peered":"2024-03-26T04:08:04.554779+0000","last_clean":"2024-03-26T04:08:04.554779+0000","last_became_active":"2024-03-26T04:02:52.489626+0000","last_became_peered":"2024-03-26T04:02:52.489626+0000","last_unstale":"2024-03-26T04:08:04.554779+0000","last_undegraded":"2024-03-26T04:08:04.554779+0000","last_fullsized":"2024-03-26T04:08:04.554779+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.739925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.739925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":841,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106813+0000","last_change":"2024-03-26T04:07:49.951615+0000","last_active":"2024-03-26T04:08:09.106813+0000","last_peered":"2024-03-26T04:08:09.106813+0000","last_clean":"2024-03-26T04:08:09.106813+0000","last_became_active":"2024-03-26T04:02:52.498923+0000","last_became_peered":"2024-03-26T04:02:52.498923+0000","last_unstale":"2024-03-26T04:08:09.106813+0000","last_undegraded":"2024-03-26T04:08:09.106813+0000","last_fullsized":"2024-03-26T04:08:09.106813+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.951520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.951520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.144662+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"241'19","reported_seq":843,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590004+0000","last_change":"2024-03-26T04:05:29.535609+0000","last_active":"2024-03-26T04:08:04.590004+0000","last_peered":"2024-03-26T04:08:04.590004+0000","last_clean":"2024-03-26T04:08:04.590004+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:08:04.590004+0000","last_undegraded":"2024-03-26T04:08:04.590004+0000","last_fullsized":"2024-03-26T04:08:04.590004+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:01:36.382121+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.382121+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588384+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":417,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554492+0000","last_change":"2024-03-26T04:05:29.538705+0000","last_active":"2024-03-26T04:08:04.554492+0000","last_peered":"2024-03-26T04:08:04.554492+0000","last_clean":"2024-03-26T04:08:04.554492+0000","last_became_active":"2024-03-26T04:05:29.537974+0000","last_became_peered":"2024-03-26T04:05:29.537974+0000","last_unstale":"2024-03-26T04:08:04.554492+0000","last_undegraded":"2024-03-26T04:08:04.554492+0000","last_fullsized":"2024-03-26T04:08:04.554492+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:21.709457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:01:21.709457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":810,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589899+0000","last_change":"2024-03-26T04:03:57.362649+0000","last_active":"2024-03-26T04:08:04.589899+0000","last_peered":"2024-03-26T04:08:04.589899+0000","last_clean":"2024-03-26T04:08:04.589899+0000","last_became_active":"2024-03-26T04:02:52.512541+0000","last_became_peered":"2024-03-26T04:02:52.512541+0000","last_unstale":"2024-03-26T04:08:04.589899+0000","last_undegraded":"2024-03-26T04:08:04.589899+0000","last_fullsized":"2024-03-26T04:08:04.589899+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:57.362566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:57.362566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588483+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":810,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553293+0000","last_change":"2024-03-26T04:05:29.536775+0000","last_active":"2024-03-26T04:08:09.553293+0000","last_peered":"2024-03-26T04:08:09.553293+0000","last_clean":"2024-03-26T04:08:09.553293+0000","last_became_active":"2024-03-26T04:05:29.536500+0000","last_became_peered":"2024-03-26T04:05:29.536500+0000","last_unstale":"2024-03-26T04:08:09.553293+0000","last_undegraded":"2024-03-26T04:08:09.553293+0000","last_fullsized":"2024-03-26T04:08:09.553293+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:01:35.104690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"241'19","reported_seq":868,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554007+0000","last_change":"2024-03-26T04:05:29.521309+0000","last_active":"2024-03-26T04:08:09.554007+0000","last_peered":"2024-03-26T04:08:09.554007+0000","last_clean":"2024-03-26T04:08:09.554007+0000","last_became_active":"2024-03-26T04:05:29.520817+0000","last_became_peered":"2024-03-26T04:05:29.520817+0000","last_unstale":"2024-03-26T04:08:09.554007+0000","last_undegraded":"2024-03-26T04:08:09.554007+0000","last_fullsized":"2024-03-26T04:08:09.554007+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:02:51.525641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:51.525641+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394267+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":792,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555289+0000","last_change":"2024-03-26T04:04:58.704933+0000","last_active":"2024-03-26T04:08:04.555289+0000","last_peered":"2024-03-26T04:08:04.555289+0000","last_clean":"2024-03-26T04:08:04.555289+0000","last_became_active":"2024-03-26T04:02:52.497109+0000","last_became_peered":"2024-03-26T04:02:52.497109+0000","last_unstale":"2024-03-26T04:08:04.555289+0000","last_undegraded":"2024-03-26T04:08:04.555289+0000","last_fullsized":"2024-03-26T04:08:04.555289+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:04:58.704813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":448,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590593+0000","last_change":"2024-03-26T04:03:53.367019+0000","last_active":"2024-03-26T04:08:04.590593+0000","last_peered":"2024-03-26T04:08:04.590593+0000","last_clean":"2024-03-26T04:08:04.590593+0000","last_became_active":"2024-03-26T04:02:52.511718+0000","last_became_peered":"2024-03-26T04:02:52.511718+0000","last_unstale":"2024-03-26T04:08:04.590593+0000","last_undegraded":"2024-03-26T04:08:04.590593+0000","last_fullsized":"2024-03-26T04:08:04.590593+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:53.366906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.366906+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":824,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106585+0000","last_change":"2024-03-26T04:07:32.845599+0000","last_active":"2024-03-26T04:08:09.106585+0000","last_peered":"2024-03-26T04:08:09.106585+0000","last_clean":"2024-03-26T04:08:09.106585+0000","last_became_active":"2024-03-26T04:05:29.533466+0000","last_became_peered":"2024-03-26T04:05:29.533466+0000","last_unstale":"2024-03-26T04:08:09.106585+0000","last_undegraded":"2024-03-26T04:08:09.106585+0000","last_fullsized":"2024-03-26T04:08:09.106585+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:32.845486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:32.845486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:54.200681+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"422'39","reported_seq":358,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590473+0000","last_change":"2024-03-26T04:07:42.462298+0000","last_active":"2024-03-26T04:08:04.590473+0000","last_peered":"2024-03-26T04:08:04.590473+0000","last_clean":"2024-03-26T04:08:04.590473+0000","last_became_active":"2024-03-26T04:05:29.540006+0000","last_became_peered":"2024-03-26T04:05:29.540006+0000","last_unstale":"2024-03-26T04:08:04.590473+0000","last_undegraded":"2024-03-26T04:08:04.590473+0000","last_fullsized":"2024-03-26T04:08:04.590473+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'39","last_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.462212+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:52.583235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":39,"num_write":32,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":798,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554992+0000","last_change":"2024-03-26T04:05:29.522233+0000","last_active":"2024-03-26T04:08:09.554992+0000","last_peered":"2024-03-26T04:08:09.554992+0000","last_clean":"2024-03-26T04:08:09.554992+0000","last_became_active":"2024-03-26T04:05:29.521886+0000","last_became_peered":"2024-03-26T04:05:29.521886+0000","last_unstale":"2024-03-26T04:08:09.554992+0000","last_undegraded":"2024-03-26T04:08:09.554992+0000","last_fullsized":"2024-03-26T04:08:09.554992+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:27.428232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:27.428232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394177+0000","scrub_duration":145,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"379'14","reported_seq":224,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661911+0000","last_change":"2024-03-26T04:05:29.541223+0000","last_active":"2024-03-26T04:08:07.661911+0000","last_peered":"2024-03-26T04:08:07.661911+0000","last_clean":"2024-03-26T04:08:07.661911+0000","last_became_active":"2024-03-26T04:05:29.540855+0000","last_became_peered":"2024-03-26T04:05:29.540855+0000","last_unstale":"2024-03-26T04:08:07.661911+0000","last_undegraded":"2024-03-26T04:08:07.661911+0000","last_fullsized":"2024-03-26T04:08:07.661911+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'12","last_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_deep_scrub":"334'12","last_deep_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.907441+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":837,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554566+0000","last_change":"2024-03-26T04:04:17.541024+0000","last_active":"2024-03-26T04:08:04.554566+0000","last_peered":"2024-03-26T04:08:04.554566+0000","last_clean":"2024-03-26T04:08:04.554566+0000","last_became_active":"2024-03-26T04:02:52.492361+0000","last_became_peered":"2024-03-26T04:02:52.492361+0000","last_unstale":"2024-03-26T04:08:04.554566+0000","last_undegraded":"2024-03-26T04:08:04.554566+0000","last_fullsized":"2024-03-26T04:08:04.554566+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.538770+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4041,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"241'26","reported_seq":763,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107042+0000","last_change":"2024-03-26T04:06:21.728756+0000","last_active":"2024-03-26T04:08:09.107042+0000","last_peered":"2024-03-26T04:08:09.107042+0000","last_clean":"2024-03-26T04:08:09.107042+0000","last_became_active":"2024-03-26T04:02:52.505421+0000","last_became_peered":"2024-03-26T04:02:52.505421+0000","last_unstale":"2024-03-26T04:08:09.107042+0000","last_undegraded":"2024-03-26T04:08:09.107042+0000","last_fullsized":"2024-03-26T04:08:09.107042+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:06:21.728608+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:06:21.728608+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":708,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.662131+0000","last_change":"2024-03-26T04:05:29.548434+0000","last_active":"2024-03-26T04:08:07.662131+0000","last_peered":"2024-03-26T04:08:07.662131+0000","last_clean":"2024-03-26T04:08:07.662131+0000","last_became_active":"2024-03-26T04:05:29.546602+0000","last_became_peered":"2024-03-26T04:05:29.546602+0000","last_unstale":"2024-03-26T04:08:07.662131+0000","last_undegraded":"2024-03-26T04:08:07.662131+0000","last_fullsized":"2024-03-26T04:08:07.662131+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:01:13.279325+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:01:13.279325+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":372,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.591047+0000","last_change":"2024-03-26T04:03:58.337609+0000","last_active":"2024-03-26T04:08:04.591047+0000","last_peered":"2024-03-26T04:08:04.591047+0000","last_clean":"2024-03-26T04:08:04.591047+0000","last_became_active":"2024-03-26T04:02:52.505042+0000","last_became_peered":"2024-03-26T04:02:52.505042+0000","last_unstale":"2024-03-26T04:08:04.591047+0000","last_undegraded":"2024-03-26T04:08:04.591047+0000","last_fullsized":"2024-03-26T04:08:04.591047+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:58.337496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:58.337496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588416+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":504,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553749+0000","last_change":"2024-03-26T04:04:57.523173+0000","last_active":"2024-03-26T04:08:09.553749+0000","last_peered":"2024-03-26T04:08:09.553749+0000","last_clean":"2024-03-26T04:08:09.553749+0000","last_became_active":"2024-03-26T03:58:40.497037+0000","last_became_peered":"2024-03-26T03:58:40.497037+0000","last_unstale":"2024-03-26T04:08:09.553749+0000","last_undegraded":"2024-03-26T04:08:09.553749+0000","last_fullsized":"2024-03-26T04:08:09.553749+0000","mapping_epoch":236,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:57.522794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:57.522794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394478+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"422'33","reported_seq":906,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555657+0000","last_change":"2024-03-26T04:07:45.584228+0000","last_active":"2024-03-26T04:08:04.555657+0000","last_peered":"2024-03-26T04:08:04.555657+0000","last_clean":"2024-03-26T04:08:04.555657+0000","last_became_active":"2024-03-26T04:05:29.537550+0000","last_became_peered":"2024-03-26T04:05:29.537550+0000","last_unstale":"2024-03-26T04:08:04.555657+0000","last_undegraded":"2024-03-26T04:08:04.555657+0000","last_fullsized":"2024-03-26T04:08:04.555657+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'33","last_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.584130+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:57.820980+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":44,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":824,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106809+0000","last_change":"2024-03-26T04:07:51.009058+0000","last_active":"2024-03-26T04:08:09.106809+0000","last_peered":"2024-03-26T04:08:09.106809+0000","last_clean":"2024-03-26T04:08:09.106809+0000","last_became_active":"2024-03-26T04:01:35.799877+0000","last_became_peered":"2024-03-26T04:01:35.799877+0000","last_unstale":"2024-03-26T04:08:09.106809+0000","last_undegraded":"2024-03-26T04:08:09.106809+0000","last_fullsized":"2024-03-26T04:08:09.106809+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:51.008927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:51.008927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:09.565335+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":526,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590686+0000","last_change":"2024-03-26T04:04:10.223664+0000","last_active":"2024-03-26T04:08:04.590686+0000","last_peered":"2024-03-26T04:08:04.590686+0000","last_clean":"2024-03-26T04:08:04.590686+0000","last_became_active":"2024-03-26T04:02:52.502225+0000","last_became_peered":"2024-03-26T04:02:52.502225+0000","last_unstale":"2024-03-26T04:08:04.590686+0000","last_undegraded":"2024-03-26T04:08:04.590686+0000","last_fullsized":"2024-03-26T04:08:04.590686+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:04:10.223350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588450+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":759,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554264+0000","last_change":"2024-03-26T04:05:29.540410+0000","last_active":"2024-03-26T04:08:04.554264+0000","last_peered":"2024-03-26T04:08:04.554264+0000","last_clean":"2024-03-26T04:08:04.554264+0000","last_became_active":"2024-03-26T04:05:29.540070+0000","last_became_peered":"2024-03-26T04:05:29.540070+0000","last_unstale":"2024-03-26T04:08:04.554264+0000","last_undegraded":"2024-03-26T04:08:04.554264+0000","last_fullsized":"2024-03-26T04:08:04.554264+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:48.062863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:02:48.062863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"241'19","reported_seq":252,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.591103+0000","last_change":"2024-03-26T04:07:48.515570+0000","last_active":"2024-03-26T04:08:04.591103+0000","last_peered":"2024-03-26T04:08:04.591103+0000","last_clean":"2024-03-26T04:08:04.591103+0000","last_became_active":"2024-03-26T04:05:29.539596+0000","last_became_peered":"2024-03-26T04:05:29.539596+0000","last_unstale":"2024-03-26T04:08:04.591103+0000","last_undegraded":"2024-03-26T04:08:04.591103+0000","last_fullsized":"2024-03-26T04:08:04.591103+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:07:48.515443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.515443+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:50.000297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":21,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590500+0000","last_change":"2024-03-26T04:08:02.634036+0000","last_active":"2024-03-26T04:08:04.590500+0000","last_peered":"2024-03-26T04:08:04.590500+0000","last_clean":"2024-03-26T04:08:04.590500+0000","last_became_active":"2024-03-26T04:08:02.633750+0000","last_became_peered":"2024-03-26T04:08:02.633750+0000","last_unstale":"2024-03-26T04:08:04.590500+0000","last_undegraded":"2024-03-26T04:08:04.590500+0000","last_fullsized":"2024-03-26T04:08:04.590500+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.648311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":512,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554744+0000","last_change":"2024-03-26T04:05:29.537569+0000","last_active":"2024-03-26T04:08:04.554744+0000","last_peered":"2024-03-26T04:08:04.554744+0000","last_clean":"2024-03-26T04:08:04.554744+0000","last_became_active":"2024-03-26T04:05:29.537211+0000","last_became_peered":"2024-03-26T04:05:29.537211+0000","last_unstale":"2024-03-26T04:08:04.554744+0000","last_undegraded":"2024-03-26T04:08:04.554744+0000","last_fullsized":"2024-03-26T04:08:04.554744+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:50.068294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:50.068294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":828,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107442+0000","last_change":"2024-03-26T04:06:27.796450+0000","last_active":"2024-03-26T04:08:09.107442+0000","last_peered":"2024-03-26T04:08:09.107442+0000","last_clean":"2024-03-26T04:08:09.107442+0000","last_became_active":"2024-03-26T03:48:44.019072+0000","last_became_peered":"2024-03-26T03:48:44.019072+0000","last_unstale":"2024-03-26T04:08:09.107442+0000","last_undegraded":"2024-03-26T04:08:09.107442+0000","last_fullsized":"2024-03-26T04:08:09.107442+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-03-26T04:06:27.796333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:27.796333+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"241'39","reported_seq":796,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106281+0000","last_change":"2024-03-26T04:07:25.796444+0000","last_active":"2024-03-26T04:08:09.106281+0000","last_peered":"2024-03-26T04:08:09.106281+0000","last_clean":"2024-03-26T04:08:09.106281+0000","last_became_active":"2024-03-26T04:02:52.505008+0000","last_became_peered":"2024-03-26T04:02:52.505008+0000","last_unstale":"2024-03-26T04:08:09.106281+0000","last_undegraded":"2024-03-26T04:08:09.106281+0000","last_fullsized":"2024-03-26T04:08:09.106281+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:07:25.796341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.796341+0000","objects_scrubbed":9,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:27.516967+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"100'9","reported_seq":1067,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106586+0000","last_change":"2024-03-26T04:07:54.941310+0000","last_active":"2024-03-26T04:08:09.106586+0000","last_peered":"2024-03-26T04:08:09.106586+0000","last_clean":"2024-03-26T04:08:09.106586+0000","last_became_active":"2024-03-26T03:50:45.590734+0000","last_became_peered":"2024-03-26T03:50:45.590734+0000","last_unstale":"2024-03-26T04:08:09.106586+0000","last_undegraded":"2024-03-26T04:08:09.106586+0000","last_fullsized":"2024-03-26T04:08:09.106586+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:07:54.941219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.941219+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:05.480364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555109+0000","last_change":"2024-03-26T04:08:02.632658+0000","last_active":"2024-03-26T04:08:04.555109+0000","last_peered":"2024-03-26T04:08:04.555109+0000","last_clean":"2024-03-26T04:08:04.555109+0000","last_became_active":"2024-03-26T04:08:02.631983+0000","last_became_peered":"2024-03-26T04:08:02.631983+0000","last_unstale":"2024-03-26T04:08:04.555109+0000","last_undegraded":"2024-03-26T04:08:04.555109+0000","last_fullsized":"2024-03-26T04:08:04.555109+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:19.417923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":393,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106681+0000","last_change":"2024-03-26T04:06:01.998434+0000","last_active":"2024-03-26T04:08:09.106681+0000","last_peered":"2024-03-26T04:08:09.106681+0000","last_clean":"2024-03-26T04:08:09.106681+0000","last_became_active":"2024-03-26T04:02:52.505076+0000","last_became_peered":"2024-03-26T04:02:52.505076+0000","last_unstale":"2024-03-26T04:08:09.106681+0000","last_undegraded":"2024-03-26T04:08:09.106681+0000","last_fullsized":"2024-03-26T04:08:09.106681+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:01.998326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:06:01.998326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":141,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":815,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107121+0000","last_change":"2024-03-26T04:06:06.861624+0000","last_active":"2024-03-26T04:08:09.107121+0000","last_peered":"2024-03-26T04:08:09.107121+0000","last_clean":"2024-03-26T04:08:09.107121+0000","last_became_active":"2024-03-26T03:48:44.017390+0000","last_became_peered":"2024-03-26T03:48:44.017390+0000","last_unstale":"2024-03-26T04:08:09.107121+0000","last_undegraded":"2024-03-26T04:08:09.107121+0000","last_fullsized":"2024-03-26T04:08:09.107121+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:06.861506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:06:06.861506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"241'30","reported_seq":913,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553911+0000","last_change":"2024-03-26T04:04:19.443761+0000","last_active":"2024-03-26T04:08:09.553911+0000","last_peered":"2024-03-26T04:08:09.553911+0000","last_clean":"2024-03-26T04:08:09.553911+0000","last_became_active":"2024-03-26T04:02:52.513309+0000","last_became_peered":"2024-03-26T04:02:52.513309+0000","last_unstale":"2024-03-26T04:08:09.553911+0000","last_undegraded":"2024-03-26T04:08:09.553911+0000","last_fullsized":"2024-03-26T04:08:09.553911+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:04:19.443608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.443608+0000","objects_scrubbed":7,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394426+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":41,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":718,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660998+0000","last_change":"2024-03-26T04:07:53.660073+0000","last_active":"2024-03-26T04:08:07.660998+0000","last_peered":"2024-03-26T04:08:07.660998+0000","last_clean":"2024-03-26T04:08:07.660998+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:08:07.660998+0000","last_undegraded":"2024-03-26T04:08:07.660998+0000","last_fullsized":"2024-03-26T04:08:07.660998+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:53.659974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:53.659974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:57.025416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554952+0000","last_change":"2024-03-26T04:08:02.633179+0000","last_active":"2024-03-26T04:08:04.554952+0000","last_peered":"2024-03-26T04:08:04.554952+0000","last_clean":"2024-03-26T04:08:04.554952+0000","last_became_active":"2024-03-26T04:08:02.632854+0000","last_became_peered":"2024-03-26T04:08:02.632854+0000","last_unstale":"2024-03-26T04:08:04.554952+0000","last_undegraded":"2024-03-26T04:08:04.554952+0000","last_fullsized":"2024-03-26T04:08:04.554952+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:20.675835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":665,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660744+0000","last_change":"2024-03-26T04:05:29.543839+0000","last_active":"2024-03-26T04:08:07.660744+0000","last_peered":"2024-03-26T04:08:07.660744+0000","last_clean":"2024-03-26T04:08:07.660744+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:08:07.660744+0000","last_undegraded":"2024-03-26T04:08:07.660744+0000","last_fullsized":"2024-03-26T04:08:07.660744+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:37.891864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.891864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682116+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":711,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661279+0000","last_change":"2024-03-26T04:05:29.535670+0000","last_active":"2024-03-26T04:08:07.661279+0000","last_peered":"2024-03-26T04:08:07.661279+0000","last_clean":"2024-03-26T04:08:07.661279+0000","last_became_active":"2024-03-26T04:05:29.534559+0000","last_became_peered":"2024-03-26T04:05:29.534559+0000","last_unstale":"2024-03-26T04:08:07.661279+0000","last_undegraded":"2024-03-26T04:08:07.661279+0000","last_fullsized":"2024-03-26T04:08:07.661279+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.965300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682239+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":602,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554026+0000","last_change":"2024-03-26T04:04:14.382770+0000","last_active":"2024-03-26T04:08:09.554026+0000","last_peered":"2024-03-26T04:08:09.554026+0000","last_clean":"2024-03-26T04:08:09.554026+0000","last_became_active":"2024-03-26T04:02:52.500569+0000","last_became_peered":"2024-03-26T04:02:52.500569+0000","last_unstale":"2024-03-26T04:08:09.554026+0000","last_undegraded":"2024-03-26T04:08:09.554026+0000","last_fullsized":"2024-03-26T04:08:09.554026+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:14.382655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:14.382655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394335+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"356'51","reported_seq":879,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589835+0000","last_change":"2024-03-26T04:04:03.371506+0000","last_active":"2024-03-26T04:08:04.589835+0000","last_peered":"2024-03-26T04:08:04.589835+0000","last_clean":"2024-03-26T04:08:04.589835+0000","last_became_active":"2024-03-26T04:01:35.818705+0000","last_became_peered":"2024-03-26T04:01:35.818705+0000","last_unstale":"2024-03-26T04:08:04.589835+0000","last_undegraded":"2024-03-26T04:08:04.589835+0000","last_fullsized":"2024-03-26T04:08:04.589835+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'51","last_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.371335+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588554+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":64,"num_read_kb":52,"num_write":42,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":21,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106523+0000","last_change":"2024-03-26T04:08:02.630906+0000","last_active":"2024-03-26T04:08:09.106523+0000","last_peered":"2024-03-26T04:08:09.106523+0000","last_clean":"2024-03-26T04:08:09.106523+0000","last_became_active":"2024-03-26T04:08:02.630589+0000","last_became_peered":"2024-03-26T04:08:02.630589+0000","last_unstale":"2024-03-26T04:08:09.106523+0000","last_undegraded":"2024-03-26T04:08:09.106523+0000","last_fullsized":"2024-03-26T04:08:09.106523+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.349873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":776,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554692+0000","last_change":"2024-03-26T04:05:29.543248+0000","last_active":"2024-03-26T04:08:04.554692+0000","last_peered":"2024-03-26T04:08:04.554692+0000","last_clean":"2024-03-26T04:08:04.554692+0000","last_became_active":"2024-03-26T04:05:29.541737+0000","last_became_peered":"2024-03-26T04:05:29.541737+0000","last_unstale":"2024-03-26T04:08:04.554692+0000","last_undegraded":"2024-03-26T04:08:04.554692+0000","last_fullsized":"2024-03-26T04:08:04.554692+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:00:22.410420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:00:22.410420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":727,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660955+0000","last_change":"2024-03-26T04:07:37.592031+0000","last_active":"2024-03-26T04:08:07.660955+0000","last_peered":"2024-03-26T04:08:07.660955+0000","last_clean":"2024-03-26T04:08:07.660955+0000","last_became_active":"2024-03-26T04:05:29.524644+0000","last_became_peered":"2024-03-26T04:05:29.524644+0000","last_unstale":"2024-03-26T04:08:07.660955+0000","last_undegraded":"2024-03-26T04:08:07.660955+0000","last_fullsized":"2024-03-26T04:08:07.660955+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.591946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.591946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.371327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":557,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590052+0000","last_change":"2024-03-26T04:07:34.405175+0000","last_active":"2024-03-26T04:08:04.590052+0000","last_peered":"2024-03-26T04:08:04.590052+0000","last_clean":"2024-03-26T04:08:04.590052+0000","last_became_active":"2024-03-26T04:05:29.531951+0000","last_became_peered":"2024-03-26T04:05:29.531951+0000","last_unstale":"2024-03-26T04:08:04.590052+0000","last_undegraded":"2024-03-26T04:08:04.590052+0000","last_fullsized":"2024-03-26T04:08:04.590052+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.404972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.404972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.885121+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"241'21","reported_seq":873,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555165+0000","last_change":"2024-03-26T04:04:55.743092+0000","last_active":"2024-03-26T04:08:04.555165+0000","last_peered":"2024-03-26T04:08:04.555165+0000","last_clean":"2024-03-26T04:08:04.555165+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:08:04.555165+0000","last_undegraded":"2024-03-26T04:08:04.555165+0000","last_fullsized":"2024-03-26T04:08:04.555165+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:04:55.742967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.742967+0000","objects_scrubbed":5,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":35,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590098+0000","last_change":"2024-03-26T04:08:02.633493+0000","last_active":"2024-03-26T04:08:04.590098+0000","last_peered":"2024-03-26T04:08:04.590098+0000","last_clean":"2024-03-26T04:08:04.590098+0000","last_became_active":"2024-03-26T04:08:02.633214+0000","last_became_peered":"2024-03-26T04:08:02.633214+0000","last_unstale":"2024-03-26T04:08:04.590098+0000","last_undegraded":"2024-03-26T04:08:04.590098+0000","last_fullsized":"2024-03-26T04:08:04.590098+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:26.202173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":598,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554207+0000","last_change":"2024-03-26T04:05:29.538976+0000","last_active":"2024-03-26T04:08:04.554207+0000","last_peered":"2024-03-26T04:08:04.554207+0000","last_clean":"2024-03-26T04:08:04.554207+0000","last_became_active":"2024-03-26T04:05:29.538433+0000","last_became_peered":"2024-03-26T04:05:29.538433+0000","last_unstale":"2024-03-26T04:08:04.554207+0000","last_undegraded":"2024-03-26T04:08:04.554207+0000","last_fullsized":"2024-03-26T04:08:04.554207+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:37.939498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.939498+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":390,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554373+0000","last_change":"2024-03-26T04:04:03.264229+0000","last_active":"2024-03-26T04:08:09.554373+0000","last_peered":"2024-03-26T04:08:09.554373+0000","last_clean":"2024-03-26T04:08:09.554373+0000","last_became_active":"2024-03-26T04:02:52.520574+0000","last_became_peered":"2024-03-26T04:02:52.520574+0000","last_unstale":"2024-03-26T04:08:09.554373+0000","last_undegraded":"2024-03-26T04:08:09.554373+0000","last_fullsized":"2024-03-26T04:08:09.554373+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:03.264133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.264133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394389+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":682,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589062+0000","last_change":"2024-03-26T04:04:17.110459+0000","last_active":"2024-03-26T04:08:04.589062+0000","last_peered":"2024-03-26T04:08:04.589062+0000","last_clean":"2024-03-26T04:08:04.589062+0000","last_became_active":"2024-03-26T04:02:52.495493+0000","last_became_peered":"2024-03-26T04:02:52.495493+0000","last_unstale":"2024-03-26T04:08:04.589062+0000","last_undegraded":"2024-03-26T04:08:04.589062+0000","last_fullsized":"2024-03-26T04:08:04.589062+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:04:17.110315+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.110315+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588571+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":802,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553541+0000","last_change":"2024-03-26T04:04:52.483535+0000","last_active":"2024-03-26T04:08:09.553541+0000","last_peered":"2024-03-26T04:08:09.553541+0000","last_clean":"2024-03-26T04:08:09.553541+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:08:09.553541+0000","last_undegraded":"2024-03-26T04:08:09.553541+0000","last_fullsized":"2024-03-26T04:08:09.553541+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:52.483429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:04:52.483429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394541+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"379'51","reported_seq":1194,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590407+0000","last_change":"2024-03-26T04:05:29.540308+0000","last_active":"2024-03-26T04:08:04.590407+0000","last_peered":"2024-03-26T04:08:04.590407+0000","last_clean":"2024-03-26T04:08:04.590407+0000","last_became_active":"2024-03-26T04:05:29.539914+0000","last_became_peered":"2024-03-26T04:05:29.539914+0000","last_unstale":"2024-03-26T04:08:04.590407+0000","last_undegraded":"2024-03-26T04:08:04.590407+0000","last_fullsized":"2024-03-26T04:08:04.590407+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'48","last_scrub_stamp":"2024-03-26T04:01:23.451122+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:23.451122+0000","objects_scrubbed":11,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588367+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":75,"num_read_kb":59,"num_write":44,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":788,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555179+0000","last_change":"2024-03-26T04:05:04.831962+0000","last_active":"2024-03-26T04:08:04.555179+0000","last_peered":"2024-03-26T04:08:04.555179+0000","last_clean":"2024-03-26T04:08:04.555179+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:08:04.555179+0000","last_undegraded":"2024-03-26T04:08:04.555179+0000","last_fullsized":"2024-03-26T04:08:04.555179+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:04.831457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:04.831457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.553934+0000","last_change":"2024-03-26T04:08:02.632925+0000","last_active":"2024-03-26T04:08:04.553934+0000","last_peered":"2024-03-26T04:08:04.553934+0000","last_clean":"2024-03-26T04:08:04.553934+0000","last_became_active":"2024-03-26T04:08:02.631353+0000","last_became_peered":"2024-03-26T04:08:02.631353+0000","last_unstale":"2024-03-26T04:08:04.553934+0000","last_undegraded":"2024-03-26T04:08:04.553934+0000","last_fullsized":"2024-03-26T04:08:04.553934+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:13.114591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":630,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554433+0000","last_change":"2024-03-26T04:04:11.458855+0000","last_active":"2024-03-26T04:08:04.554433+0000","last_peered":"2024-03-26T04:08:04.554433+0000","last_clean":"2024-03-26T04:08:04.554433+0000","last_became_active":"2024-03-26T04:00:32.425416+0000","last_became_peered":"2024-03-26T04:00:32.425416+0000","last_unstale":"2024-03-26T04:08:04.554433+0000","last_undegraded":"2024-03-26T04:08:04.554433+0000","last_fullsized":"2024-03-26T04:08:04.554433+0000","mapping_epoch":276,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":277,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:11.458734+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:04:11.458734+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":838,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107071+0000","last_change":"2024-03-26T04:07:38.023167+0000","last_active":"2024-03-26T04:08:09.107071+0000","last_peered":"2024-03-26T04:08:09.107071+0000","last_clean":"2024-03-26T04:08:09.107071+0000","last_became_active":"2024-03-26T04:05:29.542557+0000","last_became_peered":"2024-03-26T04:05:29.542557+0000","last_unstale":"2024-03-26T04:08:09.107071+0000","last_undegraded":"2024-03-26T04:08:09.107071+0000","last_fullsized":"2024-03-26T04:08:09.107071+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.023066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:59.026959+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":809,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589441+0000","last_change":"2024-03-26T04:05:29.535985+0000","last_active":"2024-03-26T04:08:04.589441+0000","last_peered":"2024-03-26T04:08:04.589441+0000","last_clean":"2024-03-26T04:08:04.589441+0000","last_became_active":"2024-03-26T04:05:29.533776+0000","last_became_peered":"2024-03-26T04:05:29.533776+0000","last_unstale":"2024-03-26T04:08:04.589441+0000","last_undegraded":"2024-03-26T04:08:04.589441+0000","last_fullsized":"2024-03-26T04:08:04.589441+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.324845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.324845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588400+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":736,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661114+0000","last_change":"2024-03-26T04:05:29.544757+0000","last_active":"2024-03-26T04:08:07.661114+0000","last_peered":"2024-03-26T04:08:07.661114+0000","last_clean":"2024-03-26T04:08:07.661114+0000","last_became_active":"2024-03-26T04:05:29.543967+0000","last_became_peered":"2024-03-26T04:05:29.543967+0000","last_unstale":"2024-03-26T04:08:07.661114+0000","last_undegraded":"2024-03-26T04:08:07.661114+0000","last_fullsized":"2024-03-26T04:08:07.661114+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:27.325285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:01:27.325285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682030+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"371'29","reported_seq":869,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590811+0000","last_change":"2024-03-26T04:04:59.217984+0000","last_active":"2024-03-26T04:08:04.590811+0000","last_peered":"2024-03-26T04:08:04.590811+0000","last_clean":"2024-03-26T04:08:04.590811+0000","last_became_active":"2024-03-26T04:01:35.795965+0000","last_became_peered":"2024-03-26T04:01:35.795965+0000","last_unstale":"2024-03-26T04:08:04.590811+0000","last_undegraded":"2024-03-26T04:08:04.590811+0000","last_fullsized":"2024-03-26T04:08:04.590811+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'29","last_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:04:59.217868+0000","objects_scrubbed":6,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588655+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":26,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":730,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661582+0000","last_change":"2024-03-26T04:05:29.536479+0000","last_active":"2024-03-26T04:08:07.661582+0000","last_peered":"2024-03-26T04:08:07.661582+0000","last_clean":"2024-03-26T04:08:07.661582+0000","last_became_active":"2024-03-26T04:05:29.536165+0000","last_became_peered":"2024-03-26T04:05:29.536165+0000","last_unstale":"2024-03-26T04:08:07.661582+0000","last_undegraded":"2024-03-26T04:08:07.661582+0000","last_fullsized":"2024-03-26T04:08:07.661582+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.247532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.247532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682101+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589587+0000","last_change":"2024-03-26T04:08:02.629892+0000","last_active":"2024-03-26T04:08:04.589587+0000","last_peered":"2024-03-26T04:08:04.589587+0000","last_clean":"2024-03-26T04:08:04.589587+0000","last_became_active":"2024-03-26T04:08:02.629608+0000","last_became_peered":"2024-03-26T04:08:02.629608+0000","last_unstale":"2024-03-26T04:08:04.589587+0000","last_undegraded":"2024-03-26T04:08:04.589587+0000","last_fullsized":"2024-03-26T04:08:04.589587+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:08.542606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":530,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554489+0000","last_change":"2024-03-26T04:04:02.403906+0000","last_active":"2024-03-26T04:08:09.554489+0000","last_peered":"2024-03-26T04:08:09.554489+0000","last_clean":"2024-03-26T04:08:09.554489+0000","last_became_active":"2024-03-26T04:02:52.504066+0000","last_became_peered":"2024-03-26T04:02:52.504066+0000","last_unstale":"2024-03-26T04:08:09.554489+0000","last_undegraded":"2024-03-26T04:08:09.554489+0000","last_fullsized":"2024-03-26T04:08:09.554489+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:04:02.403784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394317+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":1090,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106073+0000","last_change":"2024-03-26T04:06:03.860318+0000","last_active":"2024-03-26T04:08:09.106073+0000","last_peered":"2024-03-26T04:08:09.106073+0000","last_clean":"2024-03-26T04:08:09.106073+0000","last_became_active":"2024-03-26T04:00:27.442487+0000","last_became_peered":"2024-03-26T04:00:27.442487+0000","last_unstale":"2024-03-26T04:08:09.106073+0000","last_undegraded":"2024-03-26T04:08:09.106073+0000","last_fullsized":"2024-03-26T04:08:09.106073+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":272,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:06:03.860237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:06:03.860237+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":424,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.552916+0000","last_change":"2024-03-26T04:04:56.545589+0000","last_active":"2024-03-26T04:08:09.552916+0000","last_peered":"2024-03-26T04:08:09.552916+0000","last_clean":"2024-03-26T04:08:09.552916+0000","last_became_active":"2024-03-26T04:02:52.499614+0000","last_became_peered":"2024-03-26T04:02:52.499614+0000","last_unstale":"2024-03-26T04:08:09.552916+0000","last_undegraded":"2024-03-26T04:08:09.552916+0000","last_fullsized":"2024-03-26T04:08:09.552916+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:56.545464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:56.545464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394576+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"241'29","reported_seq":1014,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554355+0000","last_change":"2024-03-26T04:04:20.432746+0000","last_active":"2024-03-26T04:08:09.554355+0000","last_peered":"2024-03-26T04:08:09.554355+0000","last_clean":"2024-03-26T04:08:09.554355+0000","last_became_active":"2024-03-26T03:50:45.592703+0000","last_became_peered":"2024-03-26T03:50:45.592703+0000","last_unstale":"2024-03-26T04:08:09.554355+0000","last_undegraded":"2024-03-26T04:08:09.554355+0000","last_fullsized":"2024-03-26T04:08:09.554355+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:04:20.432551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:20.432551+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394353+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":612,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.662096+0000","last_change":"2024-03-26T04:05:29.547970+0000","last_active":"2024-03-26T04:08:07.662096+0000","last_peered":"2024-03-26T04:08:07.662096+0000","last_clean":"2024-03-26T04:08:07.662096+0000","last_became_active":"2024-03-26T04:05:29.544566+0000","last_became_peered":"2024-03-26T04:05:29.544566+0000","last_unstale":"2024-03-26T04:08:07.662096+0000","last_undegraded":"2024-03-26T04:08:07.662096+0000","last_fullsized":"2024-03-26T04:08:07.662096+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:18.381181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:01:18.381181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682046+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554394+0000","last_change":"2024-03-26T04:08:02.629770+0000","last_active":"2024-03-26T04:08:04.554394+0000","last_peered":"2024-03-26T04:08:04.554394+0000","last_clean":"2024-03-26T04:08:04.554394+0000","last_became_active":"2024-03-26T04:08:02.629493+0000","last_became_peered":"2024-03-26T04:08:02.629493+0000","last_unstale":"2024-03-26T04:08:04.554394+0000","last_undegraded":"2024-03-26T04:08:04.554394+0000","last_fullsized":"2024-03-26T04:08:04.554394+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:14.710491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":684,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589989+0000","last_change":"2024-03-26T04:04:07.285956+0000","last_active":"2024-03-26T04:08:04.589989+0000","last_peered":"2024-03-26T04:08:04.589989+0000","last_clean":"2024-03-26T04:08:04.589989+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:08:04.589989+0000","last_undegraded":"2024-03-26T04:08:04.589989+0000","last_fullsized":"2024-03-26T04:08:04.589989+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:07.285700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:07.285700+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588468+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":816,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590063+0000","last_change":"2024-03-26T04:07:49.506391+0000","last_active":"2024-03-26T04:08:04.590063+0000","last_peered":"2024-03-26T04:08:04.590063+0000","last_clean":"2024-03-26T04:08:04.590063+0000","last_became_active":"2024-03-26T04:05:29.521425+0000","last_became_peered":"2024-03-26T04:05:29.521425+0000","last_unstale":"2024-03-26T04:08:04.590063+0000","last_undegraded":"2024-03-26T04:08:04.590063+0000","last_fullsized":"2024-03-26T04:08:04.590063+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.506266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.506266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.168658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":823,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554448+0000","last_change":"2024-03-26T04:03:56.329590+0000","last_active":"2024-03-26T04:08:04.554448+0000","last_peered":"2024-03-26T04:08:04.554448+0000","last_clean":"2024-03-26T04:08:04.554448+0000","last_became_active":"2024-03-26T04:02:52.508294+0000","last_became_peered":"2024-03-26T04:02:52.508294+0000","last_unstale":"2024-03-26T04:08:04.554448+0000","last_undegraded":"2024-03-26T04:08:04.554448+0000","last_fullsized":"2024-03-26T04:08:04.554448+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:56.329511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:56.329511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1993,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106080+0000","last_change":"2024-03-26T04:07:31.850944+0000","last_active":"2024-03-26T04:08:09.106080+0000","last_peered":"2024-03-26T04:08:09.106080+0000","last_clean":"2024-03-26T04:08:09.106080+0000","last_became_active":"2024-03-26T03:48:43.010169+0000","last_became_peered":"2024-03-26T03:48:43.010169+0000","last_unstale":"2024-03-26T04:08:09.106080+0000","last_undegraded":"2024-03-26T04:08:09.106080+0000","last_fullsized":"2024-03-26T04:08:09.106080+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:07:31.850790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.850790+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.959779+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1157,"num_read_kb":1152,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"422'30","reported_seq":834,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555408+0000","last_change":"2024-03-26T04:07:49.546831+0000","last_active":"2024-03-26T04:08:04.555408+0000","last_peered":"2024-03-26T04:08:04.555408+0000","last_clean":"2024-03-26T04:08:04.555408+0000","last_became_active":"2024-03-26T04:05:29.538986+0000","last_became_peered":"2024-03-26T04:05:29.538986+0000","last_unstale":"2024-03-26T04:08:04.555408+0000","last_undegraded":"2024-03-26T04:08:04.555408+0000","last_fullsized":"2024-03-26T04:08:04.555408+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'30","last_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_deep_scrub":"422'30","last_deep_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.546733+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:04.417043+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":501,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.591174+0000","last_change":"2024-03-26T04:05:01.160620+0000","last_active":"2024-03-26T04:08:04.591174+0000","last_peered":"2024-03-26T04:08:04.591174+0000","last_clean":"2024-03-26T04:08:04.591174+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:08:04.591174+0000","last_undegraded":"2024-03-26T04:08:04.591174+0000","last_fullsized":"2024-03-26T04:08:04.591174+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:05:01.160427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588670+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590256+0000","last_change":"2024-03-26T04:08:02.631311+0000","last_active":"2024-03-26T04:08:04.590256+0000","last_peered":"2024-03-26T04:08:04.590256+0000","last_clean":"2024-03-26T04:08:04.590256+0000","last_became_active":"2024-03-26T04:08:02.630915+0000","last_became_peered":"2024-03-26T04:08:02.630915+0000","last_unstale":"2024-03-26T04:08:04.590256+0000","last_undegraded":"2024-03-26T04:08:04.590256+0000","last_fullsized":"2024-03-26T04:08:04.590256+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:29.883724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":639,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555173+0000","last_change":"2024-03-26T04:04:53.785962+0000","last_active":"2024-03-26T04:08:04.555173+0000","last_peered":"2024-03-26T04:08:04.555173+0000","last_clean":"2024-03-26T04:08:04.555173+0000","last_became_active":"2024-03-26T04:02:55.535259+0000","last_became_peered":"2024-03-26T04:02:55.535259+0000","last_unstale":"2024-03-26T04:08:04.555173+0000","last_undegraded":"2024-03-26T04:08:04.555173+0000","last_fullsized":"2024-03-26T04:08:04.555173+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:53.785762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:53.785762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":797,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589672+0000","last_change":"2024-03-26T04:07:41.426704+0000","last_active":"2024-03-26T04:08:04.589672+0000","last_peered":"2024-03-26T04:08:04.589672+0000","last_clean":"2024-03-26T04:08:04.589672+0000","last_became_active":"2024-03-26T04:05:29.540971+0000","last_became_peered":"2024-03-26T04:05:29.540971+0000","last_unstale":"2024-03-26T04:08:04.589672+0000","last_undegraded":"2024-03-26T04:08:04.589672+0000","last_fullsized":"2024-03-26T04:08:04.589672+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:41.426604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:07:41.426604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:57.590188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":818,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590313+0000","last_change":"2024-03-26T04:07:45.465265+0000","last_active":"2024-03-26T04:08:04.590313+0000","last_peered":"2024-03-26T04:08:04.590313+0000","last_clean":"2024-03-26T04:08:04.590313+0000","last_became_active":"2024-03-26T04:05:29.532029+0000","last_became_peered":"2024-03-26T04:05:29.532029+0000","last_unstale":"2024-03-26T04:08:04.590313+0000","last_undegraded":"2024-03-26T04:08:04.590313+0000","last_fullsized":"2024-03-26T04:08:04.590313+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.465179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.465179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:00.714731+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"358'44","reported_seq":899,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555081+0000","last_change":"2024-03-26T04:04:19.574250+0000","last_active":"2024-03-26T04:08:04.555081+0000","last_peered":"2024-03-26T04:08:04.555081+0000","last_clean":"2024-03-26T04:08:04.555081+0000","last_became_active":"2024-03-26T04:01:35.824185+0000","last_became_peered":"2024-03-26T04:01:35.824185+0000","last_unstale":"2024-03-26T04:08:04.555081+0000","last_undegraded":"2024-03-26T04:08:04.555081+0000","last_fullsized":"2024-03-26T04:08:04.555081+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.571532+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":63,"num_read_kb":48,"num_write":38,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":742,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554516+0000","last_change":"2024-03-26T04:05:29.537174+0000","last_active":"2024-03-26T04:08:04.554516+0000","last_peered":"2024-03-26T04:08:04.554516+0000","last_clean":"2024-03-26T04:08:04.554516+0000","last_became_active":"2024-03-26T04:05:29.536647+0000","last_became_peered":"2024-03-26T04:05:29.536647+0000","last_unstale":"2024-03-26T04:08:04.554516+0000","last_undegraded":"2024-03-26T04:08:04.554516+0000","last_fullsized":"2024-03-26T04:08:04.554516+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.058039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.058039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":19,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661233+0000","last_change":"2024-03-26T04:08:02.629735+0000","last_active":"2024-03-26T04:08:07.661233+0000","last_peered":"2024-03-26T04:08:07.661233+0000","last_clean":"2024-03-26T04:08:07.661233+0000","last_became_active":"2024-03-26T04:08:02.629370+0000","last_became_peered":"2024-03-26T04:08:02.629370+0000","last_unstale":"2024-03-26T04:08:07.661233+0000","last_undegraded":"2024-03-26T04:08:07.661233+0000","last_fullsized":"2024-03-26T04:08:07.661233+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:25.131323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":489,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.662154+0000","last_change":"2024-03-26T04:07:29.526841+0000","last_active":"2024-03-26T04:08:07.662154+0000","last_peered":"2024-03-26T04:08:07.662154+0000","last_clean":"2024-03-26T04:08:07.662154+0000","last_became_active":"2024-03-26T04:05:29.536837+0000","last_became_peered":"2024-03-26T04:05:29.536837+0000","last_unstale":"2024-03-26T04:08:07.662154+0000","last_undegraded":"2024-03-26T04:08:07.662154+0000","last_fullsized":"2024-03-26T04:08:07.662154+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:29.526591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.526591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:46.626775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":805,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554152+0000","last_change":"2024-03-26T04:05:29.539255+0000","last_active":"2024-03-26T04:08:04.554152+0000","last_peered":"2024-03-26T04:08:04.554152+0000","last_clean":"2024-03-26T04:08:04.554152+0000","last_became_active":"2024-03-26T04:05:29.538900+0000","last_became_peered":"2024-03-26T04:05:29.538900+0000","last_unstale":"2024-03-26T04:08:04.554152+0000","last_undegraded":"2024-03-26T04:08:04.554152+0000","last_fullsized":"2024-03-26T04:08:04.554152+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:10.378037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:01:10.378037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":451,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661353+0000","last_change":"2024-03-26T04:05:29.544209+0000","last_active":"2024-03-26T04:08:07.661353+0000","last_peered":"2024-03-26T04:08:07.661353+0000","last_clean":"2024-03-26T04:08:07.661353+0000","last_became_active":"2024-03-26T04:05:29.542761+0000","last_became_peered":"2024-03-26T04:05:29.542761+0000","last_unstale":"2024-03-26T04:08:07.661353+0000","last_undegraded":"2024-03-26T04:08:07.661353+0000","last_fullsized":"2024-03-26T04:08:07.661353+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:03.918739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:03.918739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682205+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"241'26","reported_seq":861,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589961+0000","last_change":"2024-03-26T04:04:12.249958+0000","last_active":"2024-03-26T04:08:04.589961+0000","last_peered":"2024-03-26T04:08:04.589961+0000","last_clean":"2024-03-26T04:08:04.589961+0000","last_became_active":"2024-03-26T04:01:35.819144+0000","last_became_peered":"2024-03-26T04:01:35.819144+0000","last_unstale":"2024-03-26T04:08:04.589961+0000","last_undegraded":"2024-03-26T04:08:04.589961+0000","last_fullsized":"2024-03-26T04:08:04.589961+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:04:12.249693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:12.249693+0000","objects_scrubbed":6,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588518+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":33,"num_write":26,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"325'15","reported_seq":1075,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554878+0000","last_change":"2024-03-26T04:02:52.536740+0000","last_active":"2024-03-26T04:08:04.554878+0000","last_peered":"2024-03-26T04:08:04.554878+0000","last_clean":"2024-03-26T04:08:04.554878+0000","last_became_active":"2024-03-26T04:02:52.495925+0000","last_became_peered":"2024-03-26T04:02:52.495925+0000","last_unstale":"2024-03-26T04:08:04.554878+0000","last_undegraded":"2024-03-26T04:08:04.554878+0000","last_fullsized":"2024-03-26T04:08:04.554878+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'14","last_scrub_stamp":"2024-03-26T04:02:46.076593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:02:46.076593+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":24,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554503+0000","last_change":"2024-03-26T04:08:02.631111+0000","last_active":"2024-03-26T04:08:09.554503+0000","last_peered":"2024-03-26T04:08:09.554503+0000","last_clean":"2024-03-26T04:08:09.554503+0000","last_became_active":"2024-03-26T04:08:02.630722+0000","last_became_peered":"2024-03-26T04:08:02.630722+0000","last_unstale":"2024-03-26T04:08:09.554503+0000","last_undegraded":"2024-03-26T04:08:09.554503+0000","last_fullsized":"2024-03-26T04:08:09.554503+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:20.222126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":999,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107046+0000","last_change":"2024-03-26T04:07:26.838873+0000","last_active":"2024-03-26T04:08:09.107046+0000","last_peered":"2024-03-26T04:08:09.107046+0000","last_clean":"2024-03-26T04:08:09.107046+0000","last_became_active":"2024-03-26T04:02:55.533229+0000","last_became_peered":"2024-03-26T04:02:55.533229+0000","last_unstale":"2024-03-26T04:08:09.107046+0000","last_undegraded":"2024-03-26T04:08:09.107046+0000","last_fullsized":"2024-03-26T04:08:09.107046+0000","mapping_epoch":326,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":327,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:26.838741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.838741+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:38.279852+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":819,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106812+0000","last_change":"2024-03-26T04:07:40.043033+0000","last_active":"2024-03-26T04:08:09.106812+0000","last_peered":"2024-03-26T04:08:09.106812+0000","last_clean":"2024-03-26T04:08:09.106812+0000","last_became_active":"2024-03-26T04:05:29.542416+0000","last_became_peered":"2024-03-26T04:05:29.542416+0000","last_unstale":"2024-03-26T04:08:09.106812+0000","last_undegraded":"2024-03-26T04:08:09.106812+0000","last_fullsized":"2024-03-26T04:08:09.106812+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:40.042936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:07:40.042936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.886296+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":797,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555444+0000","last_change":"2024-03-26T04:05:07.862785+0000","last_active":"2024-03-26T04:08:04.555444+0000","last_peered":"2024-03-26T04:08:04.555444+0000","last_clean":"2024-03-26T04:08:04.555444+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:08:04.555444+0000","last_undegraded":"2024-03-26T04:08:04.555444+0000","last_fullsized":"2024-03-26T04:08:04.555444+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:05:07.862659+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.862659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"241'34","reported_seq":903,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553892+0000","last_change":"2024-03-26T04:04:16.357551+0000","last_active":"2024-03-26T04:08:09.553892+0000","last_peered":"2024-03-26T04:08:09.553892+0000","last_clean":"2024-03-26T04:08:09.553892+0000","last_became_active":"2024-03-26T04:01:35.799996+0000","last_became_peered":"2024-03-26T04:01:35.799996+0000","last_unstale":"2024-03-26T04:08:09.553892+0000","last_undegraded":"2024-03-26T04:08:09.553892+0000","last_fullsized":"2024-03-26T04:08:09.553892+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:04:16.357311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:16.357311+0000","objects_scrubbed":8,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394372+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":43,"num_write":34,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":509,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589675+0000","last_change":"2024-03-26T04:03:51.382658+0000","last_active":"2024-03-26T04:08:04.589675+0000","last_peered":"2024-03-26T04:08:04.589675+0000","last_clean":"2024-03-26T04:08:04.589675+0000","last_became_active":"2024-03-26T04:02:52.515390+0000","last_became_peered":"2024-03-26T04:02:52.515390+0000","last_unstale":"2024-03-26T04:08:04.589675+0000","last_undegraded":"2024-03-26T04:08:04.589675+0000","last_fullsized":"2024-03-26T04:08:04.589675+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:51.382563+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:03:51.382563+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":21,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107322+0000","last_change":"2024-03-26T04:08:02.634295+0000","last_active":"2024-03-26T04:08:09.107322+0000","last_peered":"2024-03-26T04:08:09.107322+0000","last_clean":"2024-03-26T04:08:09.107322+0000","last_became_active":"2024-03-26T04:08:02.633998+0000","last_became_peered":"2024-03-26T04:08:02.633998+0000","last_unstale":"2024-03-26T04:08:09.107322+0000","last_undegraded":"2024-03-26T04:08:09.107322+0000","last_fullsized":"2024-03-26T04:08:09.107322+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:29.010359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":651,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554162+0000","last_change":"2024-03-26T04:07:39.474303+0000","last_active":"2024-03-26T04:08:09.554162+0000","last_peered":"2024-03-26T04:08:09.554162+0000","last_clean":"2024-03-26T04:08:09.554162+0000","last_became_active":"2024-03-26T04:05:29.521761+0000","last_became_peered":"2024-03-26T04:05:29.521761+0000","last_unstale":"2024-03-26T04:08:09.554162+0000","last_undegraded":"2024-03-26T04:08:09.554162+0000","last_fullsized":"2024-03-26T04:08:09.554162+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:39.474219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.474219+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:43.579453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":803,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553271+0000","last_change":"2024-03-26T04:07:30.569585+0000","last_active":"2024-03-26T04:08:09.553271+0000","last_peered":"2024-03-26T04:08:09.553271+0000","last_clean":"2024-03-26T04:08:09.553271+0000","last_became_active":"2024-03-26T04:05:29.532854+0000","last_became_peered":"2024-03-26T04:05:29.532854+0000","last_unstale":"2024-03-26T04:08:09.553271+0000","last_undegraded":"2024-03-26T04:08:09.553271+0000","last_fullsized":"2024-03-26T04:08:09.553271+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.569498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.569498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.566912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":300,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555444+0000","last_change":"2024-03-26T04:03:06.122009+0000","last_active":"2024-03-26T04:08:04.555444+0000","last_peered":"2024-03-26T04:08:04.555444+0000","last_clean":"2024-03-26T04:08:04.555444+0000","last_became_active":"2024-03-26T04:02:52.496389+0000","last_became_peered":"2024-03-26T04:02:52.496389+0000","last_unstale":"2024-03-26T04:08:04.555444+0000","last_undegraded":"2024-03-26T04:08:04.555444+0000","last_fullsized":"2024-03-26T04:08:04.555444+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:06.121917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:06.121917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"241'52","reported_seq":377,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106262+0000","last_change":"2024-03-26T04:07:35.968314+0000","last_active":"2024-03-26T04:08:09.106262+0000","last_peered":"2024-03-26T04:08:09.106262+0000","last_clean":"2024-03-26T04:08:09.106262+0000","last_became_active":"2024-03-26T04:02:52.511833+0000","last_became_peered":"2024-03-26T04:02:52.511833+0000","last_unstale":"2024-03-26T04:08:09.106262+0000","last_undegraded":"2024-03-26T04:08:09.106262+0000","last_fullsized":"2024-03-26T04:08:09.106262+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:07:35.968179+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:07:35.968179+0000","objects_scrubbed":10,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:00.483729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":806,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553551+0000","last_change":"2024-03-26T04:04:06.297839+0000","last_active":"2024-03-26T04:08:09.553551+0000","last_peered":"2024-03-26T04:08:09.553551+0000","last_clean":"2024-03-26T04:08:09.553551+0000","last_became_active":"2024-03-26T04:02:52.498870+0000","last_became_peered":"2024-03-26T04:02:52.498870+0000","last_unstale":"2024-03-26T04:08:09.553551+0000","last_undegraded":"2024-03-26T04:08:09.553551+0000","last_fullsized":"2024-03-26T04:08:09.553551+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.297701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394301+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589324+0000","last_change":"2024-03-26T04:08:02.630527+0000","last_active":"2024-03-26T04:08:04.589324+0000","last_peered":"2024-03-26T04:08:04.589324+0000","last_clean":"2024-03-26T04:08:04.589324+0000","last_became_active":"2024-03-26T04:08:02.630245+0000","last_became_peered":"2024-03-26T04:08:02.630245+0000","last_unstale":"2024-03-26T04:08:04.589324+0000","last_undegraded":"2024-03-26T04:08:04.589324+0000","last_fullsized":"2024-03-26T04:08:04.589324+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:13.720140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":484,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661225+0000","last_change":"2024-03-26T04:07:22.530369+0000","last_active":"2024-03-26T04:08:07.661225+0000","last_peered":"2024-03-26T04:08:07.661225+0000","last_clean":"2024-03-26T04:08:07.661225+0000","last_became_active":"2024-03-26T04:05:29.524691+0000","last_became_peered":"2024-03-26T04:05:29.524691+0000","last_unstale":"2024-03-26T04:08:07.661225+0000","last_undegraded":"2024-03-26T04:08:07.661225+0000","last_fullsized":"2024-03-26T04:08:07.661225+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:22.530143+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:07:22.530143+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:22.608960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":471,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590459+0000","last_change":"2024-03-26T04:05:29.532951+0000","last_active":"2024-03-26T04:08:04.590459+0000","last_peered":"2024-03-26T04:08:04.590459+0000","last_clean":"2024-03-26T04:08:04.590459+0000","last_became_active":"2024-03-26T04:05:29.532692+0000","last_became_peered":"2024-03-26T04:05:29.532692+0000","last_unstale":"2024-03-26T04:08:04.590459+0000","last_undegraded":"2024-03-26T04:08:04.590459+0000","last_fullsized":"2024-03-26T04:08:04.590459+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:25.365447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:01:25.365447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"379'13","reported_seq":1062,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554894+0000","last_change":"2024-03-26T04:07:54.544916+0000","last_active":"2024-03-26T04:08:09.554894+0000","last_peered":"2024-03-26T04:08:09.554894+0000","last_clean":"2024-03-26T04:08:09.554894+0000","last_became_active":"2024-03-26T04:05:29.521835+0000","last_became_peered":"2024-03-26T04:05:29.521835+0000","last_unstale":"2024-03-26T04:08:09.554894+0000","last_undegraded":"2024-03-26T04:08:09.554894+0000","last_fullsized":"2024-03-26T04:08:09.554894+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'13","last_scrub_stamp":"2024-03-26T04:07:54.544776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.544776+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:01.295876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":372,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.589346+0000","last_change":"2024-03-26T04:03:50.413181+0000","last_active":"2024-03-26T04:08:04.589346+0000","last_peered":"2024-03-26T04:08:04.589346+0000","last_clean":"2024-03-26T04:08:04.589346+0000","last_became_active":"2024-03-26T04:02:52.511800+0000","last_became_peered":"2024-03-26T04:02:52.511800+0000","last_unstale":"2024-03-26T04:08:04.589346+0000","last_undegraded":"2024-03-26T04:08:04.589346+0000","last_fullsized":"2024-03-26T04:08:04.589346+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:50.413099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:03:50.413099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"241'27","reported_seq":898,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590915+0000","last_change":"2024-03-26T04:04:06.282172+0000","last_active":"2024-03-26T04:08:04.590915+0000","last_peered":"2024-03-26T04:08:04.590915+0000","last_clean":"2024-03-26T04:08:04.590915+0000","last_became_active":"2024-03-26T04:02:52.513721+0000","last_became_peered":"2024-03-26T04:02:52.513721+0000","last_unstale":"2024-03-26T04:08:04.590915+0000","last_undegraded":"2024-03-26T04:08:04.590915+0000","last_fullsized":"2024-03-26T04:08:04.590915+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:04:06.281970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.281970+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588588+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":15,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.554015+0000","last_change":"2024-03-26T04:08:02.634056+0000","last_active":"2024-03-26T04:08:04.554015+0000","last_peered":"2024-03-26T04:08:04.554015+0000","last_clean":"2024-03-26T04:08:04.554015+0000","last_became_active":"2024-03-26T04:08:02.633211+0000","last_became_peered":"2024-03-26T04:08:02.633211+0000","last_unstale":"2024-03-26T04:08:04.554015+0000","last_undegraded":"2024-03-26T04:08:04.554015+0000","last_fullsized":"2024-03-26T04:08:04.554015+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:19.307816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":420,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.107167+0000","last_change":"2024-03-26T04:06:00.896785+0000","last_active":"2024-03-26T04:08:09.107167+0000","last_peered":"2024-03-26T04:08:09.107167+0000","last_clean":"2024-03-26T04:08:09.107167+0000","last_became_active":"2024-03-26T04:02:52.506309+0000","last_became_peered":"2024-03-26T04:02:52.506309+0000","last_unstale":"2024-03-26T04:08:09.107167+0000","last_undegraded":"2024-03-26T04:08:09.107167+0000","last_fullsized":"2024-03-26T04:08:09.107167+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:06:00.896583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:06:00.896583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":529,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554927+0000","last_change":"2024-03-26T04:07:34.478534+0000","last_active":"2024-03-26T04:08:09.554927+0000","last_peered":"2024-03-26T04:08:09.554927+0000","last_clean":"2024-03-26T04:08:09.554927+0000","last_became_active":"2024-03-26T04:05:29.520457+0000","last_became_peered":"2024-03-26T04:05:29.520457+0000","last_unstale":"2024-03-26T04:08:09.554927+0000","last_undegraded":"2024-03-26T04:08:09.554927+0000","last_fullsized":"2024-03-26T04:08:09.554927+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.478427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.478427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:01.934278+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":804,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553632+0000","last_change":"2024-03-26T04:07:45.504621+0000","last_active":"2024-03-26T04:08:09.553632+0000","last_peered":"2024-03-26T04:08:09.553632+0000","last_clean":"2024-03-26T04:08:09.553632+0000","last_became_active":"2024-03-26T04:05:29.536797+0000","last_became_peered":"2024-03-26T04:05:29.536797+0000","last_unstale":"2024-03-26T04:08:09.553632+0000","last_undegraded":"2024-03-26T04:08:09.553632+0000","last_fullsized":"2024-03-26T04:08:09.553632+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.504517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.504517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:07.030754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"241'25","reported_seq":931,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554552+0000","last_change":"2024-03-26T04:05:06.619430+0000","last_active":"2024-03-26T04:08:09.554552+0000","last_peered":"2024-03-26T04:08:09.554552+0000","last_clean":"2024-03-26T04:08:09.554552+0000","last_became_active":"2024-03-26T04:02:52.520710+0000","last_became_peered":"2024-03-26T04:02:52.520710+0000","last_unstale":"2024-03-26T04:08:09.554552+0000","last_undegraded":"2024-03-26T04:08:09.554552+0000","last_fullsized":"2024-03-26T04:08:09.554552+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T04:05:06.619261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:05:06.619261+0000","objects_scrubbed":6,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394524+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":41,"num_read_kb":33,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":19,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660845+0000","last_change":"2024-03-26T04:08:02.626752+0000","last_active":"2024-03-26T04:08:07.660845+0000","last_peered":"2024-03-26T04:08:07.660845+0000","last_clean":"2024-03-26T04:08:07.660845+0000","last_became_active":"2024-03-26T04:08:02.626374+0000","last_became_peered":"2024-03-26T04:08:02.626374+0000","last_unstale":"2024-03-26T04:08:07.660845+0000","last_undegraded":"2024-03-26T04:08:07.660845+0000","last_fullsized":"2024-03-26T04:08:07.660845+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:28.379876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":512,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.660617+0000","last_change":"2024-03-26T04:05:29.533929+0000","last_active":"2024-03-26T04:08:07.660617+0000","last_peered":"2024-03-26T04:08:07.660617+0000","last_clean":"2024-03-26T04:08:07.660617+0000","last_became_active":"2024-03-26T04:05:29.533470+0000","last_became_peered":"2024-03-26T04:05:29.533470+0000","last_unstale":"2024-03-26T04:08:07.660617+0000","last_undegraded":"2024-03-26T04:08:07.660617+0000","last_fullsized":"2024-03-26T04:08:07.660617+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:39.967821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:39.967821+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682173+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":723,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661867+0000","last_change":"2024-03-26T04:05:29.522502+0000","last_active":"2024-03-26T04:08:07.661867+0000","last_peered":"2024-03-26T04:08:07.661867+0000","last_clean":"2024-03-26T04:08:07.661867+0000","last_became_active":"2024-03-26T04:05:29.521089+0000","last_became_peered":"2024-03-26T04:05:29.521089+0000","last_unstale":"2024-03-26T04:08:07.661867+0000","last_undegraded":"2024-03-26T04:08:07.661867+0000","last_fullsized":"2024-03-26T04:08:07.661867+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_clean_scrub_stamp":"2024-03-26T04:02:49.166761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682189+0000","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":785,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.590527+0000","last_change":"2024-03-26T04:04:49.184140+0000","last_active":"2024-03-26T04:08:04.590527+0000","last_peered":"2024-03-26T04:08:04.590527+0000","last_clean":"2024-03-26T04:08:04.590527+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:08:04.590527+0000","last_undegraded":"2024-03-26T04:08:04.590527+0000","last_fullsized":"2024-03-26T04:08:04.590527+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:49.184018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:04:49.184018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:09.588622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":761,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553398+0000","last_change":"2024-03-26T04:05:08.666190+0000","last_active":"2024-03-26T04:08:09.553398+0000","last_peered":"2024-03-26T04:08:09.553398+0000","last_clean":"2024-03-26T04:08:09.553398+0000","last_became_active":"2024-03-26T04:02:52.505213+0000","last_became_peered":"2024-03-26T04:02:52.505213+0000","last_unstale":"2024-03-26T04:08:09.553398+0000","last_undegraded":"2024-03-26T04:08:09.553398+0000","last_fullsized":"2024-03-26T04:08:09.553398+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:08.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:08.666106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394559+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"241'31","reported_seq":849,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554466+0000","last_change":"2024-03-26T04:07:25.580036+0000","last_active":"2024-03-26T04:08:09.554466+0000","last_peered":"2024-03-26T04:08:09.554466+0000","last_clean":"2024-03-26T04:08:09.554466+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:08:09.554466+0000","last_undegraded":"2024-03-26T04:08:09.554466+0000","last_fullsized":"2024-03-26T04:08:09.554466+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:07:25.579888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.579888+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:33.986075+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":21,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106580+0000","last_change":"2024-03-26T04:08:02.631593+0000","last_active":"2024-03-26T04:08:09.106580+0000","last_peered":"2024-03-26T04:08:09.106580+0000","last_clean":"2024-03-26T04:08:09.106580+0000","last_became_active":"2024-03-26T04:08:02.631319+0000","last_became_peered":"2024-03-26T04:08:02.631319+0000","last_unstale":"2024-03-26T04:08:09.106580+0000","last_undegraded":"2024-03-26T04:08:09.106580+0000","last_fullsized":"2024-03-26T04:08:09.106580+0000","mapping_epoch":427,"log_start":"0'0","ondisk_log_start":"0'0","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:16.380747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1115,"reported_epoch":433,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.106203+0000","last_change":"2024-03-26T04:07:48.931663+0000","last_active":"2024-03-26T04:08:09.106203+0000","last_peered":"2024-03-26T04:08:09.106203+0000","last_clean":"2024-03-26T04:08:09.106203+0000","last_became_active":"2024-03-26T04:05:29.543230+0000","last_became_peered":"2024-03-26T04:05:29.543230+0000","last_unstale":"2024-03-26T04:08:09.106203+0000","last_undegraded":"2024-03-26T04:08:09.106203+0000","last_fullsized":"2024-03-26T04:08:09.106203+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.931565+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.287206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":792,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.554923+0000","last_change":"2024-03-26T04:07:27.607414+0000","last_active":"2024-03-26T04:08:09.554923+0000","last_peered":"2024-03-26T04:08:09.554923+0000","last_clean":"2024-03-26T04:08:09.554923+0000","last_became_active":"2024-03-26T04:05:29.520602+0000","last_became_peered":"2024-03-26T04:05:29.520602+0000","last_unstale":"2024-03-26T04:08:09.554923+0000","last_undegraded":"2024-03-26T04:08:09.554923+0000","last_fullsized":"2024-03-26T04:08:09.554923+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:27.607316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:07:27.607316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:35.557513+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"379'13","reported_seq":959,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-03-26T04:08:07.661508+0000","last_change":"2024-03-26T04:05:29.544439+0000","last_active":"2024-03-26T04:08:07.661508+0000","last_peered":"2024-03-26T04:08:07.661508+0000","last_clean":"2024-03-26T04:08:07.661508+0000","last_became_active":"2024-03-26T04:05:29.544028+0000","last_became_peered":"2024-03-26T04:05:29.544028+0000","last_unstale":"2024-03-26T04:08:07.661508+0000","last_undegraded":"2024-03-26T04:08:07.661508+0000","last_fullsized":"2024-03-26T04:08:07.661508+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:01:34.236967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.236967+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:11.682085+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":372,"reported_epoch":430,"state":"active+clean","last_fresh":"2024-03-26T04:08:04.555123+0000","last_change":"2024-03-26T04:04:06.376407+0000","last_active":"2024-03-26T04:08:04.555123+0000","last_peered":"2024-03-26T04:08:04.555123+0000","last_clean":"2024-03-26T04:08:04.555123+0000","last_became_active":"2024-03-26T04:02:52.491417+0000","last_became_peered":"2024-03-26T04:02:52.491417+0000","last_unstale":"2024-03-26T04:08:04.555123+0000","last_undegraded":"2024-03-26T04:08:04.555123+0000","last_fullsized":"2024-03-26T04:08:04.555123+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.376284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.376284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"241'20","reported_seq":893,"reported_epoch":434,"state":"active+clean","last_fresh":"2024-03-26T04:08:09.553441+0000","last_change":"2024-03-26T04:04:18.408633+0000","last_active":"2024-03-26T04:08:09.553441+0000","last_peered":"2024-03-26T04:08:09.553441+0000","last_clean":"2024-03-26T04:08:09.553441+0000","last_became_active":"2024-03-26T04:02:52.504477+0000","last_became_peered":"2024-03-26T04:02:52.504477+0000","last_unstale":"2024-03-26T04:08:09.553441+0000","last_undegraded":"2024-03-26T04:08:09.553441+0000","last_fullsized":"2024-03-26T04:08:09.553441+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:04:18.408372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.408372+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:08:14.394444+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1157,"num_read_kb":1152,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":4179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":5986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1399,"num_read_kb":1124,"num_write":895,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":436,"num_bytes_recovered":7718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":15688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":961,"ondisk_log_size":961,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070701,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":877888,"kb_used_data":2612,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92867264,"statfs":{"total":95995035648,"available":95096078336,"internally_reserved":0,"allocated":2674688,"data_stored":1602650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":896268746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365043,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575115,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":823748,"kb_used_data":2608,"kb_used_omap":1,"kb_used_meta":821118,"kb_avail":92921404,"statfs":{"total":95995035648,"available":95151517696,"internally_reserved":0,"allocated":2670592,"data_stored":1589255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":840825285},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738640,"num_pgs":95,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":940264,"kb_used_data":2520,"kb_used_omap":1,"kb_used_meta":937726,"kb_avail":92804888,"statfs":{"total":95995035648,"available":95032205312,"internally_reserved":0,"allocated":2580480,"data_stored":1523555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":960231877},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637921,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318292,"kb_used_data":2496,"kb_used_omap":6,"kb_used_meta":315769,"kb_avail":93426860,"statfs":{"total":95995035648,"available":95669104640,"internally_reserved":0,"allocated":2555904,"data_stored":1528355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":323347749},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953724,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":972512,"kb_used_data":2832,"kb_used_omap":1,"kb_used_meta":969662,"kb_avail":92772640,"statfs":{"total":95995035648,"available":94999183360,"internally_reserved":0,"allocated":2899968,"data_stored":1862941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":992934341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:08:12.879 INFO:tasks.thrashosds.thrasher:Setting 3.13 to [7, 1, 5, 3, 4, 6] 2024-03-26T04:08:12.880 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.13', '7', '1', '5', '3', '4', '6'] 2024-03-26T04:08:12.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pg-upmap-items 3.13 7 1 5 3 4 6 2024-03-26T04:08:12.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:12.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:12.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:12.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:12.999 DEBUG:teuthology.orchestra.run.smithi006:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-26T04:08:13.002 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-26T04:08:13.002 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-26T04:08:13.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:13.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:13.106 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:hints = 1 2024-03-26T04:08:13.106 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-26T04:08:13.106 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object prefix: benchmark_data_smithi006_185099 2024-03-26T04:08:13.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:08:13.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 0 0 0 0 0 0 - 0 2024-03-26T04:08:13.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:13.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:13 smithi006 ceph-mon[37501]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:13.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:13 smithi006 ceph-mon[29446]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:13.166 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:13.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:13.229 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:13 smithi086 ceph-mon[32965]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:08:13.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:13.299 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:13.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:13.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:08:13.401 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:13.489 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:13.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:13.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:13.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:13.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:13.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:13.769 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:13.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:13.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:13.912 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:13.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:14.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:14.053 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:14.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 1 16 3047 3031 189.426 189.438 0.0113871 0.00525356 2024-03-26T04:08:14.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:14.183 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:14.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:14.191 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:14 smithi086 ceph-mon[32965]: pgmap v928: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:14.191 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:14 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:14.192 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:14 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:14.192 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:14 smithi086 ceph-mon[32965]: osdmap e436: 8 total, 8 up, 5 in 2024-03-26T04:08:14.285 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:14.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[37501]: pgmap v928: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:14.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[37501]: osdmap e436: 8 total, 8 up, 5 in 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[29446]: pgmap v928: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:14.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-mon[29446]: osdmap e436: 8 total, 8 up, 5 in 2024-03-26T04:08:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.328 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:14.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:14.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:14.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:14.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:14.596 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:14.626 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:14.647 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:14.648 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:14.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:14.743 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:14.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:14.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:14.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:14.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:14.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:15.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:15.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:15.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 2 16 5507 5491 171.581 153.75 0.00551584 0.00581465 2024-03-26T04:08:15.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:15.145 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:15.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.146 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:15.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:15.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.287 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:15.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:15.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T04:08:15.362 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:15.362 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:15.388 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:15.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:15.587 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:15.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.598 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:15.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:15.666 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:15.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:15.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:15.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:15.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:15.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:15.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:15.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:16.015 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:08:16.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 3 16 8137 8121 169.173 164.375 0.00498166 0.0058997 2024-03-26T04:08:16.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[37501]: pgmap v930: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 899 B/s rd, 0 op/s 2024-03-26T04:08:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4008141161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[29446]: pgmap v930: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 899 B/s rd, 0 op/s 2024-03-26T04:08:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4008141161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:16 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:16.247 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:16.290 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:16.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:16 smithi086 ceph-mon[32965]: pgmap v930: 158 pgs: 158 active+clean; 582 KiB data, 3.8 GiB used, 443 GiB / 447 GiB avail; 899 B/s rd, 0 op/s 2024-03-26T04:08:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:16 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4008141161' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:16 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:08:16.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:16.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:16.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:16.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:16.563 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:16.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:16.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:16.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:16.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:16.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:16.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:16.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:16.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:16.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:17.077 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:17.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 4 16 10807 10791 168.595 166.875 0.00780577 0.00592167 2024-03-26T04:08:17.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:17.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:17.203 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:17.203 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: osdmap e437: 8 total, 8 up, 5 in 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:08:17.204 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:17 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:08:17.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:17.252 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:17.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:17.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:17.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:17.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:17.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: osdmap e437: 8 total, 8 up, 5 in 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:08:17.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: osdmap e437: 8 total, 8 up, 5 in 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:08:17.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:17 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:08:17.482 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:17.482 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:17.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:17.523 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:17.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:17.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:17.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:17.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:17.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:17.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:17.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:17.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:17.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:17.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:18.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:18.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:18.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 5 16 13049 13033 162.898 140.125 0.00516963 0.00613009 2024-03-26T04:08:18.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:18.201 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:18.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:18.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:18 smithi086 ceph-mon[32965]: pgmap v932: 158 pgs: 158 active+clean; 400 MiB data, 4.5 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T04:08:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:18 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T04:08:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:18 smithi086 ceph-mon[32965]: 4.10 scrub ok 2024-03-26T04:08:18.348 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[37501]: pgmap v932: 158 pgs: 158 active+clean; 400 MiB data, 4.5 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T04:08:18.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T04:08:18.349 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[37501]: 4.10 scrub ok 2024-03-26T04:08:18.349 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[29446]: pgmap v932: 158 pgs: 158 active+clean; 400 MiB data, 4.5 GiB used, 442 GiB / 447 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.06k op/s 2024-03-26T04:08:18.349 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T04:08:18.349 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:18 smithi006 ceph-mon[29446]: 4.10 scrub ok 2024-03-26T04:08:18.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:18.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:18.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T04:08:18.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:18.604 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:18.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:18.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:18.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:18.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:18.847 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:18.886 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:18.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:18.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:19.013 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:19.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:19.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 6 16 15331 15315 159.517 142.625 0.00761256 0.00626145 2024-03-26T04:08:19.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:19.146 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:19.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:19.252 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:19 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T04:08:19.253 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:19 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T04:08:19.253 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:19 smithi086 ceph-mon[32965]: pgmap v933: 158 pgs: 158 active+clean; 498 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 698 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-26T04:08:19.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:19.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:19.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T04:08:19.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T04:08:19.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[37501]: pgmap v933: 158 pgs: 158 active+clean; 498 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 698 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-26T04:08:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T04:08:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T04:08:19.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:19 smithi006 ceph-mon[29446]: pgmap v933: 158 pgs: 158 active+clean; 498 MiB data, 5.0 GiB used, 442 GiB / 447 GiB avail; 698 B/s rd, 68 MiB/s wr, 1.08k op/s 2024-03-26T04:08:19.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:19.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.423 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:19.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:19.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:19.552 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:19.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:19.683 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:19.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:19.785 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:19.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:19.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:19.962 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:19.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:20.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:20.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:20.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:20.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 7 16 17308 17292 154.379 123.562 0.00471973 0.00647029 2024-03-26T04:08:20.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:20.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:20.273 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:08:20.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:20.384 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:20.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:20.533 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:20.626 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[29446]: 6.13 scrub starts 2024-03-26T04:08:20.627 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[29446]: osdmap e438: 8 total, 8 up, 5 in 2024-03-26T04:08:20.627 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[29446]: 6.13 scrub ok 2024-03-26T04:08:20.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[37501]: 6.13 scrub starts 2024-03-26T04:08:20.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[37501]: osdmap e438: 8 total, 8 up, 5 in 2024-03-26T04:08:20.627 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:20 smithi006 ceph-mon[37501]: 6.13 scrub ok 2024-03-26T04:08:20.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:20.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:20.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:20.765 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:20.766 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:20.766 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:20 smithi086 ceph-mon[32965]: 6.13 scrub starts 2024-03-26T04:08:20.766 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:20 smithi086 ceph-mon[32965]: osdmap e438: 8 total, 8 up, 5 in 2024-03-26T04:08:20.766 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:20 smithi086 ceph-mon[32965]: 6.13 scrub ok 2024-03-26T04:08:20.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:20.811 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:20.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:20.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:20.953 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:20.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:20.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:21.055 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:21.082 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:21.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 8 16 18834 18818 147.003 95.375 0.103573 0.00675032 2024-03-26T04:08:21.184 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:21.222 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:21.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:21.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:21.353 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:21.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T04:08:21.454 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:21.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:21.583 INFO:teuthology.orchestra.run.smithi006.stderr:set 3.13 pg_upmap_items mapping to [7->1,5->3,4->6] 2024-03-26T04:08:21.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:21.639 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:21.690 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:21.741 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:21.779 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:21.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:21 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T04:08:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:21 smithi086 ceph-mon[32965]: pgmap v935: 158 pgs: 158 active+clean; 690 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:21 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T04:08:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:21 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2890026222' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:21 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.845 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T04:08:21.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[29446]: pgmap v935: 158 pgs: 158 active+clean; 690 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:21.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T04:08:21.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2890026222' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.846 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T04:08:21.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[37501]: pgmap v935: 158 pgs: 158 active+clean; 690 MiB data, 5.7 GiB used, 441 GiB / 447 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:21.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T04:08:21.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2890026222' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.848 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:21 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]: dispatch 2024-03-26T04:08:21.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:21.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:21.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:21.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:22.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:22.036 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:22.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 9 16 20606 20590 142.974 110.75 0.00359606 0.00698798 2024-03-26T04:08:22.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:22.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:22.311 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:22.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:22.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:22.452 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:22.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:22.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:22.747 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:22.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T04:08:22.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T04:08:22.836 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T04:08:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T04:08:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]': finished 2024-03-26T04:08:22.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:22 smithi086 ceph-mon[32965]: osdmap e439: 8 total, 8 up, 5 in 2024-03-26T04:08:22.849 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:22.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:22.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:22.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]': finished 2024-03-26T04:08:22.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[29446]: osdmap e439: 8 total, 8 up, 5 in 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.13", "id": [7, 1, 5, 3, 4, 6]}]': finished 2024-03-26T04:08:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:22 smithi006 ceph-mon[37501]: osdmap e439: 8 total, 8 up, 5 in 2024-03-26T04:08:22.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:23.015 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:23.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 10 16 22522 22506 140.65 119.75 0.00712433 0.00710389 2024-03-26T04:08:23.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:23.146 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:23.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:23.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:23.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:23.411 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:23.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:23.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:23.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:23.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:23.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:23.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:23.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:23 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T04:08:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:23 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T04:08:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:23 smithi086 ceph-mon[32965]: pgmap v937: 158 pgs: 158 active+clean; 1.1 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 810 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-03-26T04:08:23.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:23 smithi086 ceph-mon[32965]: osdmap e440: 8 total, 8 up, 5 in 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[29446]: pgmap v937: 158 pgs: 158 active+clean; 1.1 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 810 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[29446]: osdmap e440: 8 total, 8 up, 5 in 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T04:08:23.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[37501]: pgmap v937: 158 pgs: 158 active+clean; 1.1 GiB data, 6.7 GiB used, 440 GiB / 447 GiB avail; 810 B/s rd, 164 MiB/s wr, 2.62k op/s 2024-03-26T04:08:23.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:23 smithi006 ceph-mon[37501]: osdmap e440: 8 total, 8 up, 5 in 2024-03-26T04:08:23.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:23.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:23.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:24.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:24.052 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:24.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:24.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 11 16 24003 23987 136.278 92.5625 0.00167811 0.00727453 2024-03-26T04:08:24.153 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:24.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:24.201 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:24.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:24.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:24.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:24.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:24.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:24.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:24.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:24.580 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:24.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:24.590 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:24.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:24.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.706 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:24.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:24.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:08:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:08:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: osdmap e441: 8 total, 8 up, 5 in 2024-03-26T04:08:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: osdmap e442: 8 total, 8 up, 5 in 2024-03-26T04:08:24.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:24 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:24.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: osdmap e441: 8 total, 8 up, 5 in 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: osdmap e442: 8 total, 8 up, 5 in 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: osdmap e441: 8 total, 8 up, 5 in 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: osdmap e442: 8 total, 8 up, 5 in 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-03-26T04:08:24.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:24 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]: dispatch 2024-03-26T04:08:24.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:24.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:24.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:25.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:25.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 12 15 25524 25509 132.848 95.125 0.00769422 0.00752049 2024-03-26T04:08:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:25.150 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:25.151 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:25.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:25.251 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:08:25.339 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:08:25.340 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:08:25.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:25.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:25.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:25.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:25.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:25.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:25.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.652 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:25.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:25.752 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:25.795 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:25.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: pgmap v940: 158 pgs: 158 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1006735337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]': finished 2024-03-26T04:08:25.819 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:25 smithi086 ceph-mon[32965]: osdmap e443: 8 total, 8 up, 5 in 2024-03-26T04:08:25.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: pgmap v940: 158 pgs: 158 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1006735337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T04:08:25.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]': finished 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[37501]: osdmap e443: 8 total, 8 up, 5 in 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: pgmap v940: 158 pgs: 158 active+clean; 1.2 GiB data, 7.1 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 142 MiB/s wr, 2.27k op/s 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1006735337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1c", "id": [4, 2]}]': finished 2024-03-26T04:08:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-03-26T04:08:25.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.3", "id": [1, 4]}]': finished 2024-03-26T04:08:25.892 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:25 smithi006 ceph-mon[29446]: osdmap e443: 8 total, 8 up, 5 in 2024-03-26T04:08:25.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:25.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:25.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:25.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:26.033 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:26.074 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:26.107 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 13 15 27398 27383 131.637 117.125 0.0074821 0.00759083 2024-03-26T04:08:26.170 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:26.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:26.215 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:26.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:26.358 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:26.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:26.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:26.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:26.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:26.635 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:26.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:26.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:26.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:26.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:26.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T04:08:26.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[37501]: osdmap e444: 8 total, 8 up, 5 in 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:26 smithi006 ceph-mon[29446]: osdmap e444: 8 total, 8 up, 5 in 2024-03-26T04:08:26.896 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:26.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:26.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:27.010 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:26 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T04:08:27.010 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:26 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T04:08:27.010 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:26 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T04:08:27.010 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:27.011 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:26 smithi086 ceph-mon[32965]: osdmap e444: 8 total, 8 up, 5 in 2024-03-26T04:08:27.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:27.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 14 15 29223 29208 130.381 114.062 0.00859928 0.00766383 2024-03-26T04:08:27.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:27.202 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:27.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:27.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:27.453 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:27.492 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:27.594 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:27.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T04:08:27.633 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T04:08:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:27.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:27.776 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:27.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:27.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:27.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T04:08:27.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T04:08:27.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[37501]: pgmap v943: 158 pgs: 1 peering, 157 active+clean; 1.6 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-26T04:08:27.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:27 smithi006 ceph-mon[29446]: pgmap v943: 158 pgs: 1 peering, 157 active+clean; 1.6 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-26T04:08:27.898 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:27 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T04:08:27.898 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:27 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T04:08:27.898 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:27 smithi086 ceph-mon[32965]: pgmap v943: 158 pgs: 1 peering, 157 active+clean; 1.6 GiB data, 8.4 GiB used, 439 GiB / 447 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-26T04:08:27.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:27.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:27.957 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:27.957 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-03-26T04:08:27.957 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-03-26T04:08:27.957 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-03-26T04:08:27.957 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:28.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:28.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:28.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:28.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 15 16 30922 30906 128.764 106.125 0.00673259 0.00776008 2024-03-26T04:08:28.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:28.185 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:28.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:28.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:28.325 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:28.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:28.455 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:28.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:28.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:28.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:28.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T04:08:28.660 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:08:28.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:28.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:28.736 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:08:28.821 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:28.870 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:28.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:28.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:28.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:08:28.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:08:28.911 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1535383852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T04:08:28.912 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T04:08:28.913 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T04:08:28.913 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T04:08:28.913 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1535383852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:28.913 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:28 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:28.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:08:28.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:29.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:29.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:29.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:08:29.024 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1535383852' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:29.025 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:28 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:08:29.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:29.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 16 15 32642 32627 127.437 107.562 0.00771153 0.00784166 2024-03-26T04:08:29.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:29.171 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7] out_osds: [6, 5, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-03-26T04:08:29.171 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:08:29.171 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-26T04:08:29.171 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-26T04:08:29.171 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-26T04:08:29.172 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-26T04:08:29.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:29.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:29.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:29.441 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:29.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:29.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:29.572 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:29.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:29.633 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:29.637 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:29.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:29.703 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:29.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:29.777 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: pgmap v945: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: 1.1 scrub starts 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: 1.1 scrub ok 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: osdmap e445: 8 total, 8 up, 5 in 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[37501]: osdmap e446: 8 total, 8 up, 5 in 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T04:08:29.778 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: pgmap v945: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: 1.1 scrub starts 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: 1.1 scrub ok 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: osdmap e445: 8 total, 8 up, 5 in 2024-03-26T04:08:29.779 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:29 smithi006 ceph-mon[29446]: osdmap e446: 8 total, 8 up, 5 in 2024-03-26T04:08:29.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:29.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:29.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: pgmap v945: 158 pgs: 1 peering, 157 active+clean; 1.7 GiB data, 8.7 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: 1.1 scrub starts 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: 1.1 scrub ok 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:08:29.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: osdmap e445: 8 total, 8 up, 5 in 2024-03-26T04:08:29.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:29 smithi086 ceph-mon[32965]: osdmap e446: 8 total, 8 up, 5 in 2024-03-26T04:08:29.935 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:29.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:29.981 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:29.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:30.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:30.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 17 16 34073 34057 125.197 89.375 0.00859305 0.00798094 2024-03-26T04:08:30.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:30.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:30.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:30.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:30.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:30.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:30.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.512 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:30.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:30.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:30.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:30.743 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:30.785 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:30.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:30.915 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:30.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:31.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:31.057 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:30 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T04:08:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:30 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T04:08:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:30 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T04:08:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:30 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T04:08:31.108 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 18 16 35767 35751 124.123 105.875 0.00946396 0.00805012 2024-03-26T04:08:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T04:08:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:30 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T04:08:31.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:31.198 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:31.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:31.329 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:31.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:31.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:31.481 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:08:31.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:31.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:31.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:31.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:31.720 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:31.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:31 smithi086 ceph-mon[32965]: 6.c deep-scrub starts 2024-03-26T04:08:31.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:31 smithi086 ceph-mon[32965]: 6.c deep-scrub ok 2024-03-26T04:08:31.740 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:31 smithi086 ceph-mon[32965]: pgmap v948: 158 pgs: 1 peering, 157 active+clean; 1.8 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T04:08:31.822 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:31.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:31.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[29446]: 6.c deep-scrub starts 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[29446]: 6.c deep-scrub ok 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[29446]: pgmap v948: 158 pgs: 1 peering, 157 active+clean; 1.8 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[37501]: 6.c deep-scrub starts 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[37501]: 6.c deep-scrub ok 2024-03-26T04:08:31.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:31 smithi006 ceph-mon[37501]: pgmap v948: 158 pgs: 1 peering, 157 active+clean; 1.8 GiB data, 9.3 GiB used, 438 GiB / 447 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-26T04:08:31.973 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:32.005 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:32.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:32.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:32.110 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 19 16 37268 37252 122.514 93.8125 0.0107285 0.00815533 2024-03-26T04:08:32.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:32.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:32.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:32.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:32.274 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-03-26T04:08:32.274 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-03-26T04:08:32.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-03-26T04:08:32.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [] 2024-03-26T04:08:32.274 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-03-26T04:08:32.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:32.415 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:32.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:32.578 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:32.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:32.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 3.f deep-scrub starts 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 3.f deep-scrub ok 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 4.18 scrub starts 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 4.18 scrub ok 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T04:08:32.808 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 3.f deep-scrub starts 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 3.f deep-scrub ok 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 4.18 scrub starts 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 4.18 scrub ok 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T04:08:32.809 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:32 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T04:08:32.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:32.836 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:08:32.836 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "" 2024-03-26T04:08:32.836 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:08:32.853 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:32.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:32.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:32.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:32.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:32.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:32.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 3.f deep-scrub starts 2024-03-26T04:08:32.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 3.f deep-scrub ok 2024-03-26T04:08:32.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 4.18 scrub starts 2024-03-26T04:08:32.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 4.18 scrub ok 2024-03-26T04:08:32.999 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T04:08:32.999 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:32 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T04:08:33.042 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T04:08:33.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:33.111 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:33.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:08:33.111903+0000 min lat: 0.00121124 max lat: 0.145786 avg lat: 0.00825338 2024-03-26T04:08:33.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:08:33.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 20 15 38769 38754 121.079 93.875 0.00941487 0.00825338 2024-03-26T04:08:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:33.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:33.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:33.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:33.388 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:33.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:33.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:33.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:33.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:33.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:33.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:33.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:33.793 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:33.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:33 smithi086 ceph-mon[32965]: pgmap v949: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 54 B/s, 1 objects/s recovering 2024-03-26T04:08:33.895 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:33.934 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:33.950 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:33 smithi006 ceph-mon[37501]: pgmap v949: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 54 B/s, 1 objects/s recovering 2024-03-26T04:08:33.950 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:33 smithi006 ceph-mon[29446]: pgmap v949: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 852 B/s rd, 87 MiB/s wr, 1.40k op/s; 54 B/s, 1 objects/s recovering 2024-03-26T04:08:34.036 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:34.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:34.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:34.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 21 16 40305 40289 119.882 95.9375 0.014101 0.00833541 2024-03-26T04:08:34.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:34.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:34.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:34.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:34.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:34.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:34.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:34.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:34.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:34.616 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:34.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:34.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:34.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:34.777 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:34.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:34.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:34.906 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:34.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:35.008 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:35.042 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[37501]: 4.1b deep-scrub starts 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[37501]: 4.1b deep-scrub ok 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[29446]: 4.1b deep-scrub starts 2024-03-26T04:08:35.043 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:34 smithi006 ceph-mon[29446]: 4.1b deep-scrub ok 2024-03-26T04:08:35.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:34 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:08:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:34 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:08:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:34 smithi086 ceph-mon[32965]: 4.1b deep-scrub starts 2024-03-26T04:08:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:34 smithi086 ceph-mon[32965]: 4.1b deep-scrub ok 2024-03-26T04:08:35.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 22 16 41810 41794 118.708 94.0625 0.0023302 0.0084138 2024-03-26T04:08:35.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:35.193 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:35.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:35.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:35.335 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:35.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:35.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:35.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:35.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:35.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:35.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:35.694 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:35.702 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:35.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:35.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:35.771 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:08:35.815 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:08:35.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:35.859 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:08:35.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:35.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:35.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:35.905 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:08:35.948 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[29446]: 3.e deep-scrub starts 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[29446]: 3.e deep-scrub ok 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[29446]: pgmap v950: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 656 B/s rd, 75 MiB/s wr, 1.20k op/s; 42 B/s, 1 objects/s recovering 2024-03-26T04:08:35.948 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[29446]: 5.2 deep-scrub starts 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[29446]: 5.2 deep-scrub ok 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[37501]: 3.e deep-scrub starts 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[37501]: 3.e deep-scrub ok 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[37501]: pgmap v950: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 656 B/s rd, 75 MiB/s wr, 1.20k op/s; 42 B/s, 1 objects/s recovering 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[37501]: 5.2 deep-scrub starts 2024-03-26T04:08:35.949 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:35 smithi006 ceph-mon[37501]: 5.2 deep-scrub ok 2024-03-26T04:08:35.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:35.989 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:08:36.014 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:36.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:36.027 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:08:36.069 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:08:36.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:08:36.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:08:36.077 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:08:36.078 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:08:36.078 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:08:36.078 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:08:36.078 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:08:36.078 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:08:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-mon[32965]: 3.e deep-scrub starts 2024-03-26T04:08:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-mon[32965]: 3.e deep-scrub ok 2024-03-26T04:08:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-mon[32965]: pgmap v950: 158 pgs: 158 active+clean; 2.2 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 656 B/s rd, 75 MiB/s wr, 1.20k op/s; 42 B/s, 1 objects/s recovering 2024-03-26T04:08:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-mon[32965]: 5.2 deep-scrub starts 2024-03-26T04:08:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:35 smithi086 ceph-mon[32965]: 5.2 deep-scrub ok 2024-03-26T04:08:36.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 23 16 43101 43085 117.055 80.6875 0.0145981 0.00853741 2024-03-26T04:08:36.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:36.157 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:36.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:36.259 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:36.296 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:36.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:36.439 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:36.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:36.570 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:36.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:36.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:36.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:36.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:36.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[29446]: 6.1b deep-scrub starts 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[29446]: 6.1b deep-scrub ok 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/13503245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[37501]: 6.1b deep-scrub starts 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[37501]: 6.1b deep-scrub ok 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:36.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/13503245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:36.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:36.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:36.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:36.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:37.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:37.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:36 smithi086 ceph-mon[32965]: 6.1b deep-scrub starts 2024-03-26T04:08:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:36 smithi086 ceph-mon[32965]: 6.1b deep-scrub ok 2024-03-26T04:08:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/13503245' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:37.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 24 16 44436 44420 115.654 83.4375 0.0132553 0.00864065 2024-03-26T04:08:37.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:37.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:37.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:37.458 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:37.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.511 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:37.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:37.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:37.639 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:08:37.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:37.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:37.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:37.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:37.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:37.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:38.041 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T04:08:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T04:08:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: pgmap v951: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:08:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: 6.a deep-scrub starts 2024-03-26T04:08:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: 6.a deep-scrub ok 2024-03-26T04:08:38.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:37 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:08:38.111 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 25 16 45854 45838 114.573 88.625 0.00808814 0.00872261 2024-03-26T04:08:38.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T04:08:38.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: pgmap v951: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: 6.a deep-scrub starts 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: 6.a deep-scrub ok 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: pgmap v951: 158 pgs: 158 active+clean; 2.6 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.85k op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: 6.a deep-scrub starts 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: 6.a deep-scrub ok 2024-03-26T04:08:38.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:37 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:08:38.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:38.170 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:38.272 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:38.311 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:38.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:38.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:38.583 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:38.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:38.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:38.719 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 7] out_osds: [6, 5, 3] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-03-26T04:08:38.719 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:08:38.719 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 4, 7] 2024-03-26T04:08:38.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd out 7 2024-03-26T04:08:38.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:38.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.864 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:38.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:38.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[29446]: osdmap e447: 8 total, 8 up, 5 in 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T04:08:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T04:08:38.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:38.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:08:38.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:38 smithi006 ceph-mon[37501]: osdmap e447: 8 total, 8 up, 5 in 2024-03-26T04:08:38.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:38.990 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:38.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/956401839' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:08:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:38 smithi086 ceph-mon[32965]: osdmap e447: 8 total, 8 up, 5 in 2024-03-26T04:08:39.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:39.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 26 16 47295 47279 113.628 90.0625 0.0122489 0.00879526 2024-03-26T04:08:39.118 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:39.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:39.247 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:39.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:39.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:39.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:39.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:39.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:39.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:39.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:39.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:39.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:39.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.819 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:39.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:39.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:39.945 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:39 smithi006 ceph-mon[37501]: pgmap v953: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 37 B/s, 1 objects/s recovering 2024-03-26T04:08:39.946 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:39 smithi006 ceph-mon[37501]: osdmap e448: 8 total, 8 up, 5 in 2024-03-26T04:08:39.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:39 smithi006 ceph-mon[29446]: pgmap v953: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 37 B/s, 1 objects/s recovering 2024-03-26T04:08:39.947 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:39 smithi006 ceph-mon[29446]: osdmap e448: 8 total, 8 up, 5 in 2024-03-26T04:08:39.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:39.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:40.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:39 smithi086 ceph-mon[32965]: pgmap v953: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.1 KiB/s rd, 111 MiB/s wr, 1.78k op/s; 37 B/s, 1 objects/s recovering 2024-03-26T04:08:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:39 smithi086 ceph-mon[32965]: osdmap e448: 8 total, 8 up, 5 in 2024-03-26T04:08:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:40.107 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:40.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 27 16 48607 48591 112.456 82 0.00847517 0.0088855 2024-03-26T04:08:40.180 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:40.180 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:40.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:40.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:40.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:40.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:40.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:40.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:08:40.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:40.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:40.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:40.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:40.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:40.770 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:40.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:40.901 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:40.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:40.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:41.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:41.030 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:41.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:41.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 28 16 49944 49928 111.424 83.5625 0.0105408 0.00896935 2024-03-26T04:08:41.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:41.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:41.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:08:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:41.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:41.298 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:41.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T04:08:41.400 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:41.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:41.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:41.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:41.597 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:41.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:41.746 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:41.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:41.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:41.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:41.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:42.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:41 smithi086 ceph-mon[32965]: pgmap v955: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-26T04:08:42.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 29 16 51227 51211 110.346 80.1875 0.0113943 0.00905695 2024-03-26T04:08:42.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:42.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:41 smithi006 ceph-mon[29446]: pgmap v955: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-26T04:08:42.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:41 smithi006 ceph-mon[37501]: pgmap v955: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-26T04:08:42.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:42.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:42.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:42.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:42.413 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:42.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:42.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:42.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:42.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:42.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:42.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:42.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:42.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:42.936 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:42.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:43.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:43.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:43.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:43.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 30 16 52391 52375 109.093 72.75 0.0129507 0.00916135 2024-03-26T04:08:43.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:43.192 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:43.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:43.320 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:43.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:43.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:43.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:43.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:43.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:43.704 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:43.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:43.833 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:43.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:43.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:43.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:43.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:44.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:43 smithi086 ceph-mon[32965]: pgmap v956: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T04:08:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.100 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:44.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:44.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 31 16 53679 53663 108.171 80.5 0.00989367 0.00923997 2024-03-26T04:08:44.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:43 smithi006 ceph-mon[37501]: pgmap v956: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T04:08:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:43 smithi006 ceph-mon[29446]: pgmap v956: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-26T04:08:44.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:44.242 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:44.320 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:44.330 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:44.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:44.419 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:44.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:44.532 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:08:44.548 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:44.589 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:44.649 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:44.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:44.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:44.803 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:44.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:44.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:44.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:44.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:45.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:45.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:45.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 32 16 54845 54829 107.068 72.875 0.0117611 0.00933515 2024-03-26T04:08:45.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:45.189 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:45.291 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.333 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:45.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:45.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:45.457 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:45.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:45.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:45.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:45.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:45.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:45.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:45.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:45.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:45.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:45.973 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:45.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:46.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:45 smithi086 ceph-mon[32965]: pgmap v957: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 61 MiB/s wr, 971 op/s 2024-03-26T04:08:46.102 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:46.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:46.112 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 33 16 55986 55970 105.984 71.3125 0.0109346 0.00943003 2024-03-26T04:08:46.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:45 smithi006 ceph-mon[37501]: pgmap v957: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 61 MiB/s wr, 971 op/s 2024-03-26T04:08:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:45 smithi006 ceph-mon[29446]: pgmap v957: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 639 B/s rd, 61 MiB/s wr, 971 op/s 2024-03-26T04:08:46.204 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:46.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:46.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:46.370 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:46.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:46.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:46.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:46.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:46.624 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:46.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.750 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:46.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:46.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:46.879 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:46.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:46.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:47.022 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:47.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 34 16 57026 57010 104.778 65 0.00798688 0.00952611 2024-03-26T04:08:47.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:47.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:47.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:47.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:47.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:47.312 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:47.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:47.442 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:47.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:47.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:47.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:47.687 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:47.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.718 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:47.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:47.860 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:47.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:47.989 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:47.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:47.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:47 smithi086 ceph-mon[32965]: pgmap v958: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:48.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:48.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 35 15 58089 58074 103.684 66.5 0.0185618 0.00963914 2024-03-26T04:08:48.128 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:48.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:47 smithi006 ceph-mon[37501]: pgmap v958: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:48.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:47 smithi006 ceph-mon[29446]: pgmap v958: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.2 KiB/s rd, 94 MiB/s wr, 1.51k op/s 2024-03-26T04:08:48.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:48.266 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:48.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:48.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:48.392 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:48.494 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:48.535 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:48.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:48.660 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:48.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:48.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:48.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:48.920 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:48.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:48.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:49.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:49.047 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:49.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:49.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 36 16 59177 59161 102.691 67.9375 0.0124267 0.00973252 2024-03-26T04:08:49.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:49.175 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:49.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:49.277 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.319 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:49.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:49.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:49.461 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:49.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:49.587 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:49.689 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:49.728 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:49.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:49.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.855 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:49.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:49.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:49 smithi006 ceph-mon[37501]: pgmap v959: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-26T04:08:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:49 smithi006 ceph-mon[29446]: pgmap v959: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-26T04:08:49.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:49.982 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:49.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:50.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:49 smithi086 ceph-mon[32965]: pgmap v959: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-26T04:08:50.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 37 16 60126 60110 101.519 59.3125 0.00842962 0.00984533 2024-03-26T04:08:50.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:50.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:50.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:50.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:50.393 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:50.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:50.521 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:50.623 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.666 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:50.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:50.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:50.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:50.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:50.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:50.933 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:50.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:50.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:51.035 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.085 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:51.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:51.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 38 16 61299 61283 100.776 73.3125 0.0153228 0.00991773 2024-03-26T04:08:51.186 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:51.217 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:51.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:51.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:51.348 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:51.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:51.449 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:51.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:51.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:51.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:51.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:51.771 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:51.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:51.902 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:51.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:51.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:52.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:52.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:51 smithi086 ceph-mon[32965]: pgmap v960: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T04:08:52.113 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 39 16 62615 62599 100.301 82.25 0.00922286 0.0099653 2024-03-26T04:08:52.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:52.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:51 smithi006 ceph-mon[37501]: pgmap v960: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T04:08:52.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:51 smithi006 ceph-mon[29446]: pgmap v960: 158 pgs: 158 active+clean; 3.5 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.32k op/s 2024-03-26T04:08:52.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:52.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:52.269 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:52.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:52.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:52.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:52.561 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:52.598 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:52.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:52.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:52.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:52.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:52.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:52.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:52.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:52.963 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:53.005 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:53.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:53.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:53.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:08:53.114511+0000 min lat: 0.00121124 max lat: 0.179374 avg lat: 0.0100271 2024-03-26T04:08:53.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:08:53.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 40 16 63821 63805 99.6778 75.375 0.0130007 0.0100271 2024-03-26T04:08:53.134 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:53.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:53.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:53.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:53.411 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:53.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:53.513 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:53.552 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:53.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:53.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:53.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:53.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:53.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.840 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:53.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:53.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:53.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:53.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:54.016 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:53 smithi006 ceph-mon[37501]: pgmap v961: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:08:54.016 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:53 smithi006 ceph-mon[29446]: pgmap v961: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:08:54.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:53 smithi086 ceph-mon[32965]: pgmap v961: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 853 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-26T04:08:54.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 41 16 65072 65056 99.1525 78.1875 0.0142434 0.0100796 2024-03-26T04:08:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.121 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:54.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:54.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:54.249 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:54.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:54.391 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:54.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:54.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:54.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:54.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:54.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:08:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:08:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:08:54.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:54.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:54.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:54.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:54.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:54.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:54.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:55.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:55.083 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:55.083 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:08:55.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 42 16 66315 66299 98.6411 77.6875 0.00975414 0.0101309 2024-03-26T04:08:55.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:55.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:55.218 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:55.242 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:55.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:55.370 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:55.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:55.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:55.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:55.649 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:55.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:55.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:55.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:55.818 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:08:55.850 INFO:teuthology.orchestra.run.smithi006.stderr:marked out osd.7. 2024-03-26T04:08:55.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:55.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:55.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:55.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:56.005 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:56.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-mon[32965]: pgmap v962: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.6 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-26T04:08:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2115334499' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:55 smithi086 ceph-mon[32965]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T04:08:56.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 43 16 67038 67022 97.398 45.1875 0.00294948 0.010195 2024-03-26T04:08:56.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[37501]: pgmap v962: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.6 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2115334499' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[37501]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:08:55.710+0000 7fd465f8b700 -1 mon.a@0(leader).osd e448 definitely_dead 0 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[29446]: pgmap v962: 158 pgs: 158 active+clean; 3.7 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 4.6 KiB/s rd, 64 MiB/s wr, 1.03k op/s 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2115334499' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["7"]}]: dispatch 2024-03-26T04:08:56.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:55 smithi006 ceph-mon[29446]: Client client.admin marked osd.7 out, while it was still marked up 2024-03-26T04:08:56.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:56.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:56.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:56.280 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:56.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:56.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:56.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:56.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:56.531 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:56.633 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:56.657 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:56.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:56.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:56.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:56.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:56.887 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:56.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:56.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:56.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:57.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:57.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:57.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: osdmap e449: 8 total, 8 up, 4 in 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:56 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:57.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 44 16 67038 67022 95.1846 0 - 0.010195 2024-03-26T04:08:57.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: osdmap e449: 8 total, 8 up, 4 in 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["7"]}]': finished 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: osdmap e449: 8 total, 8 up, 4 in 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:08:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:08:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:56 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:08:57.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:57.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.170 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:57.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:57.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:08:57.294 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:57.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:08:57.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:57.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.420 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:57.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:57.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:57.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:57.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:57.672 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:57.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:57.774 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:08:57.798 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:57.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:08:57.862 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:08:57.862 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:08:57.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:57.927 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:57.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:57.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:58.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:58.053 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:58.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: pgmap v964: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 71 MiB/s wr, 1.16k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: pgmap v965: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 79 MiB/s wr, 1.30k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: pgmap v966: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 91 MiB/s wr, 1.50k op/s; 25 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: osdmap e450: 8 total, 8 up, 4 in 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T04:08:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: osdmap e451: 8 total, 8 up, 4 in 2024-03-26T04:08:58.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:57 smithi086 ceph-mon[32965]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 448'4210 2024-03-26T04:08:58.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 45 16 67075 67059 93.1209 1.15625 0.0133358 0.0105751 2024-03-26T04:08:58.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: pgmap v964: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 71 MiB/s wr, 1.16k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: pgmap v965: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 79 MiB/s wr, 1.30k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: pgmap v966: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 91 MiB/s wr, 1.50k op/s; 25 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: osdmap e450: 8 total, 8 up, 4 in 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: osdmap e451: 8 total, 8 up, 4 in 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[37501]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 448'4210 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: pgmap v964: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 71 MiB/s wr, 1.16k op/s; 16 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: pgmap v965: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 79 MiB/s wr, 1.30k op/s; 19 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: pgmap v966: 158 pgs: 42 peering, 3 unknown, 113 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 91 MiB/s wr, 1.50k op/s; 25 KiB/s, 0 objects/s recovering 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: osdmap e450: 8 total, 8 up, 4 in 2024-03-26T04:08:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-26T04:08:58.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: osdmap e451: 8 total, 8 up, 4 in 2024-03-26T04:08:58.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:57 smithi006 ceph-mon[29446]: 11.2 starting backfill to osd.1 from (0'0,0'0] MAX to 448'4210 2024-03-26T04:08:58.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:58.184 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:58.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:58.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:58.309 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:58.411 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:58.452 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:58.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:58.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:08:58.585 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:58.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:08:58.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:58.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:58.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:58.814 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:58.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:58.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:58.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:58.977 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:58.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:58.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:59.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:59.106 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:59.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:59.114 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 46 16 67583 67567 91.7868 31.75 0.0119175 0.010889 2024-03-26T04:08:59.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.5 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4264 2024-03-26T04:08:59.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.9 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4215 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.0 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4228 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 448'4183 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.3 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4197 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4115 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 448'4161 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[37501]: osdmap e452: 8 total, 8 up, 4 in 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.5 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4264 2024-03-26T04:08:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.9 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4215 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.0 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4228 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 448'4183 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.3 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4197 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4115 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 448'4161 2024-03-26T04:08:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:58 smithi006 ceph-mon[29446]: osdmap e452: 8 total, 8 up, 4 in 2024-03-26T04:08:59.208 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:08:59.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.255 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:59.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:08:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.5 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4264 2024-03-26T04:08:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.9 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4215 2024-03-26T04:08:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.0 starting backfill to osd.2 from (0'0,0'0] MAX to 448'4228 2024-03-26T04:08:59.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.e starting backfill to osd.0 from (0'0,0'0] MAX to 448'4183 2024-03-26T04:08:59.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.3 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4197 2024-03-26T04:08:59.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.4 starting backfill to osd.0 from (0'0,0'0] MAX to 448'4115 2024-03-26T04:08:59.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: 11.f starting backfill to osd.0 from (0'0,0'0] MAX to 448'4161 2024-03-26T04:08:59.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:58 smithi086 ceph-mon[32965]: osdmap e452: 8 total, 8 up, 4 in 2024-03-26T04:08:59.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:08:59.381 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:08:59.483 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:08:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.543 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:08:59.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:08:59.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:59.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:59.762 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:08:59.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:08:59.798 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:08:59.811 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:08:59.858 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:08:59.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:08:59.902 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:08:59.930 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:08:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:08:59.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:08:59.953 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:08:59.984 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[29446]: pgmap v969: 158 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 42 peering, 3 unknown, 109 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 88 MiB/s wr, 1.46k op/s; 8251/201615 objects misplaced (4.092%); 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/729379348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[29446]: osdmap e453: 8 total, 8 up, 4 in 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[37501]: pgmap v969: 158 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 42 peering, 3 unknown, 109 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 88 MiB/s wr, 1.46k op/s; 8251/201615 objects misplaced (4.092%); 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/729379348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:08:59.985 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:08:59 smithi006 ceph-mon[37501]: osdmap e453: 8 total, 8 up, 4 in 2024-03-26T04:09:00.024 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:09:00.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:00.063 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:09:00.065 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:09:00.066 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:09:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:59 smithi086 ceph-mon[32965]: pgmap v969: 158 pgs: 2 active+remapped+backfilling, 2 active+undersized+remapped, 42 peering, 3 unknown, 109 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 88 MiB/s wr, 1.46k op/s; 8251/201615 objects misplaced (4.092%); 2.0 MiB/s, 31 objects/s recovering 2024-03-26T04:09:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/729379348' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:09:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:08:59 smithi086 ceph-mon[32965]: osdmap e453: 8 total, 8 up, 4 in 2024-03-26T04:09:00.115 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 47 16 68371 68355 90.8818 49.25 0.0039736 0.0109578 2024-03-26T04:09:00.168 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:00.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:00.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:00.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:00.328 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:00.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:00.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:00.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:00.608 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:00.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:00.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T04:09:00.710 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:00.755 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:00.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:00.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:00.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:00.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:00.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:01.012 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:01.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:01.116 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 48 16 69192 69176 90.0556 51.3125 0.0200056 0.0110987 2024-03-26T04:09:01.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:00 smithi006 ceph-mon[37501]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:00 smithi006 ceph-mon[29446]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:01.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:01.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:01.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:01.330 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:00 smithi086 ceph-mon[32965]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:01.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:01.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:01.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:01.547 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:01.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:01.578 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4] out_osds: [6, 5, 3, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-03-26T04:09:01.579 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:09:01.579 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-26T04:09:01.579 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-26T04:09:01.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:09:01.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:01.675 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:01.704 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:01.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:01.801 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:01.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:01.925 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:09:01.928 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:01.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:02.030 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:02.082 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:02.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:02.116 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 49 16 70140 70124 89.4268 59.25 0.0165226 0.0111766 2024-03-26T04:09:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[37501]: pgmap v972: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+remapped, 43 peering, 3 unknown, 107 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 174 op/s; 12535/202119 objects misplaced (6.202%); 1.5 MiB/s, 32 objects/s recovering 2024-03-26T04:09:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3991749346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[29446]: pgmap v972: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+remapped, 43 peering, 3 unknown, 107 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 174 op/s; 12535/202119 objects misplaced (6.202%); 1.5 MiB/s, 32 objects/s recovering 2024-03-26T04:09:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3991749346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:01 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:02.210 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:02.312 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:01 smithi086 ceph-mon[32965]: pgmap v972: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+remapped, 43 peering, 3 unknown, 107 active+clean; 4.1 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 174 op/s; 12535/202119 objects misplaced (6.202%); 1.5 MiB/s, 32 objects/s recovering 2024-03-26T04:09:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3991749346' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:01 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:09:02.352 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:02.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:02.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:02.580 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:02.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:02.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:02.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:02.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:02.851 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:02.897 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:02.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:02.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:03.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:03.116 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 50 16 71216 71200 88.9832 67.25 0.0142343 0.0112322 2024-03-26T04:09:03.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:03.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:03.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:09:03.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[29446]: osdmap e454: 8 total, 8 up, 4 in 2024-03-26T04:09:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:09:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:02 smithi006 ceph-mon[37501]: osdmap e454: 8 total, 8 up, 4 in 2024-03-26T04:09:03.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.164 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:03.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:03.265 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:03.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:02 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:02 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:09:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:02 smithi086 ceph-mon[32965]: osdmap e454: 8 total, 8 up, 4 in 2024-03-26T04:09:03.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:03.435 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:03.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:03.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:03.580 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:03.662 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:09:03.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:03.711 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:03.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.792 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:03.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:03.893 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:03.961 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:03.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:03.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:04.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:04.087 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:04.117 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 51 15 72172 72157 88.41 59.8125 0.015698 0.0113061 2024-03-26T04:09:04.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:03 smithi006 ceph-mon[29446]: pgmap v974: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 2 active+undersized+remapped, 1 peering, 147 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 9.6 KiB/s rd, 29 MiB/s wr, 482 op/s; 32239/207117 objects misplaced (15.566%); 27 MiB/s, 459 objects/s recovering 2024-03-26T04:09:04.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:03 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-26T04:09:04.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:03 smithi006 ceph-mon[37501]: pgmap v974: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 2 active+undersized+remapped, 1 peering, 147 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 9.6 KiB/s rd, 29 MiB/s wr, 482 op/s; 32239/207117 objects misplaced (15.566%); 27 MiB/s, 459 objects/s recovering 2024-03-26T04:09:04.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:03 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-26T04:09:04.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:04.214 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:04.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:03 smithi086 ceph-mon[32965]: pgmap v974: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 2 active+undersized+remapped, 1 peering, 147 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 9.6 KiB/s rd, 29 MiB/s wr, 482 op/s; 32239/207117 objects misplaced (15.566%); 27 MiB/s, 459 objects/s recovering 2024-03-26T04:09:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:03 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-26T04:09:04.341 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:04.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:04.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:04.530 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "size": 20, 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 600, 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241769 10.9 10:985cbb0c:::benchmark_data_smithi006_159841_object69674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.283575+0000", 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.24353395899999, 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.030854519, 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.531 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241769 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283575+0000", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283575+0000", 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.532 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283576+0000", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.341e-06 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283581+0000", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2459999999999997e-06 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283590+0000", 2024-03-26T04:09:04.533 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2029999999999998e-06 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283597+0000", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3470000000000001e-06 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283650+0000", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3322999999999998e-05 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.283659+0000", 2024-03-26T04:09:04.534 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4190000000000002e-06 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.308456+0000", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0247968999999999 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.308573+0000", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011713 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.308693+0000", 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012008899999999999 2024-03-26T04:09:04.535 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.311888+0000", 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0031951330000000002 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314288+0000", 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0023998259999999999 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.536 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314368+0000", 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9942000000000001e-05 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314399+0000", 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0658999999999999e-05 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314430+0000", 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0960999999999997e-05 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.537 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241775 10.9 10:90a38b23:::benchmark_data_smithi006_159841_object69680:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.287900+0000", 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.23920894, 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0267337430000001, 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241775 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.538 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287900+0000", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287900+0000", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287902+0000", 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.885e-06 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.539 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287907+0000", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4040000000000003e-06 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287918+0000", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0798999999999999e-05 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287927+0000", 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1349999999999998e-06 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.540 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287983+0000", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5572000000000003e-05 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.287991+0000", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3229999999999998e-06 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.310578+0000", 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.022586674 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.541 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.310698+0000", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120265 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.310819+0000", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000120511 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314439+0000", 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0036203279999999999 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.542 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314494+0000", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5142999999999997e-05 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314549+0000", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5095e-05 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314601+0000", 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1442e-05 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.543 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314634+0000", 2024-03-26T04:09:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3167000000000003e-05 2024-03-26T04:09:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.544 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241779 10.9 10:92d6f17d:::benchmark_data_smithi006_159841_object69684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.291127+0000", 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.23598213299999, 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0249830289999999, 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241779 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.545 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291127+0000", 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291127+0000", 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.546 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291128+0000", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.494e-06 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291133+0000", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.882e-06 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291144+0000", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0339e-05 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291151+0000", 2024-03-26T04:09:04.547 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2540000000000003e-06 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291224+0000", 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3286000000000003e-05 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291232+0000", 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3429999999999996e-06 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.548 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.311930+0000", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.020697685 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.312050+0000", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012023000000000001 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.312174+0000", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012386999999999999 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.549 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314445+0000", 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0022706240000000002 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.315981+0000", 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0015360129999999999 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316024+0000", 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3166e-05 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.550 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316071+0000", 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7018000000000003e-05 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316110+0000", 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8825000000000002e-05 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241780 10.9 10:93d57ef2:::benchmark_data_smithi006_159841_object69685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.551 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.291169+0000", 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.23594023800001, 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.029188435, 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241780 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.552 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291169+0000", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291169+0000", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291169+0000", 2024-03-26T04:09:04.553 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5900000000000002e-07 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291172+0000", 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6639999999999998e-06 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291177+0000", 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6590000000000004e-06 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.554 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291183+0000", 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3290000000000001e-06 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291255+0000", 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.2206999999999995e-05 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.555 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.291261+0000", 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0299999999999999e-06 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.313169+0000", 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0219081059999999 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.313263+0000", 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4240000000000006e-05 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.556 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.313328+0000", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5060000000000004e-05 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316146+0000", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.002817417 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316180+0000", 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3866999999999999e-05 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320193+0000", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0040128239999999999 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320310+0000", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011743000000000001 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320357+0000", 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7142999999999999e-05 2024-03-26T04:09:04.558 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241801 10.9 10:9f83e62f:::benchmark_data_smithi006_159841_object69706:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.306137+0000", 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.220972179, 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0279442190000001, 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.559 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241801 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306137+0000", 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306137+0000", 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.560 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306139+0000", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.8059999999999999e-06 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306154+0000", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5056e-05 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306170+0000", 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.6087999999999999e-05 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.561 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306178+0000", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1689999999999992e-06 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306723+0000", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00054470400000000002 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.306732+0000", 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4299999999999995e-06 2024-03-26T04:09:04.562 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314661+0000", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0079285520000001 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314763+0000", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010233100000000001 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.314876+0000", 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113455 2024-03-26T04:09:04.563 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318396+0000", 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0035197980000000002 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318437+0000", 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0717999999999999e-05 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.564 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.333981+0000", 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015544138000000001 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.334034+0000", 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3353999999999998e-05 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.334081+0000", 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6619999999999997e-05 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.565 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241814 10.9 10:92d4c9c3:::benchmark_data_smithi006_159841_object69719:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.314110+0000", 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.212999182, 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.020044706, 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241814 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.566 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314110+0000", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314110+0000", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314111+0000", 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3829999999999999e-06 2024-03-26T04:09:04.567 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314115+0000", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0500000000000002e-06 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314125+0000", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6090000000000007e-06 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314132+0000", 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.7780000000000002e-06 2024-03-26T04:09:04.568 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314184+0000", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1949e-05 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.314192+0000", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2910000000000007e-06 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316206+0000", 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0020144630000001 2024-03-26T04:09:04.569 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316303+0000", 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6983000000000003e-05 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.316400+0000", 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6643000000000003e-05 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.570 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318474+0000", 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0020743739999999999 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318510+0000", 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5938e-05 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.571 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.334089+0000", 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015578953 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.334121+0000", 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1862000000000001e-05 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.572 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.334154+0000", 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.3429999999999997e-05 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241842 10.9 10:9baf980e:::benchmark_data_smithi006_159841_object69747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.330715+0000", 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.196394097, 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.006746363, 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.573 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241842 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330715+0000", 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.574 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330715+0000", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330717+0000", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7120000000000001e-06 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330722+0000", 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3229999999999997e-06 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.575 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330742+0000", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9894999999999998e-05 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330750+0000", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3189999999999995e-06 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330868+0000", 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117935 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.576 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.330877+0000", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2450000000000004e-06 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318539+0000", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98766206000000001 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318648+0000", 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000108171 2024-03-26T04:09:04.577 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.318758+0000", 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110054 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.322052+0000", 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032946109999999998 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.578 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.322099+0000", 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7274000000000002e-05 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337369+0000", 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.015269084 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337421+0000", 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2135000000000001e-05 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.579 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337461+0000", 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0544999999999999e-05 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241855 10.9 10:9ba5a85f:::benchmark_data_smithi006_159841_object69760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.340143+0000", 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.18696611999999, 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99737226199999995, 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.580 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241855 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340143+0000", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340143+0000", 2024-03-26T04:09:04.581 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340144+0000", 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.584e-06 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340150+0000", 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.344e-06 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.582 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340161+0000", 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.113e-05 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340168+0000", 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4510000000000003e-06 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340224+0000", 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.5229000000000003e-05 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.583 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.340232+0000", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3410000000000003e-06 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320396+0000", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.98016376699999996 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320497+0000", 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000101556 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.584 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.320604+0000", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010726399999999999 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.323910+0000", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003305887 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.323960+0000", 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9459999999999997e-05 2024-03-26T04:09:04.585 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337468+0000", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.013508328 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337495+0000", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6874999999999999e-05 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337515+0000", 2024-03-26T04:09:04.586 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0046000000000001e-05 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:241859 10.9 10:9a5d8854:::benchmark_data_smithi006_159841_object69764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e392)", 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:06:18.342868+0000", 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 166.184240445, 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99472012799999998, 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.587 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 241859 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342868+0000", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342868+0000", 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.588 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342870+0000", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.48e-06 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342875+0000", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0359999999999997e-06 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342886+0000", 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0994e-05 2024-03-26T04:09:04.589 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342894+0000", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0199999999999994e-06 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342951+0000", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6975000000000003e-05 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:18.342961+0000", 2024-03-26T04:09:04.590 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.6719999999999999e-06 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.322132+0000", 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.97917147699999996 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.322238+0000", 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106337 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 4,7", 2024-03-26T04:09:04.591 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.322360+0000", 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012115799999999999 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.324014+0000", 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016548190000000001 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.327407+0000", 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003393021 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.592 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337519+0000", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.010111505999999999 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337551+0000", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.2354000000000002e-05 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:06:19.337589+0000", 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7279e-05 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.593 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343008 10.7 10:e75b0d70:::benchmark_data_smithi006_159841_object170913:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.550198+0000", 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.976910826999998, 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0318863149999999, 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "reached pg", 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.594 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343008 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550198+0000", 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550198+0000", 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.595 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550200+0000", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5510000000000001e-06 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550207+0000", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.114e-06 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550219+0000", 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1737e-05 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.596 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550228+0000", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.0110000000000006e-06 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550371+0000", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00014365800000000001 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op must wait for map", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.550379+0000", 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.038e-06 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.597 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.559485+0000", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091053439999999996 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.559494+0000", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9649999999999997e-06 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.582026+0000", 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0225322020000001 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.598 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.582084+0000", 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8695e-05 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.599 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343028 10.7 10:e96333e5:::benchmark_data_smithi006_159841_object170933:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.560176+0000", 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.966933268999995, 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0271609589999999, 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.601 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343028 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560176+0000", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560176+0000", 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.602 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560177+0000", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.5099999999999999e-06 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560184+0000", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4459999999999998e-06 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560194+0000", 2024-03-26T04:09:04.603 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0088999999999999e-05 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.560202+0000", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1510000000000004e-06 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.565203+0000", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0050007089999999999 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.565211+0000", 2024-03-26T04:09:04.604 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5969999999999992e-06 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.582548+0000", 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0173366420000001 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.582662+0000", 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114009 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.605 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.582766+0000", 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010399 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587130+0000", 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0043645510000000004 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587267+0000", 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000136978 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.606 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587278+0000", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0633e-05 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587308+0000", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9964e-05 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587337+0000", 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8690000000000001e-05 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343044 10.7 10:ecea8d1b:::benchmark_data_smithi006_159841_object170949:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.579947+0000", 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.947161445999996, 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0091601050000001, 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343044 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.608 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579947+0000", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579947+0000", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579949+0000", 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.082e-06 2024-03-26T04:09:04.609 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579952+0000", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.7790000000000002e-06 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579964+0000", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2111e-05 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.579972+0000", 2024-03-26T04:09:04.610 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7670000000000003e-06 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.580029+0000", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6644999999999997e-05 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.580037+0000", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0800000000000006e-06 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.584099+0000", 2024-03-26T04:09:04.611 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0040619150000001 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.584231+0000", 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000132397 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.584362+0000", 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000130438 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.612 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587224+0000", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0028625880000000001 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587346+0000", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00012209699999999999 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589012+0000", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001665393 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.613 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589065+0000", 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3001999999999998e-05 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589108+0000", 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2811000000000001e-05 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343059 10.7 10:e3249f71:::benchmark_data_smithi006_159841_object170964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.599670+0000", 2024-03-26T04:09:04.614 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.927439211000006, 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.99198912900000003, 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343059 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599670+0000", 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.615 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599670+0000", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599671+0000", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.3379999999999999e-06 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599677+0000", 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7409999999999998e-06 2024-03-26T04:09:04.616 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599687+0000", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0107e-05 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599695+0000", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8890000000000002e-06 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599772+0000", 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7564999999999993e-05 2024-03-26T04:09:04.617 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.599781+0000", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4950000000000008e-06 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.585677+0000", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.985896573 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.585791+0000", 2024-03-26T04:09:04.618 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000113109 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.585894+0000", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000103217 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.588945+0000", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0030508269999999999 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.588953+0000", 2024-03-26T04:09:04.619 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.585e-06 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591569+0000", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026155190000000002 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591616+0000", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6941999999999999e-05 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591659+0000", 2024-03-26T04:09:04.620 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3222000000000001e-05 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343141 10.7 10:eb924b51:::benchmark_data_smithi006_159841_object171046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.645229+0000", 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.881879654000002, 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.94654198099999998, 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.621 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343141 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645229+0000", 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645229+0000", 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.622 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645230+0000", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8800000000000001e-07 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645235+0000", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6770000000000003e-06 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645245+0000", 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.8549999999999997e-06 2024-03-26T04:09:04.623 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645252+0000", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6769999999999995e-06 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645338+0000", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.5836999999999998e-05 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.645346+0000", 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.0390000000000005e-06 2024-03-26T04:09:04.624 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587390+0000", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.94204323999999995 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587497+0000", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010789300000000001 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.587607+0000", 2024-03-26T04:09:04.625 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000109576 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591520+0000", 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0039126999999999999 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591674+0000", 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00015405700000000001 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.626 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591720+0000", 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6412000000000002e-05 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591748+0000", 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8245000000000001e-05 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591771+0000", 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.2884999999999999e-05 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.627 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343151 10.7 10:eeef14f0:::benchmark_data_smithi006_159841_object171056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.650791+0000", 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.876318096999995, 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.94421579499999997, 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343151 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.628 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650791+0000", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650791+0000", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650792+0000", 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1999999999999998e-07 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.629 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650795+0000", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.0620000000000001e-06 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650801+0000", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5189999999999998e-06 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650807+0000", 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6330000000000001e-06 2024-03-26T04:09:04.630 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650874+0000", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6556000000000004e-05 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.650882+0000", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.4270000000000008e-06 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589154+0000", 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.93827163400000002 2024-03-26T04:09:04.631 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589249+0000", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.5194999999999999e-05 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.589337+0000", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8688000000000002e-05 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.593179+0000", 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0038410610000000002 2024-03-26T04:09:04.632 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.594851+0000", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0016722970000000001 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.594901+0000", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9808000000000002e-05 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.594952+0000", 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1136e-05 2024-03-26T04:09:04.633 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.595007+0000", 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4858999999999998e-05 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343159 10.7 10:e897c7de:::benchmark_data_smithi006_159841_object171064:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.654832+0000", 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.872277359999998, 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.94644051100000004, 2024-03-26T04:09:04.634 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343159 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654832+0000", 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.635 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654832+0000", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654833+0000", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2759999999999999e-06 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654838+0000", 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8779999999999997e-06 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.636 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654849+0000", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1656e-05 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654858+0000", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2360000000000004e-06 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654923+0000", 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5458999999999998e-05 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.637 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.654931+0000", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3920000000000003e-06 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.590394+0000", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.93546256800000005 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.590462+0000", 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.8448000000000006e-05 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.638 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.590522+0000", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9917999999999998e-05 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.593184+0000", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026617720000000002 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.595065+0000", 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0018809810000000001 2024-03-26T04:09:04.639 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601182+0000", 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0061166290000000002 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.640 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601232+0000", 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9758e-05 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601272+0000", 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.0540000000000001e-05 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.642 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343188 10.7 10:eedd784d:::benchmark_data_smithi006_159841_object171093:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.672789+0000", 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.854319739000005, 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.92868162600000004, 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343188 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.643 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672789+0000", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672789+0000", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672790+0000", 2024-03-26T04:09:04.644 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6499999999999998e-07 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672795+0000", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7759999999999997e-06 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672805+0000", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.9399999999999997e-06 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672813+0000", 2024-03-26T04:09:04.645 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.8159999999999994e-06 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672862+0000", 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9712000000000001e-05 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.672871+0000", 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.9360000000000005e-06 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.646 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591812+0000", 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91894133099999997 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.591937+0000", 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000124034 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.592058+0000", 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000121753 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.647 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596645+0000", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0045866179999999998 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596681+0000", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.5828999999999999e-05 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601382+0000", 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0047016289999999997 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.648 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601432+0000", 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.9867999999999998e-05 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601471+0000", 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.8618999999999997e-05 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343204 10.7 10:e54528ec:::benchmark_data_smithi006_159841_object171109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.649 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.686370+0000", 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.840739190999997, 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91533743499999998, 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343204 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.650 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686370+0000", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686370+0000", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686371+0000", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.2099999999999995e-07 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.651 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686375+0000", 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8489999999999997e-06 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686385+0000", 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1309999999999995e-06 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686392+0000", 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.357e-06 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.652 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686440+0000", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.8043000000000001e-05 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.686448+0000", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.9910000000000002e-06 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.593211+0000", 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.90676355099999995 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.653 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.593306+0000", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.4017000000000005e-05 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.593398+0000", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.2188000000000006e-05 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596692+0000", 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0032943249999999999 2024-03-26T04:09:04.654 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596721+0000", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.9247999999999998e-05 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601573+0000", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0048522030000000002 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601660+0000", 2024-03-26T04:09:04.655 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.6403999999999995e-05 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601707+0000", 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7307e-05 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343206 10.7 10:e5ee698c:::benchmark_data_smithi006_159841_object171111:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.689547+0000", 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.837562027999994, 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.91233235999999995, 2024-03-26T04:09:04.656 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343206 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689547+0000", 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.657 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689547+0000", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689548+0000", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.209e-06 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689553+0000", 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3700000000000003e-06 2024-03-26T04:09:04.658 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689563+0000", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1269999999999992e-06 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689568+0000", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6210000000000001e-06 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689622+0000", 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4086999999999999e-05 2024-03-26T04:09:04.659 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.689631+0000", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8710000000000003e-06 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.595110+0000", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.90547836900000001 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.595224+0000", 2024-03-26T04:09:04.660 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000114103 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.595336+0000", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011246100000000001 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596755+0000", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0014183959999999999 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601141+0000", 2024-03-26T04:09:04.661 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.004386455 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601781+0000", 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00064023699999999999 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601836+0000", 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.4778999999999998e-05 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.662 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601879+0000", 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.3275000000000002e-05 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.15540.0:343233 10.7 10:ee077faa:::benchmark_data_smithi006_159841_object171138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e420)", 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:07:34.708356+0000", 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 89.818753305000001, 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.89365871100000005, 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-26T04:09:04.663 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.15540", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/3534710028", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 343233 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708356+0000", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708356+0000", 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:09:04.664 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708357+0000", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2899999999999999e-06 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708362+0000", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0640000000000001e-06 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708374+0000", 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.2126999999999999e-05 2024-03-26T04:09:04.665 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708382+0000", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.7880000000000007e-06 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708443+0000", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.0921999999999997e-05 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for peered", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:34.708451+0000", 2024-03-26T04:09:04.666 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3629999999999994e-06 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596801+0000", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.88834983700000003 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.596905+0000", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010357300000000001 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.597042+0000", 2024-03-26T04:09:04.667 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000137009 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601290+0000", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0042482090000000002 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601331+0000", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.087e-05 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601941+0000", 2024-03-26T04:09:04.668 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00061075699999999997 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "commit_sent", 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.601970+0000", 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8428e-05 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "done", 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:07:35.602014+0000", 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4474000000000001e-05 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:09:04.669 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:09:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.670 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:04.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:09:04.772 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:04.816 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:04.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.817 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:04.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:04.944 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:04.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:04.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:05.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:05.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:05.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 52 16 73111 73095 87.836 58.625 0.0113674 0.0113799 2024-03-26T04:09:05.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:05.198 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:05.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:05.300 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:05.345 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:05.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:05.493 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:05.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:05.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:05.607 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:05 smithi006 ceph-mon[29446]: pgmap v975: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 149 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 682 B/s rd, 27 MiB/s wr, 440 op/s; 31458/209532 objects misplaced (15.013%); 34 MiB/s, 569 objects/s recovering 2024-03-26T04:09:05.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:05.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:05.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:05.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:05.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:05 smithi086 ceph-mon[32965]: pgmap v975: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 149 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 682 B/s rd, 27 MiB/s wr, 440 op/s; 31458/209532 objects misplaced (15.013%); 34 MiB/s, 569 objects/s recovering 2024-03-26T04:09:05.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:05.866 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:05.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:05 smithi006 ceph-mon[37501]: pgmap v975: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 1 peering, 149 active+clean; 4.2 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 682 B/s rd, 27 MiB/s wr, 440 op/s; 31458/209532 objects misplaced (15.013%); 34 MiB/s, 569 objects/s recovering 2024-03-26T04:09:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:05.910 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:05.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:06.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:06.037 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:06.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:06.117 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 53 16 74159 74143 87.4145 65.5 0.008829 0.0114347 2024-03-26T04:09:06.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:06.164 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:06.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:06.290 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:06.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:06.392 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:06.440 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:06.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:06.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:06.566 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:06.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:06.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:06.693 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:06.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:06.795 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.840 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:06.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:06.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:06.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:06.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:06.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[29446]: osdmap e455: 8 total, 8 up, 4 in 2024-03-26T04:09:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:06.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:06 smithi006 ceph-mon[37501]: osdmap e455: 8 total, 8 up, 4 in 2024-03-26T04:09:06.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:06.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:06.983 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:06.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:07.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:06 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:06 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:06 smithi086 ceph-mon[32965]: osdmap e455: 8 total, 8 up, 4 in 2024-03-26T04:09:07.117 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 54 16 75149 75133 86.9415 61.875 0.013452 0.0114961 2024-03-26T04:09:07.128 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:07.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:07.256 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:07.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.256 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:07.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:07.382 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:07.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-26T04:09:07.435 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:07.451 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:07.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:07.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:07.635 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:07.737 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:07.814 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:07.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:07.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:07 smithi006 ceph-mon[29446]: pgmap v977: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 935 op/s; 31564/221472 objects misplaced (14.252%); 44 MiB/s, 712 objects/s recovering 2024-03-26T04:09:07.853 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:07 smithi006 ceph-mon[37501]: pgmap v977: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 935 op/s; 31564/221472 objects misplaced (14.252%); 44 MiB/s, 712 objects/s recovering 2024-03-26T04:09:07.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:07.941 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:07.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:07.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:08.043 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:08.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.084 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:08.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:07 smithi086 ceph-mon[32965]: pgmap v977: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.4 KiB/s rd, 58 MiB/s wr, 935 op/s; 31564/221472 objects misplaced (14.252%); 44 MiB/s, 712 objects/s recovering 2024-03-26T04:09:08.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 55 16 76073 76057 86.4096 57.75 0.0158691 0.0115672 2024-03-26T04:09:08.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:08.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:08.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:08.363 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:08.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:08.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:08.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:08.634 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:08.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:08.761 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:08.862 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:08.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:08.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:08.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:09.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:09.056 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:09.118 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 56 16 77107 77091 86.0206 64.625 0.0149637 0.0116198 2024-03-26T04:09:09.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:09.182 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:09.284 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:09.327 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:09.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:09.471 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:09.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:09.612 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:09.714 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:09.856 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:09.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:09.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:09.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[37501]: pgmap v978: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 922 op/s; 31243/224244 objects misplaced (13.933%); 44 MiB/s, 718 objects/s recovering 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[37501]: osdmap e456: 8 total, 8 up, 4 in 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[29446]: pgmap v978: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 922 op/s; 31243/224244 objects misplaced (13.933%); 44 MiB/s, 718 objects/s recovering 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:09.938 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:09 smithi006 ceph-mon[29446]: osdmap e456: 8 total, 8 up, 4 in 2024-03-26T04:09:10.007 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:10.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:09 smithi086 ceph-mon[32965]: pgmap v978: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.5 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 922 op/s; 31243/224244 objects misplaced (13.933%); 44 MiB/s, 718 objects/s recovering 2024-03-26T04:09:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:09 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:09 smithi086 ceph-mon[32965]: osdmap e456: 8 total, 8 up, 4 in 2024-03-26T04:09:10.118 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 57 16 78050 78034 85.5454 58.9375 0.0233803 0.0116842 2024-03-26T04:09:10.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:10.177 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:10.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:10.294 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:10.303 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:10.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:10.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:10.429 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:10.447 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:09:10.473 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:09:10.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:10.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.582 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:10.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:10.683 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:10.688 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:09:10.731 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:10.737 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:09:10.781 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:09:10.826 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:09:10.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:10.867 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:09:10.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:10.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:10.909 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:09:10.951 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:09:10.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:10.994 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:09:10.999 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:09:10.999 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:09:10.999 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:09:10.999 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:09:11.000 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:09:11.000 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:09:11.000 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:09:11.000 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:09:11.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:11.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 58 15 78905 78890 84.9929 53.5 0.00951903 0.0117603 2024-03-26T04:09:11.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:11.169 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:11.274 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:11.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:11.372 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":979,"stamp":"2024-03-26T04:09:09.575836+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4884315379,"num_objects":74748,"num_object_clones":0,"num_object_copies":224244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31243,"num_objects_unfound":0,"num_objects_dirty":74748,"num_whiteouts":0,"num_read":1901,"num_read_kb":1638,"num_write":75750,"num_write_kb":4769635,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6353,"num_bytes_recovered":376897191,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43071,"ondisk_log_size":43071,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":15741176,"kb_used_data":12237328,"kb_used_omap":11,"kb_used_meta":3503796,"kb_avail":359239432,"statfs":{"total":383980142592,"available":367861178368,"internally_reserved":0,"allocated":12531023872,"data_stored":12526211462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11660,"internal_metadata":3587887732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,5,6,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":374145024,"num_objects":5709,"num_object_clones":0,"num_object_copies":17127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-996,"num_objects_unfound":0,"num_objects_dirty":5709,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5709,"num_write_kb":365376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3628,"num_bytes_recovered":237764608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.665537"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":844,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929831+0000","last_change":"2024-03-26T04:08:56.865789+0000","last_active":"2024-03-26T04:09:01.929831+0000","last_peered":"2024-03-26T04:09:01.929831+0000","last_clean":"2024-03-26T04:09:01.929831+0000","last_became_active":"2024-03-26T04:08:56.862531+0000","last_became_peered":"2024-03-26T04:08:56.862531+0000","last_unstale":"2024-03-26T04:09:01.929831+0000","last_undegraded":"2024-03-26T04:09:01.929831+0000","last_fullsized":"2024-03-26T04:09:01.929831+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:03.632384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:03.632384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407440+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":856,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616086+0000","last_change":"2024-03-26T04:08:56.863479+0000","last_active":"2024-03-26T04:09:05.616086+0000","last_peered":"2024-03-26T04:09:05.616086+0000","last_clean":"2024-03-26T04:09:05.616086+0000","last_became_active":"2024-03-26T04:08:56.863125+0000","last_became_peered":"2024-03-26T04:08:56.863125+0000","last_unstale":"2024-03-26T04:09:05.616086+0000","last_undegraded":"2024-03-26T04:09:05.616086+0000","last_fullsized":"2024-03-26T04:09:05.616086+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:55.521958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:07:55.521958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:17.041476+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":862,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620793+0000","last_change":"2024-03-26T04:05:29.539024+0000","last_active":"2024-03-26T04:09:05.620793+0000","last_peered":"2024-03-26T04:09:05.620793+0000","last_clean":"2024-03-26T04:09:05.620793+0000","last_became_active":"2024-03-26T04:05:29.538364+0000","last_became_peered":"2024-03-26T04:05:29.538364+0000","last_unstale":"2024-03-26T04:09:05.620793+0000","last_undegraded":"2024-03-26T04:09:05.620793+0000","last_fullsized":"2024-03-26T04:09:05.620793+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:52.117692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:52.117692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"448'27","reported_seq":958,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929479+0000","last_change":"2024-03-26T04:08:31.314138+0000","last_active":"2024-03-26T04:09:01.929479+0000","last_peered":"2024-03-26T04:09:01.929479+0000","last_clean":"2024-03-26T04:09:01.929479+0000","last_became_active":"2024-03-26T04:05:29.520084+0000","last_became_peered":"2024-03-26T04:05:29.520084+0000","last_unstale":"2024-03-26T04:09:01.929479+0000","last_undegraded":"2024-03-26T04:09:01.929479+0000","last_fullsized":"2024-03-26T04:09:01.929479+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:08:31.313993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.313993+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:56.054763+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":228,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616050+0000","last_change":"2024-03-26T04:08:56.869793+0000","last_active":"2024-03-26T04:09:05.616050+0000","last_peered":"2024-03-26T04:09:05.616050+0000","last_clean":"2024-03-26T04:09:05.616050+0000","last_became_active":"2024-03-26T04:08:56.869461+0000","last_became_peered":"2024-03-26T04:08:56.869461+0000","last_unstale":"2024-03-26T04:09:05.616050+0000","last_undegraded":"2024-03-26T04:09:05.616050+0000","last_fullsized":"2024-03-26T04:09:05.616050+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_clean_scrub_stamp":"2024-03-26T04:08:35.321382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:38.350690+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929307+0000","last_change":"2024-03-26T04:08:36.329964+0000","last_active":"2024-03-26T04:09:01.929307+0000","last_peered":"2024-03-26T04:09:01.929307+0000","last_clean":"2024-03-26T04:09:01.929307+0000","last_became_active":"2024-03-26T04:05:29.519486+0000","last_became_peered":"2024-03-26T04:05:29.519486+0000","last_unstale":"2024-03-26T04:09:01.929307+0000","last_undegraded":"2024-03-26T04:09:01.929307+0000","last_fullsized":"2024-03-26T04:09:01.929307+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:36.329835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:36.329835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:44.714906+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":407,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622306+0000","last_change":"2024-03-26T04:08:56.855912+0000","last_active":"2024-03-26T04:09:05.622306+0000","last_peered":"2024-03-26T04:09:05.622306+0000","last_clean":"2024-03-26T04:09:05.622306+0000","last_became_active":"2024-03-26T04:08:56.854666+0000","last_became_peered":"2024-03-26T04:08:56.854666+0000","last_unstale":"2024-03-26T04:09:05.622306+0000","last_undegraded":"2024-03-26T04:09:05.622306+0000","last_fullsized":"2024-03-26T04:09:05.622306+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"448'63","reported_seq":330,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621689+0000","last_change":"2024-03-26T04:07:44.566663+0000","last_active":"2024-03-26T04:09:05.621689+0000","last_peered":"2024-03-26T04:09:05.621689+0000","last_clean":"2024-03-26T04:09:05.621689+0000","last_became_active":"2024-03-26T04:05:29.533552+0000","last_became_peered":"2024-03-26T04:05:29.533552+0000","last_unstale":"2024-03-26T04:09:05.621689+0000","last_undegraded":"2024-03-26T04:09:05.621689+0000","last_fullsized":"2024-03-26T04:09:05.621689+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:07:44.566551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:44.566551+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:09.048408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":831,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618869+0000","last_change":"2024-03-26T04:08:56.863386+0000","last_active":"2024-03-26T04:09:05.618869+0000","last_peered":"2024-03-26T04:09:05.618869+0000","last_clean":"2024-03-26T04:09:05.618869+0000","last_became_active":"2024-03-26T04:08:56.863103+0000","last_became_peered":"2024-03-26T04:08:56.863103+0000","last_unstale":"2024-03-26T04:09:05.618869+0000","last_undegraded":"2024-03-26T04:09:05.618869+0000","last_fullsized":"2024-03-26T04:09:05.618869+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:21.537431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:21.537431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":592,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619373+0000","last_change":"2024-03-26T04:08:56.865865+0000","last_active":"2024-03-26T04:09:05.619373+0000","last_peered":"2024-03-26T04:09:05.619373+0000","last_clean":"2024-03-26T04:09:05.619373+0000","last_became_active":"2024-03-26T04:08:56.865530+0000","last_became_peered":"2024-03-26T04:08:56.865530+0000","last_unstale":"2024-03-26T04:09:05.619373+0000","last_undegraded":"2024-03-26T04:09:05.619373+0000","last_fullsized":"2024-03-26T04:09:05.619373+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:53.342172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.342172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":545,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930760+0000","last_change":"2024-03-26T04:08:56.866879+0000","last_active":"2024-03-26T04:09:01.930760+0000","last_peered":"2024-03-26T04:09:01.930760+0000","last_clean":"2024-03-26T04:09:01.930760+0000","last_became_active":"2024-03-26T04:08:56.865538+0000","last_became_peered":"2024-03-26T04:08:56.865538+0000","last_unstale":"2024-03-26T04:09:01.930760+0000","last_undegraded":"2024-03-26T04:09:01.930760+0000","last_fullsized":"2024-03-26T04:09:01.930760+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:55.527836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.527836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407468+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"448'34","reported_seq":959,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930074+0000","last_change":"2024-03-26T04:07:26.622146+0000","last_active":"2024-03-26T04:09:01.930074+0000","last_peered":"2024-03-26T04:09:01.930074+0000","last_clean":"2024-03-26T04:09:01.930074+0000","last_became_active":"2024-03-26T04:05:29.521715+0000","last_became_peered":"2024-03-26T04:05:29.521715+0000","last_unstale":"2024-03-26T04:09:01.930074+0000","last_undegraded":"2024-03-26T04:09:01.930074+0000","last_fullsized":"2024-03-26T04:09:01.930074+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T04:07:26.622024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.622024+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403745+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":644,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615991+0000","last_change":"2024-03-26T04:07:38.459491+0000","last_active":"2024-03-26T04:09:05.615991+0000","last_peered":"2024-03-26T04:09:05.615991+0000","last_clean":"2024-03-26T04:09:05.615991+0000","last_became_active":"2024-03-26T04:05:29.532847+0000","last_became_peered":"2024-03-26T04:05:29.532847+0000","last_unstale":"2024-03-26T04:09:05.615991+0000","last_undegraded":"2024-03-26T04:09:05.615991+0000","last_fullsized":"2024-03-26T04:09:05.615991+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.459399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.459399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":20,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930478+0000","last_change":"2024-03-26T04:08:56.866656+0000","last_active":"2024-03-26T04:09:01.930478+0000","last_peered":"2024-03-26T04:09:01.930478+0000","last_clean":"2024-03-26T04:09:01.930478+0000","last_became_active":"2024-03-26T04:08:56.862969+0000","last_became_peered":"2024-03-26T04:08:56.862969+0000","last_unstale":"2024-03-26T04:09:01.930478+0000","last_undegraded":"2024-03-26T04:09:01.930478+0000","last_fullsized":"2024-03-26T04:09:01.930478+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.766403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.766403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":19,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929795+0000","last_change":"2024-03-26T04:08:56.863129+0000","last_active":"2024-03-26T04:09:01.929795+0000","last_peered":"2024-03-26T04:09:01.929795+0000","last_clean":"2024-03-26T04:09:01.929795+0000","last_became_active":"2024-03-26T04:08:56.862318+0000","last_became_peered":"2024-03-26T04:08:56.862318+0000","last_unstale":"2024-03-26T04:09:01.929795+0000","last_undegraded":"2024-03-26T04:09:01.929795+0000","last_fullsized":"2024-03-26T04:09:01.929795+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:01.292182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:01.292182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"448'18","reported_seq":913,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930227+0000","last_change":"2024-03-26T04:08:33.351243+0000","last_active":"2024-03-26T04:09:01.930227+0000","last_peered":"2024-03-26T04:09:01.930227+0000","last_clean":"2024-03-26T04:09:01.930227+0000","last_became_active":"2024-03-26T04:05:29.519855+0000","last_became_peered":"2024-03-26T04:05:29.519855+0000","last_unstale":"2024-03-26T04:09:01.930227+0000","last_undegraded":"2024-03-26T04:09:01.930227+0000","last_fullsized":"2024-03-26T04:09:01.930227+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_deep_scrub":"241'16","last_deep_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_clean_scrub_stamp":"2024-03-26T04:08:33.351097+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:10:00.792941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":870,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931085+0000","last_change":"2024-03-26T04:08:56.847258+0000","last_active":"2024-03-26T04:09:01.931085+0000","last_peered":"2024-03-26T04:09:01.931085+0000","last_clean":"2024-03-26T04:09:01.931085+0000","last_became_active":"2024-03-26T04:08:56.845524+0000","last_became_peered":"2024-03-26T04:08:56.845524+0000","last_unstale":"2024-03-26T04:09:01.931085+0000","last_undegraded":"2024-03-26T04:09:01.931085+0000","last_fullsized":"2024-03-26T04:09:01.931085+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:02.671615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:02.671615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407498+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":834,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616233+0000","last_change":"2024-03-26T04:04:51.186722+0000","last_active":"2024-03-26T04:09:05.616233+0000","last_peered":"2024-03-26T04:09:05.616233+0000","last_clean":"2024-03-26T04:09:05.616233+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:09:05.616233+0000","last_undegraded":"2024-03-26T04:09:05.616233+0000","last_fullsized":"2024-03-26T04:09:05.616233+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.186605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.186605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":851,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619647+0000","last_change":"2024-03-26T04:08:56.866535+0000","last_active":"2024-03-26T04:09:05.619647+0000","last_peered":"2024-03-26T04:09:05.619647+0000","last_clean":"2024-03-26T04:09:05.619647+0000","last_became_active":"2024-03-26T04:08:56.866193+0000","last_became_peered":"2024-03-26T04:08:56.866193+0000","last_unstale":"2024-03-26T04:09:05.619647+0000","last_undegraded":"2024-03-26T04:09:05.619647+0000","last_fullsized":"2024-03-26T04:09:05.619647+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:54.321548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:54.321548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"448'39","reported_seq":885,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622790+0000","last_change":"2024-03-26T04:05:29.533402+0000","last_active":"2024-03-26T04:09:05.622790+0000","last_peered":"2024-03-26T04:09:05.622790+0000","last_clean":"2024-03-26T04:09:05.622790+0000","last_became_active":"2024-03-26T04:05:29.525450+0000","last_became_peered":"2024-03-26T04:05:29.525450+0000","last_unstale":"2024-03-26T04:09:05.622790+0000","last_undegraded":"2024-03-26T04:09:05.622790+0000","last_fullsized":"2024-03-26T04:09:05.622790+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:02:45.014541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:45.014541+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":474,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616721+0000","last_change":"2024-03-26T04:08:24.139810+0000","last_active":"2024-03-26T04:09:05.616721+0000","last_peered":"2024-03-26T04:09:05.616721+0000","last_clean":"2024-03-26T04:09:05.616721+0000","last_became_active":"2024-03-26T04:05:29.533287+0000","last_became_peered":"2024-03-26T04:05:29.533287+0000","last_unstale":"2024-03-26T04:09:05.616721+0000","last_undegraded":"2024-03-26T04:09:05.616721+0000","last_fullsized":"2024-03-26T04:09:05.616721+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:24.139706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.139706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:35.971431+0000","scrub_duration":360,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":860,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930143+0000","last_change":"2024-03-26T04:08:56.863537+0000","last_active":"2024-03-26T04:09:01.930143+0000","last_peered":"2024-03-26T04:09:01.930143+0000","last_clean":"2024-03-26T04:09:01.930143+0000","last_became_active":"2024-03-26T04:08:56.862795+0000","last_became_peered":"2024-03-26T04:08:56.862795+0000","last_unstale":"2024-03-26T04:09:01.930143+0000","last_undegraded":"2024-03-26T04:09:01.930143+0000","last_fullsized":"2024-03-26T04:09:01.930143+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.463158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.463158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":844,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930235+0000","last_change":"2024-03-26T04:07:24.624918+0000","last_active":"2024-03-26T04:09:01.930235+0000","last_peered":"2024-03-26T04:09:01.930235+0000","last_clean":"2024-03-26T04:09:01.930235+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:09:01.930235+0000","last_undegraded":"2024-03-26T04:09:01.930235+0000","last_fullsized":"2024-03-26T04:09:01.930235+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.624793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.624793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403761+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"448'30","reported_seq":399,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.614973+0000","last_change":"2024-03-26T04:07:28.446713+0000","last_active":"2024-03-26T04:09:05.614973+0000","last_peered":"2024-03-26T04:09:05.614973+0000","last_clean":"2024-03-26T04:09:05.614973+0000","last_became_active":"2024-03-26T04:05:29.521493+0000","last_became_peered":"2024-03-26T04:05:29.521493+0000","last_unstale":"2024-03-26T04:09:05.614973+0000","last_undegraded":"2024-03-26T04:09:05.614973+0000","last_fullsized":"2024-03-26T04:09:05.614973+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:07:28.446562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:28.446562+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":315,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622524+0000","last_change":"2024-03-26T04:08:56.880502+0000","last_active":"2024-03-26T04:09:05.622524+0000","last_peered":"2024-03-26T04:09:05.622524+0000","last_clean":"2024-03-26T04:09:05.622524+0000","last_became_active":"2024-03-26T04:08:56.861772+0000","last_became_peered":"2024-03-26T04:08:56.861772+0000","last_unstale":"2024-03-26T04:09:05.622524+0000","last_undegraded":"2024-03-26T04:09:05.622524+0000","last_fullsized":"2024-03-26T04:09:05.622524+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:02:40.908630+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:02:40.908630+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"450'12","reported_seq":1043,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931132+0000","last_change":"2024-03-26T04:08:56.893670+0000","last_active":"2024-03-26T04:09:01.931132+0000","last_peered":"2024-03-26T04:09:01.931132+0000","last_clean":"2024-03-26T04:09:01.931132+0000","last_became_active":"2024-03-26T04:08:56.862824+0000","last_became_peered":"2024-03-26T04:08:56.862824+0000","last_unstale":"2024-03-26T04:09:01.931132+0000","last_undegraded":"2024-03-26T04:09:01.931132+0000","last_fullsized":"2024-03-26T04:09:01.931132+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'11","last_scrub_stamp":"2024-03-26T04:08:27.362944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.362944+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:54.362705+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":884,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615737+0000","last_change":"2024-03-26T04:08:56.885318+0000","last_active":"2024-03-26T04:09:05.615737+0000","last_peered":"2024-03-26T04:09:05.615737+0000","last_clean":"2024-03-26T04:09:05.615737+0000","last_became_active":"2024-03-26T04:08:56.861744+0000","last_became_peered":"2024-03-26T04:08:56.861744+0000","last_unstale":"2024-03-26T04:09:05.615737+0000","last_undegraded":"2024-03-26T04:09:05.615737+0000","last_fullsized":"2024-03-26T04:09:05.615737+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:04:18.067462+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.067462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"448'42","reported_seq":978,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621418+0000","last_change":"2024-03-26T04:08:56.917670+0000","last_active":"2024-03-26T04:09:05.621418+0000","last_peered":"2024-03-26T04:09:05.621418+0000","last_clean":"2024-03-26T04:09:05.621418+0000","last_became_active":"2024-03-26T04:08:56.864036+0000","last_became_peered":"2024-03-26T04:08:56.864036+0000","last_unstale":"2024-03-26T04:09:05.621418+0000","last_undegraded":"2024-03-26T04:09:05.621418+0000","last_fullsized":"2024-03-26T04:09:05.621418+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'32","last_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:04:05.344849+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":846,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.614877+0000","last_change":"2024-03-26T04:07:47.538830+0000","last_active":"2024-03-26T04:09:05.614877+0000","last_peered":"2024-03-26T04:09:05.614877+0000","last_clean":"2024-03-26T04:09:05.614877+0000","last_became_active":"2024-03-26T04:05:29.520352+0000","last_became_peered":"2024-03-26T04:05:29.520352+0000","last_unstale":"2024-03-26T04:09:05.614877+0000","last_undegraded":"2024-03-26T04:09:05.614877+0000","last_fullsized":"2024-03-26T04:09:05.614877+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:47.538728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:47.538728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":846,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622077+0000","last_change":"2024-03-26T04:08:56.864071+0000","last_active":"2024-03-26T04:09:05.622077+0000","last_peered":"2024-03-26T04:09:05.622077+0000","last_clean":"2024-03-26T04:09:05.622077+0000","last_became_active":"2024-03-26T04:08:56.863159+0000","last_became_peered":"2024-03-26T04:08:56.863159+0000","last_unstale":"2024-03-26T04:09:05.622077+0000","last_undegraded":"2024-03-26T04:09:05.622077+0000","last_fullsized":"2024-03-26T04:09:05.622077+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:29.533765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.533765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":431,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618780+0000","last_change":"2024-03-26T04:07:31.484578+0000","last_active":"2024-03-26T04:09:05.618780+0000","last_peered":"2024-03-26T04:09:05.618780+0000","last_clean":"2024-03-26T04:09:05.618780+0000","last_became_active":"2024-03-26T04:05:29.540582+0000","last_became_peered":"2024-03-26T04:05:29.540582+0000","last_unstale":"2024-03-26T04:09:05.618780+0000","last_undegraded":"2024-03-26T04:09:05.618780+0000","last_fullsized":"2024-03-26T04:09:05.618780+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:31.484355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.484355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"448'35","reported_seq":35,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621520+0000","last_change":"2024-03-26T04:08:57.325236+0000","last_active":"2024-03-26T04:09:05.621520+0000","last_peered":"2024-03-26T04:09:05.621520+0000","last_clean":"2024-03-26T04:09:05.621520+0000","last_became_active":"2024-03-26T04:08:56.871545+0000","last_became_peered":"2024-03-26T04:08:56.871545+0000","last_unstale":"2024-03-26T04:09:05.621520+0000","last_undegraded":"2024-03-26T04:09:05.621520+0000","last_fullsized":"2024-03-26T04:09:05.621520+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":700,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619731+0000","last_change":"2024-03-26T04:08:56.860375+0000","last_active":"2024-03-26T04:09:05.619731+0000","last_peered":"2024-03-26T04:09:05.619731+0000","last_clean":"2024-03-26T04:09:05.619731+0000","last_became_active":"2024-03-26T04:08:56.859656+0000","last_became_peered":"2024-03-26T04:08:56.859656+0000","last_unstale":"2024-03-26T04:09:05.619731+0000","last_undegraded":"2024-03-26T04:09:05.619731+0000","last_fullsized":"2024-03-26T04:09:05.619731+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.011134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":771,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929547+0000","last_change":"2024-03-26T04:08:56.866943+0000","last_active":"2024-03-26T04:09:01.929547+0000","last_peered":"2024-03-26T04:09:01.929547+0000","last_clean":"2024-03-26T04:09:01.929547+0000","last_became_active":"2024-03-26T04:08:56.866332+0000","last_became_peered":"2024-03-26T04:08:56.866332+0000","last_unstale":"2024-03-26T04:09:01.929547+0000","last_undegraded":"2024-03-26T04:09:01.929547+0000","last_fullsized":"2024-03-26T04:09:01.929547+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:17.385765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.385765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407244+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":879,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621191+0000","last_change":"2024-03-26T04:07:37.490184+0000","last_active":"2024-03-26T04:09:05.621191+0000","last_peered":"2024-03-26T04:09:05.621191+0000","last_clean":"2024-03-26T04:09:05.621191+0000","last_became_active":"2024-03-26T04:05:29.538915+0000","last_became_peered":"2024-03-26T04:05:29.538915+0000","last_unstale":"2024-03-26T04:09:05.621191+0000","last_undegraded":"2024-03-26T04:09:05.621191+0000","last_fullsized":"2024-03-26T04:09:05.621191+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.490105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.490105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"448'14","reported_seq":380,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.612604+0000","last_change":"2024-03-26T04:08:17.636274+0000","last_active":"2024-03-26T04:09:05.612604+0000","last_peered":"2024-03-26T04:09:05.612604+0000","last_clean":"2024-03-26T04:09:05.612604+0000","last_became_active":"2024-03-26T04:05:29.520584+0000","last_became_peered":"2024-03-26T04:05:29.520584+0000","last_unstale":"2024-03-26T04:09:05.612604+0000","last_undegraded":"2024-03-26T04:09:05.612604+0000","last_fullsized":"2024-03-26T04:09:05.612604+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:08:17.636015+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:08:17.636015+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:26.765567+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":497,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.617746+0000","last_change":"2024-03-26T04:08:19.648443+0000","last_active":"2024-03-26T04:09:05.617746+0000","last_peered":"2024-03-26T04:09:05.617746+0000","last_clean":"2024-03-26T04:09:05.617746+0000","last_became_active":"2024-03-26T04:05:29.536699+0000","last_became_peered":"2024-03-26T04:05:29.536699+0000","last_unstale":"2024-03-26T04:09:05.617746+0000","last_undegraded":"2024-03-26T04:09:05.617746+0000","last_fullsized":"2024-03-26T04:09:05.617746+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:19.648229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:08:19.648229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:20.571200+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":90,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929662+0000","last_change":"2024-03-26T04:08:56.845579+0000","last_active":"2024-03-26T04:09:01.929662+0000","last_peered":"2024-03-26T04:09:01.929662+0000","last_clean":"2024-03-26T04:09:01.929662+0000","last_became_active":"2024-03-26T04:08:56.845214+0000","last_became_peered":"2024-03-26T04:08:56.845214+0000","last_unstale":"2024-03-26T04:09:01.929662+0000","last_undegraded":"2024-03-26T04:09:01.929662+0000","last_fullsized":"2024-03-26T04:09:01.929662+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:27.148413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.148413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:55.239665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":589,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615765+0000","last_change":"2024-03-26T04:08:56.869009+0000","last_active":"2024-03-26T04:09:05.615765+0000","last_peered":"2024-03-26T04:09:05.615765+0000","last_clean":"2024-03-26T04:09:05.615765+0000","last_became_active":"2024-03-26T04:08:56.868654+0000","last_became_peered":"2024-03-26T04:08:56.868654+0000","last_unstale":"2024-03-26T04:09:05.615765+0000","last_undegraded":"2024-03-26T04:09:05.615765+0000","last_fullsized":"2024-03-26T04:09:05.615765+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:39.423550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.423550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"448'39","reported_seq":547,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616595+0000","last_change":"2024-03-26T04:08:56.882927+0000","last_active":"2024-03-26T04:09:05.616595+0000","last_peered":"2024-03-26T04:09:05.616595+0000","last_clean":"2024-03-26T04:09:05.616595+0000","last_became_active":"2024-03-26T04:08:56.839608+0000","last_became_peered":"2024-03-26T04:08:56.839608+0000","last_unstale":"2024-03-26T04:09:05.616595+0000","last_undegraded":"2024-03-26T04:09:05.616595+0000","last_fullsized":"2024-03-26T04:09:05.616595+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:08:24.491666+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.491666+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:39.827265+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":846,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616310+0000","last_change":"2024-03-26T04:03:07.149962+0000","last_active":"2024-03-26T04:09:05.616310+0000","last_peered":"2024-03-26T04:09:05.616310+0000","last_clean":"2024-03-26T04:09:05.616310+0000","last_became_active":"2024-03-26T04:02:52.506113+0000","last_became_peered":"2024-03-26T04:02:52.506113+0000","last_unstale":"2024-03-26T04:09:05.616310+0000","last_undegraded":"2024-03-26T04:09:05.616310+0000","last_fullsized":"2024-03-26T04:09:05.616310+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:07.149881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.149881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":431,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620208+0000","last_change":"2024-03-26T04:04:51.740064+0000","last_active":"2024-03-26T04:09:05.620208+0000","last_peered":"2024-03-26T04:09:05.620208+0000","last_clean":"2024-03-26T04:09:05.620208+0000","last_became_active":"2024-03-26T04:02:52.489626+0000","last_became_peered":"2024-03-26T04:02:52.489626+0000","last_unstale":"2024-03-26T04:09:05.620208+0000","last_undegraded":"2024-03-26T04:09:05.620208+0000","last_fullsized":"2024-03-26T04:09:05.620208+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.739925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.739925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":609,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930655+0000","last_change":"2024-03-26T04:08:56.847038+0000","last_active":"2024-03-26T04:09:01.930655+0000","last_peered":"2024-03-26T04:09:01.930655+0000","last_clean":"2024-03-26T04:09:01.930655+0000","last_became_active":"2024-03-26T04:08:56.846241+0000","last_became_peered":"2024-03-26T04:08:56.846241+0000","last_unstale":"2024-03-26T04:09:01.930655+0000","last_undegraded":"2024-03-26T04:09:01.930655+0000","last_fullsized":"2024-03-26T04:09:01.930655+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.951520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.951520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407733+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"448'25","reported_seq":909,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615854+0000","last_change":"2024-03-26T04:08:27.876373+0000","last_active":"2024-03-26T04:09:05.615854+0000","last_peered":"2024-03-26T04:09:05.615854+0000","last_clean":"2024-03-26T04:09:05.615854+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:09:05.615854+0000","last_undegraded":"2024-03-26T04:09:05.615854+0000","last_fullsized":"2024-03-26T04:09:05.615854+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:08:27.876252+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.876252+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:47.452492+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":475,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618657+0000","last_change":"2024-03-26T04:08:27.562884+0000","last_active":"2024-03-26T04:09:05.618657+0000","last_peered":"2024-03-26T04:09:05.618657+0000","last_clean":"2024-03-26T04:09:05.618657+0000","last_became_active":"2024-03-26T04:05:29.537974+0000","last_became_peered":"2024-03-26T04:05:29.537974+0000","last_unstale":"2024-03-26T04:09:05.618657+0000","last_undegraded":"2024-03-26T04:09:05.618657+0000","last_fullsized":"2024-03-26T04:09:05.618657+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:27.562766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.562766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:27.598458+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":865,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615818+0000","last_change":"2024-03-26T04:08:56.867471+0000","last_active":"2024-03-26T04:09:05.615818+0000","last_peered":"2024-03-26T04:09:05.615818+0000","last_clean":"2024-03-26T04:09:05.615818+0000","last_became_active":"2024-03-26T04:08:56.865835+0000","last_became_peered":"2024-03-26T04:08:56.865835+0000","last_unstale":"2024-03-26T04:09:05.615818+0000","last_undegraded":"2024-03-26T04:09:05.615818+0000","last_fullsized":"2024-03-26T04:09:05.615818+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:57.362566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:57.362566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":856,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929955+0000","last_change":"2024-03-26T04:08:56.845321+0000","last_active":"2024-03-26T04:09:01.929955+0000","last_peered":"2024-03-26T04:09:01.929955+0000","last_clean":"2024-03-26T04:09:01.929955+0000","last_became_active":"2024-03-26T04:08:56.844954+0000","last_became_peered":"2024-03-26T04:08:56.844954+0000","last_unstale":"2024-03-26T04:09:01.929955+0000","last_undegraded":"2024-03-26T04:09:01.929955+0000","last_fullsized":"2024-03-26T04:09:01.929955+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:01:35.104690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"448'27","reported_seq":920,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930337+0000","last_change":"2024-03-26T04:05:29.521309+0000","last_active":"2024-03-26T04:09:01.930337+0000","last_peered":"2024-03-26T04:09:01.930337+0000","last_clean":"2024-03-26T04:09:01.930337+0000","last_became_active":"2024-03-26T04:05:29.520817+0000","last_became_peered":"2024-03-26T04:05:29.520817+0000","last_unstale":"2024-03-26T04:09:01.930337+0000","last_undegraded":"2024-03-26T04:09:01.930337+0000","last_fullsized":"2024-03-26T04:09:01.930337+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:02:51.525641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:51.525641+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403650+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":849,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621073+0000","last_change":"2024-03-26T04:08:56.864462+0000","last_active":"2024-03-26T04:09:05.621073+0000","last_peered":"2024-03-26T04:09:05.621073+0000","last_clean":"2024-03-26T04:09:05.621073+0000","last_became_active":"2024-03-26T04:08:56.863736+0000","last_became_peered":"2024-03-26T04:08:56.863736+0000","last_unstale":"2024-03-26T04:09:05.621073+0000","last_undegraded":"2024-03-26T04:09:05.621073+0000","last_fullsized":"2024-03-26T04:09:05.621073+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:04:58.704813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":510,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616132+0000","last_change":"2024-03-26T04:03:53.367019+0000","last_active":"2024-03-26T04:09:05.616132+0000","last_peered":"2024-03-26T04:09:05.616132+0000","last_clean":"2024-03-26T04:09:05.616132+0000","last_became_active":"2024-03-26T04:02:52.511718+0000","last_became_peered":"2024-03-26T04:02:52.511718+0000","last_unstale":"2024-03-26T04:09:05.616132+0000","last_undegraded":"2024-03-26T04:09:05.616132+0000","last_fullsized":"2024-03-26T04:09:05.616132+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:53.366906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.366906+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":710,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930307+0000","last_change":"2024-03-26T04:08:56.846589+0000","last_active":"2024-03-26T04:09:01.930307+0000","last_peered":"2024-03-26T04:09:01.930307+0000","last_clean":"2024-03-26T04:09:01.930307+0000","last_became_active":"2024-03-26T04:08:56.845018+0000","last_became_peered":"2024-03-26T04:08:56.845018+0000","last_unstale":"2024-03-26T04:09:01.930307+0000","last_undegraded":"2024-03-26T04:09:01.930307+0000","last_fullsized":"2024-03-26T04:09:01.930307+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:32.845486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:32.845486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407674+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"448'49","reported_seq":444,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.614859+0000","last_change":"2024-03-26T04:08:56.918497+0000","last_active":"2024-03-26T04:09:05.614859+0000","last_peered":"2024-03-26T04:09:05.614859+0000","last_clean":"2024-03-26T04:09:05.614859+0000","last_became_active":"2024-03-26T04:08:56.871518+0000","last_became_peered":"2024-03-26T04:08:56.871518+0000","last_unstale":"2024-03-26T04:09:05.614859+0000","last_undegraded":"2024-03-26T04:09:05.614859+0000","last_fullsized":"2024-03-26T04:09:05.614859+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'39","last_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.462212+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":847,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931181+0000","last_change":"2024-03-26T04:08:29.657075+0000","last_active":"2024-03-26T04:09:01.931181+0000","last_peered":"2024-03-26T04:09:01.931181+0000","last_clean":"2024-03-26T04:09:01.931181+0000","last_became_active":"2024-03-26T04:05:29.521886+0000","last_became_peered":"2024-03-26T04:05:29.521886+0000","last_unstale":"2024-03-26T04:09:01.931181+0000","last_undegraded":"2024-03-26T04:09:01.931181+0000","last_fullsized":"2024-03-26T04:09:01.931181+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:29.656935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:29.656935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:37.252344+0000","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"450'15","reported_seq":293,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622689+0000","last_change":"2024-03-26T04:08:56.881240+0000","last_active":"2024-03-26T04:09:05.622689+0000","last_peered":"2024-03-26T04:09:05.622689+0000","last_clean":"2024-03-26T04:09:05.622689+0000","last_became_active":"2024-03-26T04:08:56.862068+0000","last_became_peered":"2024-03-26T04:08:56.862068+0000","last_unstale":"2024-03-26T04:09:05.622689+0000","last_undegraded":"2024-03-26T04:09:05.622689+0000","last_fullsized":"2024-03-26T04:09:05.622689+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'12","last_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_deep_scrub":"334'12","last_deep_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.907441+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":899,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618084+0000","last_change":"2024-03-26T04:08:56.907028+0000","last_active":"2024-03-26T04:09:05.618084+0000","last_peered":"2024-03-26T04:09:05.618084+0000","last_clean":"2024-03-26T04:09:05.618084+0000","last_became_active":"2024-03-26T04:08:56.863922+0000","last_became_peered":"2024-03-26T04:08:56.863922+0000","last_unstale":"2024-03-26T04:09:05.618084+0000","last_undegraded":"2024-03-26T04:09:05.618084+0000","last_fullsized":"2024-03-26T04:09:05.618084+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.538770+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"448'32","reported_seq":842,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.625927+0000","last_change":"2024-03-26T04:08:56.921574+0000","last_active":"2024-03-26T04:09:05.625927+0000","last_peered":"2024-03-26T04:09:05.625927+0000","last_clean":"2024-03-26T04:09:05.625927+0000","last_became_active":"2024-03-26T04:08:56.864919+0000","last_became_peered":"2024-03-26T04:08:56.864919+0000","last_unstale":"2024-03-26T04:09:05.625927+0000","last_undegraded":"2024-03-26T04:09:05.625927+0000","last_fullsized":"2024-03-26T04:09:05.625927+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:08:33.108111+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:08:33.108111+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:50.756329+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":772,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622251+0000","last_change":"2024-03-26T04:08:56.848343+0000","last_active":"2024-03-26T04:09:05.622251+0000","last_peered":"2024-03-26T04:09:05.622251+0000","last_clean":"2024-03-26T04:09:05.622251+0000","last_became_active":"2024-03-26T04:08:56.833658+0000","last_became_peered":"2024-03-26T04:08:56.833658+0000","last_unstale":"2024-03-26T04:09:05.622251+0000","last_undegraded":"2024-03-26T04:09:05.622251+0000","last_fullsized":"2024-03-26T04:09:05.622251+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:08:21.812923+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:08:21.812923+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:36.120634+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":427,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616816+0000","last_change":"2024-03-26T04:08:56.871548+0000","last_active":"2024-03-26T04:09:05.616816+0000","last_peered":"2024-03-26T04:09:05.616816+0000","last_clean":"2024-03-26T04:09:05.616816+0000","last_became_active":"2024-03-26T04:08:56.870098+0000","last_became_peered":"2024-03-26T04:08:56.870098+0000","last_unstale":"2024-03-26T04:09:05.616816+0000","last_undegraded":"2024-03-26T04:09:05.616816+0000","last_fullsized":"2024-03-26T04:09:05.616816+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:58.337496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:58.337496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":550,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930157+0000","last_change":"2024-03-26T04:08:56.847634+0000","last_active":"2024-03-26T04:09:01.930157+0000","last_peered":"2024-03-26T04:09:01.930157+0000","last_clean":"2024-03-26T04:09:01.930157+0000","last_became_active":"2024-03-26T04:08:56.845794+0000","last_became_peered":"2024-03-26T04:08:56.845794+0000","last_unstale":"2024-03-26T04:09:01.930157+0000","last_undegraded":"2024-03-26T04:09:01.930157+0000","last_fullsized":"2024-03-26T04:09:01.930157+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:57.522794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:57.522794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407525+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"448'39","reported_seq":986,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621909+0000","last_change":"2024-03-26T04:08:56.904108+0000","last_active":"2024-03-26T04:09:05.621909+0000","last_peered":"2024-03-26T04:09:05.621909+0000","last_clean":"2024-03-26T04:09:05.621909+0000","last_became_active":"2024-03-26T04:08:56.862596+0000","last_became_peered":"2024-03-26T04:08:56.862596+0000","last_unstale":"2024-03-26T04:09:05.621909+0000","last_undegraded":"2024-03-26T04:09:05.621909+0000","last_fullsized":"2024-03-26T04:09:05.621909+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'33","last_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.584130+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:10.128764+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":532,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930478+0000","last_change":"2024-03-26T04:08:56.864440+0000","last_active":"2024-03-26T04:09:01.930478+0000","last_peered":"2024-03-26T04:09:01.930478+0000","last_clean":"2024-03-26T04:09:01.930478+0000","last_became_active":"2024-03-26T04:08:56.862509+0000","last_became_peered":"2024-03-26T04:08:56.862509+0000","last_unstale":"2024-03-26T04:09:01.930478+0000","last_undegraded":"2024-03-26T04:09:01.930478+0000","last_fullsized":"2024-03-26T04:09:01.930478+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:51.008927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:51.008927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:14.376997+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616121+0000","last_change":"2024-03-26T04:08:56.868847+0000","last_active":"2024-03-26T04:09:05.616121+0000","last_peered":"2024-03-26T04:09:05.616121+0000","last_clean":"2024-03-26T04:09:05.616121+0000","last_became_active":"2024-03-26T04:08:56.868262+0000","last_became_peered":"2024-03-26T04:08:56.868262+0000","last_unstale":"2024-03-26T04:09:05.616121+0000","last_undegraded":"2024-03-26T04:09:05.616121+0000","last_fullsized":"2024-03-26T04:09:05.616121+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:04:10.223350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":817,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618690+0000","last_change":"2024-03-26T04:08:37.529848+0000","last_active":"2024-03-26T04:09:05.618690+0000","last_peered":"2024-03-26T04:09:05.618690+0000","last_clean":"2024-03-26T04:09:05.618690+0000","last_became_active":"2024-03-26T04:05:29.540070+0000","last_became_peered":"2024-03-26T04:05:29.540070+0000","last_unstale":"2024-03-26T04:09:05.618690+0000","last_undegraded":"2024-03-26T04:09:05.618690+0000","last_fullsized":"2024-03-26T04:09:05.618690+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:37.529261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:08:37.529261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:10:06.280263+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"448'25","reported_seq":326,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616615+0000","last_change":"2024-03-26T04:08:56.903278+0000","last_active":"2024-03-26T04:09:05.616615+0000","last_peered":"2024-03-26T04:09:05.616615+0000","last_clean":"2024-03-26T04:09:05.616615+0000","last_became_active":"2024-03-26T04:08:56.868889+0000","last_became_peered":"2024-03-26T04:08:56.868889+0000","last_unstale":"2024-03-26T04:09:05.616615+0000","last_undegraded":"2024-03-26T04:09:05.616615+0000","last_fullsized":"2024-03-26T04:09:05.616615+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:07:48.515443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.515443+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:18.491364+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"455'4720","reported_seq":4785,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.137745+0000","last_change":"2024-03-26T04:08:02.634036+0000","last_active":"2024-03-26T04:09:06.137745+0000","last_peered":"2024-03-26T04:09:06.137745+0000","last_clean":"2024-03-26T04:09:06.137745+0000","last_became_active":"2024-03-26T04:08:02.633750+0000","last_became_peered":"2024-03-26T04:08:02.633750+0000","last_unstale":"2024-03-26T04:09:06.137745+0000","last_undegraded":"2024-03-26T04:09:06.137745+0000","last_fullsized":"2024-03-26T04:09:06.137745+0000","mapping_epoch":427,"log_start":"446'2311","ondisk_log_start":"446'2311","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2409,"log_dups_size":2311,"ondisk_log_size":2409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:25.254299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309329920,"num_objects":4720,"num_object_clones":0,"num_object_copies":14160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4720,"num_write_kb":302080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":570,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618262+0000","last_change":"2024-03-26T04:08:36.535719+0000","last_active":"2024-03-26T04:09:05.618262+0000","last_peered":"2024-03-26T04:09:05.618262+0000","last_clean":"2024-03-26T04:09:05.618262+0000","last_became_active":"2024-03-26T04:05:29.537211+0000","last_became_peered":"2024-03-26T04:05:29.537211+0000","last_unstale":"2024-03-26T04:09:05.618262+0000","last_undegraded":"2024-03-26T04:09:05.618262+0000","last_fullsized":"2024-03-26T04:09:05.618262+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:36.535552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:36.535552+0000","last_clean_scrub_stamp":"2024-03-26T04:08:36.535552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:44.295893+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"446'4","reported_seq":883,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930976+0000","last_change":"2024-03-26T04:08:56.908939+0000","last_active":"2024-03-26T04:09:01.930976+0000","last_peered":"2024-03-26T04:09:01.930976+0000","last_clean":"2024-03-26T04:09:01.930976+0000","last_became_active":"2024-03-26T04:08:56.863670+0000","last_became_peered":"2024-03-26T04:08:56.863670+0000","last_unstale":"2024-03-26T04:09:01.930976+0000","last_undegraded":"2024-03-26T04:09:01.930976+0000","last_fullsized":"2024-03-26T04:09:01.930976+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'4","last_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_deep_scrub":"446'4","last_deep_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.098553+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:34.930654+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"448'51","reported_seq":890,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929362+0000","last_change":"2024-03-26T04:08:56.897366+0000","last_active":"2024-03-26T04:09:01.929362+0000","last_peered":"2024-03-26T04:09:01.929362+0000","last_clean":"2024-03-26T04:09:01.929362+0000","last_became_active":"2024-03-26T04:08:56.862064+0000","last_became_peered":"2024-03-26T04:08:56.862064+0000","last_unstale":"2024-03-26T04:09:01.929362+0000","last_undegraded":"2024-03-26T04:09:01.929362+0000","last_fullsized":"2024-03-26T04:09:01.929362+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:08:37.152100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:37.152100+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:45.328243+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"450'10","reported_seq":52,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929490+0000","last_change":"2024-03-26T04:08:56.918039+0000","last_active":"2024-03-26T04:09:01.929490+0000","last_peered":"2024-03-26T04:09:01.929490+0000","last_clean":"2024-03-26T04:09:01.929490+0000","last_became_active":"2024-03-26T04:08:56.864436+0000","last_became_peered":"2024-03-26T04:08:56.864436+0000","last_unstale":"2024-03-26T04:09:01.929490+0000","last_undegraded":"2024-03-26T04:09:01.929490+0000","last_fullsized":"2024-03-26T04:09:01.929490+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:07:54.941219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.941219+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:11.167113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"455'4596","reported_seq":4662,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.359526+0000","last_change":"2024-03-26T04:08:02.632658+0000","last_active":"2024-03-26T04:09:06.359526+0000","last_peered":"2024-03-26T04:09:06.359526+0000","last_clean":"2024-03-26T04:09:06.359526+0000","last_became_active":"2024-03-26T04:08:02.631983+0000","last_became_peered":"2024-03-26T04:08:02.631983+0000","last_unstale":"2024-03-26T04:09:06.359526+0000","last_undegraded":"2024-03-26T04:09:06.359526+0000","last_fullsized":"2024-03-26T04:09:06.359526+0000","mapping_epoch":427,"log_start":"445'1990","ondisk_log_start":"445'1990","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2606,"log_dups_size":857,"ondisk_log_size":2606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:10.773166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301203456,"num_objects":4596,"num_object_clones":0,"num_object_copies":13788,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4596,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4596,"num_write_kb":294144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":163,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616150+0000","last_change":"2024-03-26T04:08:56.863588+0000","last_active":"2024-03-26T04:09:05.616150+0000","last_peered":"2024-03-26T04:09:05.616150+0000","last_clean":"2024-03-26T04:09:05.616150+0000","last_became_active":"2024-03-26T04:08:56.863164+0000","last_became_peered":"2024-03-26T04:08:56.863164+0000","last_unstale":"2024-03-26T04:09:05.616150+0000","last_undegraded":"2024-03-26T04:09:05.616150+0000","last_fullsized":"2024-03-26T04:09:05.616150+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:20.512660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:08:20.512660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:24.241422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930685+0000","last_change":"2024-03-26T04:08:56.867356+0000","last_active":"2024-03-26T04:09:01.930685+0000","last_peered":"2024-03-26T04:09:01.930685+0000","last_clean":"2024-03-26T04:09:01.930685+0000","last_became_active":"2024-03-26T04:08:56.865701+0000","last_became_peered":"2024-03-26T04:08:56.865701+0000","last_unstale":"2024-03-26T04:09:01.930685+0000","last_undegraded":"2024-03-26T04:09:01.930685+0000","last_fullsized":"2024-03-26T04:09:01.930685+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_clean_scrub_stamp":"2024-03-26T04:08:34.104519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:10:03.704719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"448'36","reported_seq":963,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930278+0000","last_change":"2024-03-26T04:04:19.443761+0000","last_active":"2024-03-26T04:09:01.930278+0000","last_peered":"2024-03-26T04:09:01.930278+0000","last_clean":"2024-03-26T04:09:01.930278+0000","last_became_active":"2024-03-26T04:02:52.513309+0000","last_became_peered":"2024-03-26T04:02:52.513309+0000","last_unstale":"2024-03-26T04:09:01.930278+0000","last_undegraded":"2024-03-26T04:09:01.930278+0000","last_fullsized":"2024-03-26T04:09:01.930278+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:04:19.443608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.443608+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403671+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":764,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618169+0000","last_change":"2024-03-26T04:07:53.660073+0000","last_active":"2024-03-26T04:09:05.618169+0000","last_peered":"2024-03-26T04:09:05.618169+0000","last_clean":"2024-03-26T04:09:05.618169+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:09:05.618169+0000","last_undegraded":"2024-03-26T04:09:05.618169+0000","last_fullsized":"2024-03-26T04:09:05.618169+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:53.659974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:53.659974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"455'4709","reported_seq":6625,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:06.382760+0000","last_change":"2024-03-26T04:08:57.931058+0000","last_active":"2024-03-26T04:09:06.382760+0000","last_peered":"2024-03-26T04:09:06.382760+0000","last_clean":"2024-03-26T04:08:55.774068+0000","last_became_active":"2024-03-26T04:08:57.907295+0000","last_became_peered":"2024-03-26T04:08:57.907295+0000","last_unstale":"2024-03-26T04:09:06.382760+0000","last_undegraded":"2024-03-26T04:09:06.382760+0000","last_fullsized":"2024-03-26T04:09:06.382760+0000","mapping_epoch":450,"log_start":"446'2122","ondisk_log_start":"446'2122","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":857,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":308609024,"num_objects":4709,"num_object_clones":0,"num_object_copies":14127,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3724,"num_objects_unfound":0,"num_objects_dirty":4709,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4709,"num_write_kb":301376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":60555264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":4709}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":717,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618457+0000","last_change":"2024-03-26T04:08:56.856822+0000","last_active":"2024-03-26T04:09:05.618457+0000","last_peered":"2024-03-26T04:09:05.618457+0000","last_clean":"2024-03-26T04:09:05.618457+0000","last_became_active":"2024-03-26T04:08:56.855035+0000","last_became_peered":"2024-03-26T04:08:56.855035+0000","last_unstale":"2024-03-26T04:09:05.618457+0000","last_undegraded":"2024-03-26T04:09:05.618457+0000","last_fullsized":"2024-03-26T04:09:05.618457+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:37.891864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.891864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":763,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619683+0000","last_change":"2024-03-26T04:08:56.849260+0000","last_active":"2024-03-26T04:09:05.619683+0000","last_peered":"2024-03-26T04:09:05.619683+0000","last_clean":"2024-03-26T04:09:05.619683+0000","last_became_active":"2024-03-26T04:08:56.848815+0000","last_became_peered":"2024-03-26T04:08:56.848815+0000","last_unstale":"2024-03-26T04:09:05.619683+0000","last_undegraded":"2024-03-26T04:09:05.619683+0000","last_fullsized":"2024-03-26T04:09:05.619683+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.965300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":648,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930297+0000","last_change":"2024-03-26T04:08:56.848363+0000","last_active":"2024-03-26T04:09:01.930297+0000","last_peered":"2024-03-26T04:09:01.930297+0000","last_clean":"2024-03-26T04:09:01.930297+0000","last_became_active":"2024-03-26T04:08:56.847365+0000","last_became_peered":"2024-03-26T04:08:56.847365+0000","last_unstale":"2024-03-26T04:09:01.930297+0000","last_undegraded":"2024-03-26T04:09:01.930297+0000","last_fullsized":"2024-03-26T04:09:01.930297+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:14.382655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:14.382655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407337+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"448'67","reported_seq":981,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615683+0000","last_change":"2024-03-26T04:08:56.893591+0000","last_active":"2024-03-26T04:09:05.615683+0000","last_peered":"2024-03-26T04:09:05.615683+0000","last_clean":"2024-03-26T04:09:05.615683+0000","last_became_active":"2024-03-26T04:08:56.867218+0000","last_became_peered":"2024-03-26T04:08:56.867218+0000","last_unstale":"2024-03-26T04:09:05.615683+0000","last_undegraded":"2024-03-26T04:09:05.615683+0000","last_fullsized":"2024-03-26T04:09:05.615683+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'51","last_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.371335+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"455'4722","reported_seq":6983,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:08.143657+0000","last_change":"2024-03-26T04:08:57.961851+0000","last_active":"2024-03-26T04:09:08.143657+0000","last_peered":"2024-03-26T04:09:08.143657+0000","last_clean":"2024-03-26T04:08:55.803020+0000","last_became_active":"2024-03-26T04:08:57.925706+0000","last_became_peered":"2024-03-26T04:08:57.925706+0000","last_unstale":"2024-03-26T04:09:08.143657+0000","last_undegraded":"2024-03-26T04:09:08.143657+0000","last_fullsized":"2024-03-26T04:09:08.143657+0000","mapping_epoch":450,"log_start":"446'2215","ondisk_log_start":"446'2215","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2507,"log_dups_size":1099,"ondisk_log_size":2507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309460992,"num_objects":4722,"num_object_clones":0,"num_object_copies":14166,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3549,"num_objects_unfound":0,"num_objects_dirty":4722,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4722,"num_write_kb":302208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1095,"num_bytes_recovered":71761920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,7],"avail_no_missing":["2","1","7"],"object_location_counts":[{"shards":"1,2,7","objects":4722}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":841,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618931+0000","last_change":"2024-03-26T04:08:56.867601+0000","last_active":"2024-03-26T04:09:05.618931+0000","last_peered":"2024-03-26T04:09:05.618931+0000","last_clean":"2024-03-26T04:09:05.618931+0000","last_became_active":"2024-03-26T04:08:56.866823+0000","last_became_peered":"2024-03-26T04:08:56.866823+0000","last_unstale":"2024-03-26T04:09:05.618931+0000","last_undegraded":"2024-03-26T04:09:05.618931+0000","last_fullsized":"2024-03-26T04:09:05.618931+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:17.661725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:17.661725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:19.113467+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":773,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620136+0000","last_change":"2024-03-26T04:07:37.592031+0000","last_active":"2024-03-26T04:09:05.620136+0000","last_peered":"2024-03-26T04:09:05.620136+0000","last_clean":"2024-03-26T04:09:05.620136+0000","last_became_active":"2024-03-26T04:05:29.524644+0000","last_became_peered":"2024-03-26T04:05:29.524644+0000","last_unstale":"2024-03-26T04:09:05.620136+0000","last_undegraded":"2024-03-26T04:09:05.620136+0000","last_fullsized":"2024-03-26T04:09:05.620136+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.591946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.591946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":606,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619600+0000","last_change":"2024-03-26T04:07:34.405175+0000","last_active":"2024-03-26T04:09:05.619600+0000","last_peered":"2024-03-26T04:09:05.619600+0000","last_clean":"2024-03-26T04:09:05.619600+0000","last_became_active":"2024-03-26T04:05:29.531951+0000","last_became_peered":"2024-03-26T04:05:29.531951+0000","last_unstale":"2024-03-26T04:09:05.619600+0000","last_undegraded":"2024-03-26T04:09:05.619600+0000","last_fullsized":"2024-03-26T04:09:05.619600+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.404972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.404972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"448'27","reported_seq":933,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620060+0000","last_change":"2024-03-26T04:04:55.743092+0000","last_active":"2024-03-26T04:09:05.620060+0000","last_peered":"2024-03-26T04:09:05.620060+0000","last_clean":"2024-03-26T04:09:05.620060+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:09:05.620060+0000","last_undegraded":"2024-03-26T04:09:05.620060+0000","last_fullsized":"2024-03-26T04:09:05.620060+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:04:55.742967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.742967+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"455'4685","reported_seq":4750,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.137867+0000","last_change":"2024-03-26T04:08:02.633493+0000","last_active":"2024-03-26T04:09:06.137867+0000","last_peered":"2024-03-26T04:09:06.137867+0000","last_clean":"2024-03-26T04:09:06.137867+0000","last_became_active":"2024-03-26T04:08:02.633214+0000","last_became_peered":"2024-03-26T04:08:02.633214+0000","last_unstale":"2024-03-26T04:09:06.137867+0000","last_undegraded":"2024-03-26T04:09:06.137867+0000","last_fullsized":"2024-03-26T04:09:06.137867+0000","mapping_epoch":427,"log_start":"446'2284","ondisk_log_start":"446'2284","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2401,"log_dups_size":2284,"ondisk_log_size":2401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:24.022778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307036160,"num_objects":4685,"num_object_clones":0,"num_object_copies":14055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4685,"num_write_kb":299840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":657,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618030+0000","last_change":"2024-03-26T04:08:28.514002+0000","last_active":"2024-03-26T04:09:05.618030+0000","last_peered":"2024-03-26T04:09:05.618030+0000","last_clean":"2024-03-26T04:09:05.618030+0000","last_became_active":"2024-03-26T04:05:29.538433+0000","last_became_peered":"2024-03-26T04:05:29.538433+0000","last_unstale":"2024-03-26T04:09:05.618030+0000","last_undegraded":"2024-03-26T04:09:05.618030+0000","last_fullsized":"2024-03-26T04:09:05.618030+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:08:28.513772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:08:28.513772+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:32.954302+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":436,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930681+0000","last_change":"2024-03-26T04:08:56.865114+0000","last_active":"2024-03-26T04:09:01.930681+0000","last_peered":"2024-03-26T04:09:01.930681+0000","last_clean":"2024-03-26T04:09:01.930681+0000","last_became_active":"2024-03-26T04:08:56.862781+0000","last_became_peered":"2024-03-26T04:08:56.862781+0000","last_unstale":"2024-03-26T04:09:01.930681+0000","last_undegraded":"2024-03-26T04:09:01.930681+0000","last_fullsized":"2024-03-26T04:09:01.930681+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:03.264133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.264133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407308+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":741,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615621+0000","last_change":"2024-03-26T04:08:56.894926+0000","last_active":"2024-03-26T04:09:05.615621+0000","last_peered":"2024-03-26T04:09:05.615621+0000","last_clean":"2024-03-26T04:09:05.615621+0000","last_became_active":"2024-03-26T04:08:56.867760+0000","last_became_peered":"2024-03-26T04:08:56.867760+0000","last_unstale":"2024-03-26T04:09:05.615621+0000","last_undegraded":"2024-03-26T04:09:05.615621+0000","last_fullsized":"2024-03-26T04:09:05.615621+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:04:17.110315+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.110315+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":842,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929980+0000","last_change":"2024-03-26T04:04:52.483535+0000","last_active":"2024-03-26T04:09:01.929980+0000","last_peered":"2024-03-26T04:09:01.929980+0000","last_clean":"2024-03-26T04:09:01.929980+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:09:01.929980+0000","last_undegraded":"2024-03-26T04:09:01.929980+0000","last_fullsized":"2024-03-26T04:09:01.929980+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:52.483429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:04:52.483429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403689+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"450'64","reported_seq":1312,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.614967+0000","last_change":"2024-03-26T04:08:56.924513+0000","last_active":"2024-03-26T04:09:05.614967+0000","last_peered":"2024-03-26T04:09:05.614967+0000","last_clean":"2024-03-26T04:09:05.614967+0000","last_became_active":"2024-03-26T04:08:56.872084+0000","last_became_peered":"2024-03-26T04:08:56.872084+0000","last_unstale":"2024-03-26T04:09:05.614967+0000","last_undegraded":"2024-03-26T04:09:05.614967+0000","last_fullsized":"2024-03-26T04:09:05.614967+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'51","last_scrub_stamp":"2024-03-26T04:08:24.824658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.824658+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:26.619095+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":838,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.627511+0000","last_change":"2024-03-26T04:05:04.831962+0000","last_active":"2024-03-26T04:09:05.627511+0000","last_peered":"2024-03-26T04:09:05.627511+0000","last_clean":"2024-03-26T04:09:05.627511+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:09:05.627511+0000","last_undegraded":"2024-03-26T04:09:05.627511+0000","last_fullsized":"2024-03-26T04:09:05.627511+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:04.831457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:04.831457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"455'4577","reported_seq":4643,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.367072+0000","last_change":"2024-03-26T04:08:02.632925+0000","last_active":"2024-03-26T04:09:06.367072+0000","last_peered":"2024-03-26T04:09:06.367072+0000","last_clean":"2024-03-26T04:09:06.367072+0000","last_became_active":"2024-03-26T04:08:02.631353+0000","last_became_peered":"2024-03-26T04:08:02.631353+0000","last_unstale":"2024-03-26T04:09:06.367072+0000","last_undegraded":"2024-03-26T04:09:06.367072+0000","last_fullsized":"2024-03-26T04:09:06.367072+0000","mapping_epoch":427,"log_start":"444'2016","ondisk_log_start":"444'2016","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2561,"log_dups_size":857,"ondisk_log_size":2561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:13.206035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299958272,"num_objects":4577,"num_object_clones":0,"num_object_copies":13731,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4577,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4577,"num_write_kb":292928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":690,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.617810+0000","last_change":"2024-03-26T04:08:59.851553+0000","last_active":"2024-03-26T04:09:05.617810+0000","last_peered":"2024-03-26T04:09:05.617810+0000","last_clean":"2024-03-26T04:09:05.617810+0000","last_became_active":"2024-03-26T04:08:59.851169+0000","last_became_peered":"2024-03-26T04:08:59.851169+0000","last_unstale":"2024-03-26T04:09:05.617810+0000","last_undegraded":"2024-03-26T04:09:05.617810+0000","last_fullsized":"2024-03-26T04:09:05.617810+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:11.458734+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:04:11.458734+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":725,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.623959+0000","last_change":"2024-03-26T04:08:56.855459+0000","last_active":"2024-03-26T04:09:05.623959+0000","last_peered":"2024-03-26T04:09:05.623959+0000","last_clean":"2024-03-26T04:09:05.623959+0000","last_became_active":"2024-03-26T04:08:56.853944+0000","last_became_peered":"2024-03-26T04:08:56.853944+0000","last_unstale":"2024-03-26T04:09:05.623959+0000","last_undegraded":"2024-03-26T04:09:05.623959+0000","last_fullsized":"2024-03-26T04:09:05.623959+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.023066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.749979+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":867,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.612707+0000","last_change":"2024-03-26T04:08:29.947935+0000","last_active":"2024-03-26T04:09:05.612707+0000","last_peered":"2024-03-26T04:09:05.612707+0000","last_clean":"2024-03-26T04:09:05.612707+0000","last_became_active":"2024-03-26T04:05:29.533776+0000","last_became_peered":"2024-03-26T04:05:29.533776+0000","last_unstale":"2024-03-26T04:09:05.612707+0000","last_undegraded":"2024-03-26T04:09:05.612707+0000","last_fullsized":"2024-03-26T04:09:05.612707+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:29.947803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:08:29.947803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:57.711878+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":790,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621164+0000","last_change":"2024-03-26T04:08:24.797749+0000","last_active":"2024-03-26T04:09:05.621164+0000","last_peered":"2024-03-26T04:09:05.621164+0000","last_clean":"2024-03-26T04:09:05.621164+0000","last_became_active":"2024-03-26T04:05:29.543967+0000","last_became_peered":"2024-03-26T04:05:29.543967+0000","last_unstale":"2024-03-26T04:09:05.621164+0000","last_undegraded":"2024-03-26T04:09:05.621164+0000","last_fullsized":"2024-03-26T04:09:05.621164+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:24.797631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.797631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:38.015959+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"448'33","reported_seq":945,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.617076+0000","last_change":"2024-03-26T04:08:56.913082+0000","last_active":"2024-03-26T04:09:05.617076+0000","last_peered":"2024-03-26T04:09:05.617076+0000","last_clean":"2024-03-26T04:09:05.617076+0000","last_became_active":"2024-03-26T04:08:56.870173+0000","last_became_peered":"2024-03-26T04:08:56.870173+0000","last_unstale":"2024-03-26T04:09:05.617076+0000","last_undegraded":"2024-03-26T04:09:05.617076+0000","last_fullsized":"2024-03-26T04:09:05.617076+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'29","last_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:04:59.217868+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":782,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622020+0000","last_change":"2024-03-26T04:08:56.855137+0000","last_active":"2024-03-26T04:09:05.622020+0000","last_peered":"2024-03-26T04:09:05.622020+0000","last_clean":"2024-03-26T04:09:05.622020+0000","last_became_active":"2024-03-26T04:08:56.854146+0000","last_became_peered":"2024-03-26T04:08:56.854146+0000","last_unstale":"2024-03-26T04:09:05.622020+0000","last_undegraded":"2024-03-26T04:09:05.622020+0000","last_fullsized":"2024-03-26T04:09:05.622020+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:01:32.247532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:32.247532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"455'4594","reported_seq":4657,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.091708+0000","last_change":"2024-03-26T04:08:02.629892+0000","last_active":"2024-03-26T04:09:06.091708+0000","last_peered":"2024-03-26T04:09:06.091708+0000","last_clean":"2024-03-26T04:09:06.091708+0000","last_became_active":"2024-03-26T04:08:02.629608+0000","last_became_peered":"2024-03-26T04:08:02.629608+0000","last_unstale":"2024-03-26T04:09:06.091708+0000","last_undegraded":"2024-03-26T04:09:06.091708+0000","last_fullsized":"2024-03-26T04:09:06.091708+0000","mapping_epoch":427,"log_start":"446'2166","ondisk_log_start":"446'2166","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2428,"log_dups_size":2166,"ondisk_log_size":2428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:19.481427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301072384,"num_objects":4594,"num_object_clones":0,"num_object_copies":13782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":294016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":576,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930758+0000","last_change":"2024-03-26T04:08:56.866030+0000","last_active":"2024-03-26T04:09:01.930758+0000","last_peered":"2024-03-26T04:09:01.930758+0000","last_clean":"2024-03-26T04:09:01.930758+0000","last_became_active":"2024-03-26T04:08:56.862727+0000","last_became_peered":"2024-03-26T04:08:56.862727+0000","last_unstale":"2024-03-26T04:09:01.930758+0000","last_undegraded":"2024-03-26T04:09:01.930758+0000","last_fullsized":"2024-03-26T04:09:01.930758+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:04:02.403784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407216+0000","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":308,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616469+0000","last_change":"2024-03-26T04:08:59.853118+0000","last_active":"2024-03-26T04:09:05.616469+0000","last_peered":"2024-03-26T04:09:05.616469+0000","last_clean":"2024-03-26T04:09:05.616469+0000","last_became_active":"2024-03-26T04:08:59.852745+0000","last_became_peered":"2024-03-26T04:08:59.852745+0000","last_unstale":"2024-03-26T04:09:05.616469+0000","last_undegraded":"2024-03-26T04:09:05.616469+0000","last_fullsized":"2024-03-26T04:09:05.616469+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:08:16.158354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:08:16.158354+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:05.766265+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":470,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929523+0000","last_change":"2024-03-26T04:08:56.866956+0000","last_active":"2024-03-26T04:09:01.929523+0000","last_peered":"2024-03-26T04:09:01.929523+0000","last_clean":"2024-03-26T04:09:01.929523+0000","last_became_active":"2024-03-26T04:08:56.866010+0000","last_became_peered":"2024-03-26T04:08:56.866010+0000","last_unstale":"2024-03-26T04:09:01.929523+0000","last_undegraded":"2024-03-26T04:09:01.929523+0000","last_fullsized":"2024-03-26T04:09:01.929523+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:56.545464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:56.545464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407554+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"450'42","reported_seq":1108,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930771+0000","last_change":"2024-03-26T04:08:56.916941+0000","last_active":"2024-03-26T04:09:01.930771+0000","last_peered":"2024-03-26T04:09:01.930771+0000","last_clean":"2024-03-26T04:09:01.930771+0000","last_became_active":"2024-03-26T04:08:56.863726+0000","last_became_peered":"2024-03-26T04:08:56.863726+0000","last_unstale":"2024-03-26T04:09:01.930771+0000","last_undegraded":"2024-03-26T04:09:01.930771+0000","last_fullsized":"2024-03-26T04:09:01.930771+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:04:20.432551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:20.432551+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:08.355429+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":672,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.622359+0000","last_change":"2024-03-26T04:08:56.856134+0000","last_active":"2024-03-26T04:09:05.622359+0000","last_peered":"2024-03-26T04:09:05.622359+0000","last_clean":"2024-03-26T04:09:05.622359+0000","last_became_active":"2024-03-26T04:08:56.855807+0000","last_became_peered":"2024-03-26T04:08:56.855807+0000","last_unstale":"2024-03-26T04:09:05.622359+0000","last_undegraded":"2024-03-26T04:09:05.622359+0000","last_fullsized":"2024-03-26T04:09:05.622359+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:25.836032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:08:25.836032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:28.972138+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"455'4673","reported_seq":6486,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:06.387941+0000","last_change":"2024-03-26T04:08:57.964659+0000","last_active":"2024-03-26T04:09:06.387941+0000","last_peered":"2024-03-26T04:09:06.387941+0000","last_clean":"2024-03-26T04:08:55.793085+0000","last_became_active":"2024-03-26T04:08:57.927228+0000","last_became_peered":"2024-03-26T04:08:57.927228+0000","last_unstale":"2024-03-26T04:09:06.387941+0000","last_undegraded":"2024-03-26T04:09:06.387941+0000","last_fullsized":"2024-03-26T04:09:06.387941+0000","mapping_epoch":450,"log_start":"446'2073","ondisk_log_start":"446'2073","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2600,"log_dups_size":857,"ondisk_log_size":2600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":306249728,"num_objects":4673,"num_object_clones":0,"num_object_copies":14019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3743,"num_objects_unfound":0,"num_objects_dirty":4673,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4673,"num_write_kb":299072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":57081856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":4673}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":734,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.617585+0000","last_change":"2024-03-26T04:04:07.285956+0000","last_active":"2024-03-26T04:09:05.617585+0000","last_peered":"2024-03-26T04:09:05.617585+0000","last_clean":"2024-03-26T04:09:05.617585+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:09:05.617585+0000","last_undegraded":"2024-03-26T04:09:05.617585+0000","last_fullsized":"2024-03-26T04:09:05.617585+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:07.285700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:07.285700+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":865,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619594+0000","last_change":"2024-03-26T04:07:49.506391+0000","last_active":"2024-03-26T04:09:05.619594+0000","last_peered":"2024-03-26T04:09:05.619594+0000","last_clean":"2024-03-26T04:09:05.619594+0000","last_became_active":"2024-03-26T04:05:29.521425+0000","last_became_peered":"2024-03-26T04:05:29.521425+0000","last_unstale":"2024-03-26T04:09:05.619594+0000","last_undegraded":"2024-03-26T04:09:05.619594+0000","last_fullsized":"2024-03-26T04:09:05.619594+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.506266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.506266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.168658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":879,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618566+0000","last_change":"2024-03-26T04:08:56.867840+0000","last_active":"2024-03-26T04:09:05.618566+0000","last_peered":"2024-03-26T04:09:05.618566+0000","last_clean":"2024-03-26T04:09:05.618566+0000","last_became_active":"2024-03-26T04:08:56.867263+0000","last_became_peered":"2024-03-26T04:08:56.867263+0000","last_unstale":"2024-03-26T04:09:05.618566+0000","last_undegraded":"2024-03-26T04:09:05.618566+0000","last_fullsized":"2024-03-26T04:09:05.618566+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:56.329511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:56.329511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":79,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616479+0000","last_change":"2024-03-26T04:08:56.883694+0000","last_active":"2024-03-26T04:09:05.616479+0000","last_peered":"2024-03-26T04:09:05.616479+0000","last_clean":"2024-03-26T04:09:05.616479+0000","last_became_active":"2024-03-26T04:08:56.867031+0000","last_became_peered":"2024-03-26T04:08:56.867031+0000","last_unstale":"2024-03-26T04:09:05.616479+0000","last_undegraded":"2024-03-26T04:09:05.616479+0000","last_fullsized":"2024-03-26T04:09:05.616479+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:07:31.850790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.850790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":16,"num_read_kb":11,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"448'42","reported_seq":903,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616794+0000","last_change":"2024-03-26T04:07:49.546831+0000","last_active":"2024-03-26T04:09:05.616794+0000","last_peered":"2024-03-26T04:09:05.616794+0000","last_clean":"2024-03-26T04:09:05.616794+0000","last_became_active":"2024-03-26T04:05:29.538986+0000","last_became_peered":"2024-03-26T04:05:29.538986+0000","last_unstale":"2024-03-26T04:09:05.616794+0000","last_undegraded":"2024-03-26T04:09:05.616794+0000","last_fullsized":"2024-03-26T04:09:05.616794+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'30","last_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_deep_scrub":"422'30","last_deep_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.546733+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":550,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616519+0000","last_change":"2024-03-26T04:05:01.160620+0000","last_active":"2024-03-26T04:09:05.616519+0000","last_peered":"2024-03-26T04:09:05.616519+0000","last_clean":"2024-03-26T04:09:05.616519+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:09:05.616519+0000","last_undegraded":"2024-03-26T04:09:05.616519+0000","last_fullsized":"2024-03-26T04:09:05.616519+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:05:01.160427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"455'4632","reported_seq":4697,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.112299+0000","last_change":"2024-03-26T04:08:02.631311+0000","last_active":"2024-03-26T04:09:06.112299+0000","last_peered":"2024-03-26T04:09:06.112299+0000","last_clean":"2024-03-26T04:09:06.112299+0000","last_became_active":"2024-03-26T04:08:02.630915+0000","last_became_peered":"2024-03-26T04:08:02.630915+0000","last_unstale":"2024-03-26T04:09:06.112299+0000","last_undegraded":"2024-03-26T04:09:06.112299+0000","last_fullsized":"2024-03-26T04:09:06.112299+0000","mapping_epoch":427,"log_start":"446'2245","ondisk_log_start":"446'2245","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2387,"log_dups_size":2245,"ondisk_log_size":2387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:25.239641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303562752,"num_objects":4632,"num_object_clones":0,"num_object_copies":13896,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4632,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4632,"num_write_kb":296448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":698,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620952+0000","last_change":"2024-03-26T04:08:59.860837+0000","last_active":"2024-03-26T04:09:05.620952+0000","last_peered":"2024-03-26T04:09:05.620952+0000","last_clean":"2024-03-26T04:09:05.620952+0000","last_became_active":"2024-03-26T04:08:59.860447+0000","last_became_peered":"2024-03-26T04:08:59.860447+0000","last_unstale":"2024-03-26T04:09:05.620952+0000","last_undegraded":"2024-03-26T04:09:05.620952+0000","last_fullsized":"2024-03-26T04:09:05.620952+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:53.785762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:53.785762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":852,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615883+0000","last_change":"2024-03-26T04:08:56.868416+0000","last_active":"2024-03-26T04:09:05.615883+0000","last_peered":"2024-03-26T04:09:05.615883+0000","last_clean":"2024-03-26T04:09:05.615883+0000","last_became_active":"2024-03-26T04:08:56.866371+0000","last_became_peered":"2024-03-26T04:08:56.866371+0000","last_unstale":"2024-03-26T04:09:05.615883+0000","last_undegraded":"2024-03-26T04:09:05.615883+0000","last_fullsized":"2024-03-26T04:09:05.615883+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:41.426604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:07:41.426604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":867,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619448+0000","last_change":"2024-03-26T04:07:45.465265+0000","last_active":"2024-03-26T04:09:05.619448+0000","last_peered":"2024-03-26T04:09:05.619448+0000","last_clean":"2024-03-26T04:09:05.619448+0000","last_became_active":"2024-03-26T04:05:29.532029+0000","last_became_peered":"2024-03-26T04:05:29.532029+0000","last_unstale":"2024-03-26T04:09:05.619448+0000","last_undegraded":"2024-03-26T04:09:05.619448+0000","last_fullsized":"2024-03-26T04:09:05.619448+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.465179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.465179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"448'54","reported_seq":1016,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619736+0000","last_change":"2024-03-26T04:08:56.885200+0000","last_active":"2024-03-26T04:09:05.619736+0000","last_peered":"2024-03-26T04:09:05.619736+0000","last_clean":"2024-03-26T04:09:05.619736+0000","last_became_active":"2024-03-26T04:08:56.845070+0000","last_became_peered":"2024-03-26T04:08:56.845070+0000","last_unstale":"2024-03-26T04:09:05.619736+0000","last_undegraded":"2024-03-26T04:09:05.619736+0000","last_fullsized":"2024-03-26T04:09:05.619736+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.571532+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":806,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.617982+0000","last_change":"2024-03-26T04:08:56.841172+0000","last_active":"2024-03-26T04:09:05.617982+0000","last_peered":"2024-03-26T04:09:05.617982+0000","last_clean":"2024-03-26T04:09:05.617982+0000","last_became_active":"2024-03-26T04:08:56.840828+0000","last_became_peered":"2024-03-26T04:08:56.840828+0000","last_unstale":"2024-03-26T04:09:05.617982+0000","last_undegraded":"2024-03-26T04:09:05.617982+0000","last_fullsized":"2024-03-26T04:09:05.617982+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_clean_scrub_stamp":"2024-03-26T04:08:34.576640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:44.549530+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"455'4728","reported_seq":4791,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:08.149526+0000","last_change":"2024-03-26T04:08:02.629735+0000","last_active":"2024-03-26T04:09:08.149526+0000","last_peered":"2024-03-26T04:09:08.149526+0000","last_clean":"2024-03-26T04:09:08.149526+0000","last_became_active":"2024-03-26T04:08:02.629370+0000","last_became_peered":"2024-03-26T04:08:02.629370+0000","last_unstale":"2024-03-26T04:09:08.149526+0000","last_undegraded":"2024-03-26T04:09:08.149526+0000","last_fullsized":"2024-03-26T04:09:08.149526+0000","mapping_epoch":427,"log_start":"446'2130","ondisk_log_start":"446'2130","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2598,"log_dups_size":999,"ondisk_log_size":2598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:25.246651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309854208,"num_objects":4728,"num_object_clones":0,"num_object_copies":14184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4728,"num_write_kb":302592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":544,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619740+0000","last_change":"2024-03-26T04:08:59.865819+0000","last_active":"2024-03-26T04:09:05.619740+0000","last_peered":"2024-03-26T04:09:05.619740+0000","last_clean":"2024-03-26T04:09:05.619740+0000","last_became_active":"2024-03-26T04:08:59.865397+0000","last_became_peered":"2024-03-26T04:08:59.865397+0000","last_unstale":"2024-03-26T04:09:05.619740+0000","last_undegraded":"2024-03-26T04:09:05.619740+0000","last_fullsized":"2024-03-26T04:09:05.619740+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:29.526591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.526591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":863,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616357+0000","last_change":"2024-03-26T04:08:21.494251+0000","last_active":"2024-03-26T04:09:05.616357+0000","last_peered":"2024-03-26T04:09:05.616357+0000","last_clean":"2024-03-26T04:09:05.616357+0000","last_became_active":"2024-03-26T04:05:29.538900+0000","last_became_peered":"2024-03-26T04:05:29.538900+0000","last_unstale":"2024-03-26T04:09:05.616357+0000","last_undegraded":"2024-03-26T04:09:05.616357+0000","last_fullsized":"2024-03-26T04:09:05.616357+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:21.494131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:21.494131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:40.001098+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":503,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.623696+0000","last_change":"2024-03-26T04:08:56.851376+0000","last_active":"2024-03-26T04:09:05.623696+0000","last_peered":"2024-03-26T04:09:05.623696+0000","last_clean":"2024-03-26T04:09:05.623696+0000","last_became_active":"2024-03-26T04:08:56.850460+0000","last_became_peered":"2024-03-26T04:08:56.850460+0000","last_unstale":"2024-03-26T04:09:05.623696+0000","last_undegraded":"2024-03-26T04:09:05.623696+0000","last_fullsized":"2024-03-26T04:09:05.623696+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:03.918739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:03.918739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"448'36","reported_seq":62,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621884+0000","last_change":"2024-03-26T04:08:26.643613+0000","last_active":"2024-03-26T04:09:05.621884+0000","last_peered":"2024-03-26T04:09:05.621884+0000","last_clean":"2024-03-26T04:09:05.621884+0000","last_became_active":"2024-03-26T04:08:26.587565+0000","last_became_peered":"2024-03-26T04:08:26.587565+0000","last_unstale":"2024-03-26T04:09:05.621884+0000","last_undegraded":"2024-03-26T04:09:05.621884+0000","last_fullsized":"2024-03-26T04:09:05.621884+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:04:12.249693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:12.249693+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"450'16","reported_seq":1156,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619439+0000","last_change":"2024-03-26T04:08:56.882173+0000","last_active":"2024-03-26T04:09:05.619439+0000","last_peered":"2024-03-26T04:09:05.619439+0000","last_clean":"2024-03-26T04:09:05.619439+0000","last_became_active":"2024-03-26T04:08:56.862194+0000","last_became_peered":"2024-03-26T04:08:56.862194+0000","last_unstale":"2024-03-26T04:09:05.619439+0000","last_undegraded":"2024-03-26T04:09:05.619439+0000","last_fullsized":"2024-03-26T04:09:05.619439+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-03-26T04:08:31.583103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.583103+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:51.541812+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"454'4562","reported_seq":4633,"reported_epoch":454,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T04:09:04.742753+0000","last_change":"2024-03-26T04:08:57.891611+0000","last_active":"2024-03-26T04:09:04.742753+0000","last_peered":"2024-03-26T04:09:04.742753+0000","last_clean":"2024-03-26T04:08:55.792710+0000","last_became_active":"2024-03-26T04:08:57.888224+0000","last_became_peered":"2024-03-26T04:08:57.888224+0000","last_unstale":"2024-03-26T04:09:04.742753+0000","last_undegraded":"2024-03-26T04:09:04.742753+0000","last_fullsized":"2024-03-26T04:09:04.742753+0000","mapping_epoch":450,"log_start":"443'1731","ondisk_log_start":"443'1731","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2831,"log_dups_size":547,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4562,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":291968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,4,7","objects":4562}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":367,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929932+0000","last_change":"2024-03-26T04:08:59.842731+0000","last_active":"2024-03-26T04:09:01.929932+0000","last_peered":"2024-03-26T04:09:01.929932+0000","last_clean":"2024-03-26T04:09:01.929932+0000","last_became_active":"2024-03-26T04:08:59.842458+0000","last_became_peered":"2024-03-26T04:08:59.842458+0000","last_unstale":"2024-03-26T04:09:01.929932+0000","last_undegraded":"2024-03-26T04:09:01.929932+0000","last_fullsized":"2024-03-26T04:09:01.929932+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:26.838741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.838741+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":706,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619864+0000","last_change":"2024-03-26T04:08:56.849293+0000","last_active":"2024-03-26T04:09:05.619864+0000","last_peered":"2024-03-26T04:09:05.619864+0000","last_clean":"2024-03-26T04:09:05.619864+0000","last_became_active":"2024-03-26T04:08:56.848865+0000","last_became_peered":"2024-03-26T04:08:56.848865+0000","last_unstale":"2024-03-26T04:09:05.619864+0000","last_undegraded":"2024-03-26T04:09:05.619864+0000","last_fullsized":"2024-03-26T04:09:05.619864+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:40.042936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:07:40.042936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":848,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621315+0000","last_change":"2024-03-26T04:05:07.862785+0000","last_active":"2024-03-26T04:09:05.621315+0000","last_peered":"2024-03-26T04:09:05.621315+0000","last_clean":"2024-03-26T04:09:05.621315+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:09:05.621315+0000","last_undegraded":"2024-03-26T04:09:05.621315+0000","last_fullsized":"2024-03-26T04:09:05.621315+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:05:07.862659+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.862659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"448'48","reported_seq":988,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930262+0000","last_change":"2024-03-26T04:08:56.881594+0000","last_active":"2024-03-26T04:09:01.930262+0000","last_peered":"2024-03-26T04:09:01.930262+0000","last_clean":"2024-03-26T04:09:01.930262+0000","last_became_active":"2024-03-26T04:08:56.846958+0000","last_became_peered":"2024-03-26T04:08:56.846958+0000","last_unstale":"2024-03-26T04:09:01.930262+0000","last_undegraded":"2024-03-26T04:09:01.930262+0000","last_fullsized":"2024-03-26T04:09:01.930262+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:04:16.357311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:16.357311+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407279+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":571,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616751+0000","last_change":"2024-03-26T04:03:51.382658+0000","last_active":"2024-03-26T04:09:05.616751+0000","last_peered":"2024-03-26T04:09:05.616751+0000","last_clean":"2024-03-26T04:09:05.616751+0000","last_became_active":"2024-03-26T04:02:52.515390+0000","last_became_peered":"2024-03-26T04:02:52.515390+0000","last_unstale":"2024-03-26T04:09:05.616751+0000","last_undegraded":"2024-03-26T04:09:05.616751+0000","last_fullsized":"2024-03-26T04:09:05.616751+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:51.382563+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:03:51.382563+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"455'4619","reported_seq":6314,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:06.143089+0000","last_change":"2024-03-26T04:08:57.980575+0000","last_active":"2024-03-26T04:09:06.143089+0000","last_peered":"2024-03-26T04:09:06.143089+0000","last_clean":"2024-03-26T04:08:55.772811+0000","last_became_active":"2024-03-26T04:08:57.931431+0000","last_became_peered":"2024-03-26T04:08:57.931431+0000","last_unstale":"2024-03-26T04:09:06.143089+0000","last_undegraded":"2024-03-26T04:09:06.143089+0000","last_fullsized":"2024-03-26T04:09:06.143089+0000","mapping_epoch":450,"log_start":"446'2199","ondisk_log_start":"446'2199","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2420,"log_dups_size":1037,"ondisk_log_size":2420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302710784,"num_objects":4619,"num_object_clones":0,"num_object_copies":13857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3763,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":295616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":812,"num_bytes_recovered":53215232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4619}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":692,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930403+0000","last_change":"2024-03-26T04:07:39.474303+0000","last_active":"2024-03-26T04:09:01.930403+0000","last_peered":"2024-03-26T04:09:01.930403+0000","last_clean":"2024-03-26T04:09:01.930403+0000","last_became_active":"2024-03-26T04:05:29.521761+0000","last_became_peered":"2024-03-26T04:05:29.521761+0000","last_unstale":"2024-03-26T04:09:01.930403+0000","last_undegraded":"2024-03-26T04:09:01.930403+0000","last_fullsized":"2024-03-26T04:09:01.930403+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:39.474219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.474219+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":849,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929910+0000","last_change":"2024-03-26T04:08:56.846387+0000","last_active":"2024-03-26T04:09:01.929910+0000","last_peered":"2024-03-26T04:09:01.929910+0000","last_clean":"2024-03-26T04:09:01.929910+0000","last_became_active":"2024-03-26T04:08:56.845955+0000","last_became_peered":"2024-03-26T04:08:56.845955+0000","last_unstale":"2024-03-26T04:09:01.929910+0000","last_undegraded":"2024-03-26T04:09:01.929910+0000","last_fullsized":"2024-03-26T04:09:01.929910+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.569498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.569498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":356,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.620965+0000","last_change":"2024-03-26T04:08:56.867628+0000","last_active":"2024-03-26T04:09:05.620965+0000","last_peered":"2024-03-26T04:09:05.620965+0000","last_clean":"2024-03-26T04:09:05.620965+0000","last_became_active":"2024-03-26T04:08:56.867210+0000","last_became_peered":"2024-03-26T04:08:56.867210+0000","last_unstale":"2024-03-26T04:09:05.620965+0000","last_undegraded":"2024-03-26T04:09:05.620965+0000","last_fullsized":"2024-03-26T04:09:05.620965+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:06.121917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:06.121917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"448'64","reported_seq":467,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619085+0000","last_change":"2024-03-26T04:08:56.906816+0000","last_active":"2024-03-26T04:09:05.619085+0000","last_peered":"2024-03-26T04:09:05.619085+0000","last_clean":"2024-03-26T04:09:05.619085+0000","last_became_active":"2024-03-26T04:08:56.862404+0000","last_became_peered":"2024-03-26T04:08:56.862404+0000","last_unstale":"2024-03-26T04:09:05.619085+0000","last_undegraded":"2024-03-26T04:09:05.619085+0000","last_fullsized":"2024-03-26T04:09:05.619085+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:07:35.968179+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:07:35.968179+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":852,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929816+0000","last_change":"2024-03-26T04:08:56.848799+0000","last_active":"2024-03-26T04:09:01.929816+0000","last_peered":"2024-03-26T04:09:01.929816+0000","last_clean":"2024-03-26T04:09:01.929816+0000","last_became_active":"2024-03-26T04:08:56.847255+0000","last_became_peered":"2024-03-26T04:08:56.847255+0000","last_unstale":"2024-03-26T04:09:01.929816+0000","last_undegraded":"2024-03-26T04:09:01.929816+0000","last_fullsized":"2024-03-26T04:09:01.929816+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.297701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407186+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"455'4692","reported_seq":6686,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:06.149504+0000","last_change":"2024-03-26T04:08:57.896739+0000","last_active":"2024-03-26T04:09:06.149504+0000","last_peered":"2024-03-26T04:09:06.149504+0000","last_clean":"2024-03-26T04:08:55.811495+0000","last_became_active":"2024-03-26T04:08:57.892432+0000","last_became_peered":"2024-03-26T04:08:57.892432+0000","last_unstale":"2024-03-26T04:09:06.149504+0000","last_undegraded":"2024-03-26T04:09:06.149504+0000","last_fullsized":"2024-03-26T04:09:06.149504+0000","mapping_epoch":450,"log_start":"446'2266","ondisk_log_start":"446'2266","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2426,"log_dups_size":2266,"ondisk_log_size":2426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307494912,"num_objects":4692,"num_object_clones":0,"num_object_copies":14076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3660,"num_objects_unfound":0,"num_objects_dirty":4692,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4692,"num_write_kb":300288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":962,"num_bytes_recovered":63045632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,7],"avail_no_missing":["1","0","7"],"object_location_counts":[{"shards":"0,1,7","objects":4692}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":530,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619632+0000","last_change":"2024-03-26T04:07:22.530369+0000","last_active":"2024-03-26T04:09:05.619632+0000","last_peered":"2024-03-26T04:09:05.619632+0000","last_clean":"2024-03-26T04:09:05.619632+0000","last_became_active":"2024-03-26T04:05:29.524691+0000","last_became_peered":"2024-03-26T04:05:29.524691+0000","last_unstale":"2024-03-26T04:09:05.619632+0000","last_undegraded":"2024-03-26T04:09:05.619632+0000","last_fullsized":"2024-03-26T04:09:05.619632+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:22.530143+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:07:22.530143+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":529,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.615952+0000","last_change":"2024-03-26T04:08:20.860169+0000","last_active":"2024-03-26T04:09:05.615952+0000","last_peered":"2024-03-26T04:09:05.615952+0000","last_clean":"2024-03-26T04:09:05.615952+0000","last_became_active":"2024-03-26T04:05:29.532692+0000","last_became_peered":"2024-03-26T04:05:29.532692+0000","last_unstale":"2024-03-26T04:09:05.615952+0000","last_undegraded":"2024-03-26T04:09:05.615952+0000","last_fullsized":"2024-03-26T04:09:05.615952+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:20.860049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:08:20.860049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:36.127479+0000","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"379'13","reported_seq":1113,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931161+0000","last_change":"2024-03-26T04:07:54.544916+0000","last_active":"2024-03-26T04:09:01.931161+0000","last_peered":"2024-03-26T04:09:01.931161+0000","last_clean":"2024-03-26T04:09:01.931161+0000","last_became_active":"2024-03-26T04:05:29.521835+0000","last_became_peered":"2024-03-26T04:05:29.521835+0000","last_unstale":"2024-03-26T04:09:01.931161+0000","last_undegraded":"2024-03-26T04:09:01.931161+0000","last_fullsized":"2024-03-26T04:09:01.931161+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'13","last_scrub_stamp":"2024-03-26T04:07:54.544776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.544776+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:08.355942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":427,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.612511+0000","last_change":"2024-03-26T04:08:56.862714+0000","last_active":"2024-03-26T04:09:05.612511+0000","last_peered":"2024-03-26T04:09:05.612511+0000","last_clean":"2024-03-26T04:09:05.612511+0000","last_became_active":"2024-03-26T04:08:56.862406+0000","last_became_peered":"2024-03-26T04:08:56.862406+0000","last_unstale":"2024-03-26T04:09:05.612511+0000","last_undegraded":"2024-03-26T04:09:05.612511+0000","last_fullsized":"2024-03-26T04:09:05.612511+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:50.413099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:03:50.413099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"448'39","reported_seq":987,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.616968+0000","last_change":"2024-03-26T04:08:56.905523+0000","last_active":"2024-03-26T04:09:05.616968+0000","last_peered":"2024-03-26T04:09:05.616968+0000","last_clean":"2024-03-26T04:09:05.616968+0000","last_became_active":"2024-03-26T04:08:56.864029+0000","last_became_peered":"2024-03-26T04:08:56.864029+0000","last_unstale":"2024-03-26T04:09:05.616968+0000","last_undegraded":"2024-03-26T04:09:05.616968+0000","last_fullsized":"2024-03-26T04:09:05.616968+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:04:06.281970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.281970+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"455'4619","reported_seq":4685,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:06.321770+0000","last_change":"2024-03-26T04:08:02.634056+0000","last_active":"2024-03-26T04:09:06.321770+0000","last_peered":"2024-03-26T04:09:06.321770+0000","last_clean":"2024-03-26T04:09:06.321770+0000","last_became_active":"2024-03-26T04:08:02.633211+0000","last_became_peered":"2024-03-26T04:08:02.633211+0000","last_unstale":"2024-03-26T04:09:06.321770+0000","last_undegraded":"2024-03-26T04:09:06.321770+0000","last_fullsized":"2024-03-26T04:09:06.321770+0000","mapping_epoch":427,"log_start":"445'2039","ondisk_log_start":"445'2039","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2580,"log_dups_size":857,"ondisk_log_size":2580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:06.704876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302710784,"num_objects":4619,"num_object_clones":0,"num_object_copies":13857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4619,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4619,"num_write_kb":295616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619778+0000","last_change":"2024-03-26T04:08:56.872658+0000","last_active":"2024-03-26T04:09:05.619778+0000","last_peered":"2024-03-26T04:09:05.619778+0000","last_clean":"2024-03-26T04:09:05.619778+0000","last_became_active":"2024-03-26T04:08:56.871809+0000","last_became_peered":"2024-03-26T04:08:56.871809+0000","last_unstale":"2024-03-26T04:09:05.619778+0000","last_undegraded":"2024-03-26T04:09:05.619778+0000","last_fullsized":"2024-03-26T04:09:05.619778+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_clean_scrub_stamp":"2024-03-26T04:08:30.133040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:56.795906+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":570,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931086+0000","last_change":"2024-03-26T04:07:34.478534+0000","last_active":"2024-03-26T04:09:01.931086+0000","last_peered":"2024-03-26T04:09:01.931086+0000","last_clean":"2024-03-26T04:09:01.931086+0000","last_became_active":"2024-03-26T04:05:29.520457+0000","last_became_peered":"2024-03-26T04:05:29.520457+0000","last_unstale":"2024-03-26T04:09:01.931086+0000","last_undegraded":"2024-03-26T04:09:01.931086+0000","last_fullsized":"2024-03-26T04:09:01.931086+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.478427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.478427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:01.934278+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":851,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930023+0000","last_change":"2024-03-26T04:08:56.865804+0000","last_active":"2024-03-26T04:09:01.930023+0000","last_peered":"2024-03-26T04:09:01.930023+0000","last_clean":"2024-03-26T04:09:01.930023+0000","last_became_active":"2024-03-26T04:08:56.865055+0000","last_became_peered":"2024-03-26T04:08:56.865055+0000","last_unstale":"2024-03-26T04:09:01.930023+0000","last_undegraded":"2024-03-26T04:09:01.930023+0000","last_fullsized":"2024-03-26T04:09:01.930023+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.504517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.504517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"448'33","reported_seq":1004,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931021+0000","last_change":"2024-03-26T04:08:56.915010+0000","last_active":"2024-03-26T04:09:01.931021+0000","last_peered":"2024-03-26T04:09:01.931021+0000","last_clean":"2024-03-26T04:09:01.931021+0000","last_became_active":"2024-03-26T04:08:56.864260+0000","last_became_peered":"2024-03-26T04:08:56.864260+0000","last_unstale":"2024-03-26T04:09:01.931021+0000","last_undegraded":"2024-03-26T04:09:01.931021+0000","last_fullsized":"2024-03-26T04:09:01.931021+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T04:05:06.619261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:05:06.619261+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:08.355553+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"455'4748","reported_seq":6972,"reported_epoch":455,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:09:08.141682+0000","last_change":"2024-03-26T04:08:57.970048+0000","last_active":"2024-03-26T04:09:08.141682+0000","last_peered":"2024-03-26T04:09:08.141682+0000","last_clean":"2024-03-26T04:08:55.813179+0000","last_became_active":"2024-03-26T04:08:57.905506+0000","last_became_peered":"2024-03-26T04:08:57.905506+0000","last_unstale":"2024-03-26T04:09:08.141682+0000","last_undegraded":"2024-03-26T04:09:08.141682+0000","last_fullsized":"2024-03-26T04:09:08.141682+0000","mapping_epoch":450,"log_start":"446'2210","ondisk_log_start":"446'2210","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2538,"log_dups_size":999,"ondisk_log_size":2538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":311164928,"num_objects":4748,"num_object_clones":0,"num_object_copies":14244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3600,"num_objects_unfound":0,"num_objects_dirty":4748,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4748,"num_write_kb":303872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1076,"num_bytes_recovered":70516736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,7],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":4748}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":558,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.618621+0000","last_change":"2024-03-26T04:05:29.533929+0000","last_active":"2024-03-26T04:09:05.618621+0000","last_peered":"2024-03-26T04:09:05.618621+0000","last_clean":"2024-03-26T04:09:05.618621+0000","last_became_active":"2024-03-26T04:05:29.533470+0000","last_became_peered":"2024-03-26T04:05:29.533470+0000","last_unstale":"2024-03-26T04:09:05.618621+0000","last_undegraded":"2024-03-26T04:09:05.618621+0000","last_fullsized":"2024-03-26T04:09:05.618621+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:39.967821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:39.967821+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":769,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619400+0000","last_change":"2024-03-26T04:05:29.522502+0000","last_active":"2024-03-26T04:09:05.619400+0000","last_peered":"2024-03-26T04:09:05.619400+0000","last_clean":"2024-03-26T04:09:05.619400+0000","last_became_active":"2024-03-26T04:05:29.521089+0000","last_became_peered":"2024-03-26T04:05:29.521089+0000","last_unstale":"2024-03-26T04:09:05.619400+0000","last_undegraded":"2024-03-26T04:09:05.619400+0000","last_fullsized":"2024-03-26T04:09:05.619400+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_clean_scrub_stamp":"2024-03-26T04:02:49.166761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":835,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.614805+0000","last_change":"2024-03-26T04:04:49.184140+0000","last_active":"2024-03-26T04:09:05.614805+0000","last_peered":"2024-03-26T04:09:05.614805+0000","last_clean":"2024-03-26T04:09:05.614805+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:09:05.614805+0000","last_undegraded":"2024-03-26T04:09:05.614805+0000","last_fullsized":"2024-03-26T04:09:05.614805+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:49.184018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:04:49.184018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":807,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.929950+0000","last_change":"2024-03-26T04:08:56.866747+0000","last_active":"2024-03-26T04:09:01.929950+0000","last_peered":"2024-03-26T04:09:01.929950+0000","last_clean":"2024-03-26T04:09:01.929950+0000","last_became_active":"2024-03-26T04:08:56.865172+0000","last_became_peered":"2024-03-26T04:08:56.865172+0000","last_unstale":"2024-03-26T04:09:01.929950+0000","last_undegraded":"2024-03-26T04:09:01.929950+0000","last_fullsized":"2024-03-26T04:09:01.929950+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:08.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:08.666106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407616+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"448'45","reported_seq":911,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930888+0000","last_change":"2024-03-26T04:07:25.580036+0000","last_active":"2024-03-26T04:09:01.930888+0000","last_peered":"2024-03-26T04:09:01.930888+0000","last_clean":"2024-03-26T04:09:01.930888+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:09:01.930888+0000","last_undegraded":"2024-03-26T04:09:01.930888+0000","last_fullsized":"2024-03-26T04:09:01.930888+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:07:25.579888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.579888+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:09.396059+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"455'4642","reported_seq":4715,"reported_epoch":455,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-26T04:09:06.137968+0000","last_change":"2024-03-26T04:08:57.936246+0000","last_active":"2024-03-26T04:09:06.137968+0000","last_peered":"2024-03-26T04:09:06.137968+0000","last_clean":"2024-03-26T04:08:55.796551+0000","last_became_active":"2024-03-26T04:08:57.903542+0000","last_became_peered":"2024-03-26T04:08:57.903542+0000","last_unstale":"2024-03-26T04:09:06.137968+0000","last_undegraded":"2024-03-26T04:09:06.137968+0000","last_fullsized":"2024-03-26T04:09:06.137968+0000","mapping_epoch":450,"log_start":"446'2235","ondisk_log_start":"446'2235","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2407,"log_dups_size":1037,"ondisk_log_size":2407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304218112,"num_objects":4642,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4642,"num_write_kb":297088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4642}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":767,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.621512+0000","last_change":"2024-03-26T04:08:59.849038+0000","last_active":"2024-03-26T04:09:05.621512+0000","last_peered":"2024-03-26T04:09:05.621512+0000","last_clean":"2024-03-26T04:09:05.621512+0000","last_became_active":"2024-03-26T04:08:59.847656+0000","last_became_peered":"2024-03-26T04:08:59.847656+0000","last_unstale":"2024-03-26T04:09:05.621512+0000","last_undegraded":"2024-03-26T04:09:05.621512+0000","last_fullsized":"2024-03-26T04:09:05.621512+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.931565+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:05.734154+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":214679,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":832,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.931201+0000","last_change":"2024-03-26T04:07:27.607414+0000","last_active":"2024-03-26T04:09:01.931201+0000","last_peered":"2024-03-26T04:09:01.931201+0000","last_clean":"2024-03-26T04:09:01.931201+0000","last_became_active":"2024-03-26T04:05:29.520602+0000","last_became_peered":"2024-03-26T04:05:29.520602+0000","last_unstale":"2024-03-26T04:09:01.931201+0000","last_undegraded":"2024-03-26T04:09:01.931201+0000","last_fullsized":"2024-03-26T04:09:01.931201+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:27.607316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:07:27.607316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:03.403726+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"450'14","reported_seq":1028,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.623828+0000","last_change":"2024-03-26T04:08:56.858797+0000","last_active":"2024-03-26T04:09:05.623828+0000","last_peered":"2024-03-26T04:09:05.623828+0000","last_clean":"2024-03-26T04:09:05.623828+0000","last_became_active":"2024-03-26T04:08:56.845098+0000","last_became_peered":"2024-03-26T04:08:56.845098+0000","last_unstale":"2024-03-26T04:09:05.623828+0000","last_undegraded":"2024-03-26T04:09:05.623828+0000","last_fullsized":"2024-03-26T04:09:05.623828+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"181'10","last_scrub_stamp":"2024-03-26T04:01:34.236967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:01:34.236967+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":422,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-03-26T04:09:05.619915+0000","last_change":"2024-03-26T04:04:06.376407+0000","last_active":"2024-03-26T04:09:05.619915+0000","last_peered":"2024-03-26T04:09:05.619915+0000","last_clean":"2024-03-26T04:09:05.619915+0000","last_became_active":"2024-03-26T04:02:52.491417+0000","last_became_peered":"2024-03-26T04:02:52.491417+0000","last_unstale":"2024-03-26T04:09:05.619915+0000","last_undegraded":"2024-03-26T04:09:05.619915+0000","last_fullsized":"2024-03-26T04:09:05.619915+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.376284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.376284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"448'30","reported_seq":967,"reported_epoch":454,"state":"active+clean","last_fresh":"2024-03-26T04:09:01.930121+0000","last_change":"2024-03-26T04:08:56.892368+0000","last_active":"2024-03-26T04:09:01.930121+0000","last_peered":"2024-03-26T04:09:01.930121+0000","last_clean":"2024-03-26T04:09:01.930121+0000","last_became_active":"2024-03-26T04:08:56.863470+0000","last_became_peered":"2024-03-26T04:08:56.863470+0000","last_unstale":"2024-03-26T04:09:01.930121+0000","last_undegraded":"2024-03-26T04:09:01.930121+0000","last_fullsized":"2024-03-26T04:09:01.930121+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:04:18.408372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.408372+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:09:02.407370+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4883611648,"num_objects":74518,"num_object_clones":0,"num_object_copies":223554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":31243,"num_objects_unfound":0,"num_objects_dirty":74518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":74518,"num_write_kb":4769152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5740,"num_bytes_recovered":376176640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14993850368,"data_stored":14993850368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40286,"ondisk_log_size":40286,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":11,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697707,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":37,"num_read_kb":54,"num_write":32,"num_write_kb":438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":703368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3891200,"data_stored":3869888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":7471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1824,"num_read_kb":1549,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":10320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":675840,"data_stored":17960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1249,"ondisk_log_size":1249,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365055,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575127,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4851892,"kb_used_data":3905832,"kb_used_omap":1,"kb_used_meta":946046,"kb_avail":88893260,"statfs":{"total":95995035648,"available":91026698240,"internally_reserved":0,"allocated":3999571968,"data_stored":3998350743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":968751557},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738652,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3822308,"kb_used_data":2780440,"kb_used_omap":1,"kb_used_meta":1041854,"kb_avail":89922844,"statfs":{"total":95995035648,"available":92080992256,"internally_reserved":0,"allocated":2847170560,"data_stored":2845965420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1066858949},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,1,1,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637933,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3708308,"kb_used_data":3230092,"kb_used_omap":6,"kb_used_meta":478201,"kb_avail":90036844,"statfs":{"total":95995035648,"available":92197728256,"internally_reserved":0,"allocated":3307614208,"data_stored":3306421091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":489678117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,5,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953735,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3358668,"kb_used_data":2320964,"kb_used_omap":1,"kb_used_meta":1037694,"kb_avail":90386484,"statfs":{"total":95995035648,"available":92555759616,"internally_reserved":0,"allocated":2376667136,"data_stored":2375474208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1062599109},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2874015744,"data_stored":2874015744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4020830208,"data_stored":4020830208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3327459328,"data_stored":3327459328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2386100224,"data_stored":2386100224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2385444864,"data_stored":2385444864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:09:11.379 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 1, currently 11 2024-03-26T04:09:11.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:09:11.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[37501]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3298987356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[37501]: pgmap v980: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 47 MiB/s wr, 751 op/s; 31545/225150 objects misplaced (14.011%); 28 MiB/s, 453 objects/s recovering 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[29446]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:09:11.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3298987356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:09:11.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:11 smithi006 ceph-mon[29446]: pgmap v980: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 47 MiB/s wr, 751 op/s; 31545/225150 objects misplaced (14.011%); 28 MiB/s, 453 objects/s recovering 2024-03-26T04:09:11.403 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:11 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:09:11.403 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:11 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:11.403 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:11 smithi086 ceph-mon[32965]: from='client.15852 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:09:11.403 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:11 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3298987356' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:09:11.403 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:11 smithi086 ceph-mon[32965]: pgmap v980: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.6 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 47 MiB/s wr, 751 op/s; 31545/225150 objects misplaced (14.011%); 28 MiB/s, 453 objects/s recovering 2024-03-26T04:09:11.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:11.463 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:11.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:11.589 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:11.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:11.731 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:11.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:11.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:11.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:11.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:12.001 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:12.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:12.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 59 16 79911 79895 84.6169 62.8125 0.0160578 0.0118123 2024-03-26T04:09:12.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:12.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:12.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.259 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:12.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:12.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:12.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:12.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:12.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:12.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:12.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:12.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:12.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:12.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:12.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:12.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:12.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:13.037 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:13.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:13.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:09:13.119849+0000 min lat: 0.00121124 max lat: 2.33208 avg lat: 0.0118707 2024-03-26T04:09:13.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:09:13.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 60 16 80868 80852 84.2035 59.8125 0.0189947 0.0118707 2024-03-26T04:09:13.139 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:13.181 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:13.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:13.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:13.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:13.435 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:13.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:13.563 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:13.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:13.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:13.689 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:13.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:13.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:13.817 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:13.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:13.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:13.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:14.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:14.070 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:13 smithi086 ceph-mon[32965]: pgmap v981: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s; 31240/235842 objects misplaced (13.246%); 30 MiB/s, 482 objects/s recovering 2024-03-26T04:09:14.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 61 16 81762 81746 83.739 55.875 0.0108911 0.0119369 2024-03-26T04:09:14.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:13 smithi006 ceph-mon[29446]: pgmap v981: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s; 31240/235842 objects misplaced (13.246%); 30 MiB/s, 482 objects/s recovering 2024-03-26T04:09:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:13 smithi006 ceph-mon[37501]: pgmap v981: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.10k op/s; 31240/235842 objects misplaced (13.246%); 30 MiB/s, 482 objects/s recovering 2024-03-26T04:09:14.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:14.197 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:14.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:14.344 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:14.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:14.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:14.571 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:14.610 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:14.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:14.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:14.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.736 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:14.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:14.837 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:14.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:14.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:14.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:15.013 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:15.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:14 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:14 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:15.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:14 smithi086 ceph-mon[32965]: osdmap e457: 8 total, 8 up, 4 in 2024-03-26T04:09:15.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:14 smithi086 ceph-mon[32965]: osdmap e458: 8 total, 8 up, 4 in 2024-03-26T04:09:15.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:15.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 62 16 82498 82482 83.1304 46 0.00635248 0.0120236 2024-03-26T04:09:15.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[29446]: osdmap e457: 8 total, 8 up, 4 in 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[29446]: osdmap e458: 8 total, 8 up, 4 in 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[37501]: osdmap e457: 8 total, 8 up, 4 in 2024-03-26T04:09:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:14 smithi006 ceph-mon[37501]: osdmap e458: 8 total, 8 up, 4 in 2024-03-26T04:09:15.141 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:15.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:15.268 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:15.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:15.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:15.396 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:15.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:15.498 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:15.527 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:15.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:15.655 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:15.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:15.782 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:15.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:15.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:15.884 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:15.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:15.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:15.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:16.027 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:16.064 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:16.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:15 smithi086 ceph-mon[32965]: pgmap v984: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 852 B/s rd, 50 MiB/s wr, 799 op/s; 30873/238620 objects misplaced (12.938%); 24 MiB/s, 382 objects/s recovering 2024-03-26T04:09:16.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 63 16 83440 83424 82.7453 58.875 0.0130686 0.0120801 2024-03-26T04:09:16.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:15 smithi006 ceph-mon[29446]: pgmap v984: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 852 B/s rd, 50 MiB/s wr, 799 op/s; 30873/238620 objects misplaced (12.938%); 24 MiB/s, 382 objects/s recovering 2024-03-26T04:09:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:15 smithi006 ceph-mon[37501]: pgmap v984: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 4.8 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 852 B/s rd, 50 MiB/s wr, 799 op/s; 30873/238620 objects misplaced (12.938%); 24 MiB/s, 382 objects/s recovering 2024-03-26T04:09:16.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:16.191 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:16.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:16.293 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:16.335 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:16.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:16.379 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4] out_osds: [6, 5, 3, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-03-26T04:09:16.379 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:09:16.380 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-26T04:09:16.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:09:16.437 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:16.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.504 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:16.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:16.605 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:16.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:16.751 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:16.794 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:16.794 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:16.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:16.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:16.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:16.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:17.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:17.063 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:17.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:17.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 64 16 84248 84232 82.2415 50.5 0.0864748 0.0121456 2024-03-26T04:09:17.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:17.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:17.209 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:17.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:17.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:17.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:17.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:17.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:17.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:17.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:17.604 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:17.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:17.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:17.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:17.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:17.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:17.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:18.018 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:18.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:18.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 65 16 85054 85038 81.7512 50.375 0.00881564 0.0122267 2024-03-26T04:09:18.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:17 smithi006 ceph-mon[29446]: pgmap v985: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 30738/249570 objects misplaced (12.316%); 32 MiB/s, 517 objects/s recovering 2024-03-26T04:09:18.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:17 smithi006 ceph-mon[37501]: pgmap v985: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 30738/249570 objects misplaced (12.316%); 32 MiB/s, 517 objects/s recovering 2024-03-26T04:09:18.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:18.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:18.271 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:18.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:17 smithi086 ceph-mon[32965]: pgmap v985: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 30738/249570 objects misplaced (12.316%); 32 MiB/s, 517 objects/s recovering 2024-03-26T04:09:18.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:18.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.398 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:18.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:18.499 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:18.545 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:18.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:18.670 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:18.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:18.771 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.820 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:18.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:18.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:18.946 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:18.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:18.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:19.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:19.072 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:19.119 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 66 16 85866 85850 81.2815 50.75 0.0155169 0.0122959 2024-03-26T04:09:19.174 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:19.216 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:19.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:19.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:19.343 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.344 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:19.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:19.471 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:19.572 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:19.618 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:19.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:19.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:19.750 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:19.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:19.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:19.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:19.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:19.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:20.004 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:20.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:20.106 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:20.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 67 16 86733 86717 80.8764 54.1875 0.0107508 0.0123589 2024-03-26T04:09:20.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[37501]: pgmap v986: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 30424/251976 objects misplaced (12.074%); 34 MiB/s, 546 objects/s recovering 2024-03-26T04:09:20.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:20.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[29446]: pgmap v986: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 30424/251976 objects misplaced (12.074%); 34 MiB/s, 546 objects/s recovering 2024-03-26T04:09:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:20.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:19 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:20.149 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:20.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:20.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:19 smithi086 ceph-mon[32965]: pgmap v986: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s; 30424/251976 objects misplaced (12.074%); 34 MiB/s, 546 objects/s recovering 2024-03-26T04:09:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:19 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:19 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:20.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:20.420 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:20.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.569 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:20.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:20.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:20.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:20.854 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:20.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:20.997 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:20.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:20.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:21.099 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:21.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 68 16 87636 87620 80.5169 56.4375 0.0160941 0.0124041 2024-03-26T04:09:21.122 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:21.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:20 smithi006 ceph-mon[37501]: osdmap e459: 8 total, 8 up, 4 in 2024-03-26T04:09:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:20 smithi006 ceph-mon[29446]: osdmap e459: 8 total, 8 up, 4 in 2024-03-26T04:09:21.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:21.262 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:21.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:20 smithi086 ceph-mon[32965]: osdmap e459: 8 total, 8 up, 4 in 2024-03-26T04:09:21.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:21.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:21.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:21.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:21.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:21.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:21.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:21.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:21.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:21.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:21.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:21.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:21.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:22.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:22.036 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:22.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 69 16 88386 88370 80.0294 46.875 0.0161903 0.0124908 2024-03-26T04:09:22.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:22.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:21 smithi006 ceph-mon[29446]: pgmap v988: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 738 B/s rd, 42 MiB/s wr, 679 op/s; 30683/252753 objects misplaced (12.140%); 19 MiB/s, 299 objects/s recovering 2024-03-26T04:09:22.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:21 smithi006 ceph-mon[37501]: pgmap v988: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 738 B/s rd, 42 MiB/s wr, 679 op/s; 30683/252753 objects misplaced (12.140%); 19 MiB/s, 299 objects/s recovering 2024-03-26T04:09:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:22.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:22.301 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:21 smithi086 ceph-mon[32965]: pgmap v988: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.1 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 738 B/s rd, 42 MiB/s wr, 679 op/s; 30683/252753 objects misplaced (12.140%); 19 MiB/s, 299 objects/s recovering 2024-03-26T04:09:22.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:22.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:22.529 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:22.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:22.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:22.697 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:22.799 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:22.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:22.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:22.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:22.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:22.971 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:22.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:23.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:23.098 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:23.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:23.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 70 16 89303 89287 79.7048 57.3125 0.016519 0.0125394 2024-03-26T04:09:23.200 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:23.226 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:23.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:23.328 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:23.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:23.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:23.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:22 smithi086 ceph-mon[32965]: osdmap e460: 8 total, 8 up, 4 in 2024-03-26T04:09:23.370 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:23.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:23.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:23.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:23.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[37501]: osdmap e460: 8 total, 8 up, 4 in 2024-03-26T04:09:23.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:23.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:23.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:22 smithi006 ceph-mon[29446]: osdmap e460: 8 total, 8 up, 4 in 2024-03-26T04:09:23.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:23.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:23.616 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:23.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:23.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:23.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:23.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.805 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:23.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:23.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:23.932 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:23.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:24.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:24.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:24.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 71 16 90116 90100 79.2978 50.8125 0.00915679 0.0126052 2024-03-26T04:09:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[37501]: pgmap v990: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.3 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 983 op/s; 30383/262206 objects misplaced (11.587%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:09:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[29446]: pgmap v990: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.3 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 983 op/s; 30383/262206 objects misplaced (11.587%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:09:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:23 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:24.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:24.186 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:24.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:24.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:24.334 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:23 smithi086 ceph-mon[32965]: pgmap v990: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.3 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 983 op/s; 30383/262206 objects misplaced (11.587%); 26 MiB/s, 417 objects/s recovering 2024-03-26T04:09:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:23 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:23 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:24.436 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:24.477 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:24.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:24.622 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:24.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:24.724 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:24.753 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:24.855 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:24.878 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:24.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:24.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:25.005 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:25.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:25.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:25.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 72 16 90899 90883 78.8762 48.9375 0.0147124 0.0126725 2024-03-26T04:09:25.131 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:25.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:25.257 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:24 smithi086 ceph-mon[32965]: osdmap e461: 8 total, 8 up, 4 in 2024-03-26T04:09:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:24 smithi086 ceph-mon[32965]: osdmap e462: 8 total, 8 up, 4 in 2024-03-26T04:09:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:24 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:24 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:25.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:25.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:25.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[37501]: osdmap e461: 8 total, 8 up, 4 in 2024-03-26T04:09:25.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[37501]: osdmap e462: 8 total, 8 up, 4 in 2024-03-26T04:09:25.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:25.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:25.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[29446]: osdmap e461: 8 total, 8 up, 4 in 2024-03-26T04:09:25.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[29446]: osdmap e462: 8 total, 8 up, 4 in 2024-03-26T04:09:25.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:25.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:24 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:25.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:25.525 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:25.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:25.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:25.666 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:25.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:25.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:25.810 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:25.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:25.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:25.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:25.961 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:25.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:25.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:26.063 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:26.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:26.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:26.120 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 73 16 91696 91680 78.478 49.8125 0.0200859 0.0127376 2024-03-26T04:09:26.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:26.218 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:26.320 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:25 smithi086 ceph-mon[32965]: pgmap v993: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.4 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 29950/264483 objects misplaced (11.324%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:09:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:25 smithi086 ceph-mon[32965]: osdmap e463: 8 total, 8 up, 4 in 2024-03-26T04:09:26.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:26.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:26.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[37501]: pgmap v993: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.4 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 29950/264483 objects misplaced (11.324%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:09:26.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:26.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[37501]: osdmap e463: 8 total, 8 up, 4 in 2024-03-26T04:09:26.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[29446]: pgmap v993: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.4 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 1.0 KiB/s rd, 50 MiB/s wr, 805 op/s; 29950/264483 objects misplaced (11.324%); 25 MiB/s, 406 objects/s recovering 2024-03-26T04:09:26.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:26.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:25 smithi006 ceph-mon[29446]: osdmap e463: 8 total, 8 up, 4 in 2024-03-26T04:09:26.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:26.483 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:26.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:26.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:26.613 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:26.715 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:26.756 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:26.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:26.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:26.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:26.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:26.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:26.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:27.024 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:27.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 74 15 92678 92663 78.2477 61.4375 0.0108202 0.0127751 2024-03-26T04:09:27.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:27.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:27.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:27.278 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:27.317 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:27.317 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:27.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:27.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:27.404 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:27.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:27.549 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:27.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:27.676 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:27.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:27.778 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:27.820 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:27.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:27.963 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:27.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:27.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:28.065 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:28.106 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:28.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 75 16 93633 93617 77.9993 59.625 0.0131765 0.0128126 2024-03-26T04:09:28.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:28.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:28.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:27 smithi086 ceph-mon[32965]: pgmap v995: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 856 op/s; 29820/274599 objects misplaced (10.859%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:09:28.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:28.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:28.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:28.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:27 smithi006 ceph-mon[37501]: pgmap v995: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 856 op/s; 29820/274599 objects misplaced (10.859%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:09:28.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:27 smithi006 ceph-mon[29446]: pgmap v995: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 1.0 KiB/s rd, 53 MiB/s wr, 856 op/s; 29820/274599 objects misplaced (10.859%); 26 MiB/s, 413 objects/s recovering 2024-03-26T04:09:28.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:28.496 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:28.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:28.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:28.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:28.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:28.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:28.765 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:28.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:28.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:28.907 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:28.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:28.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:29.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:29.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:29.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 76 16 94346 94330 77.5594 44.5625 0.0131449 0.0128882 2024-03-26T04:09:29.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.176 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:29.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:29.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:29.305 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:29.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:29.433 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:29.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:29.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:29.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:29.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:29.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:29.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:29.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:29.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:29.951 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:29.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:29.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:30.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:30.076 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:29 smithi086 ceph-mon[32965]: pgmap v996: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 52 MiB/s wr, 829 op/s; 29395/277134 objects misplaced (10.607%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:09:30.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:29 smithi086 ceph-mon[32965]: osdmap e464: 8 total, 8 up, 4 in 2024-03-26T04:09:30.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 77 16 95136 95120 77.1933 49.375 0.0149452 0.0129352 2024-03-26T04:09:30.178 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:30.325 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:30.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.370 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:30.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:30.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:29 smithi006 ceph-mon[37501]: pgmap v996: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 52 MiB/s wr, 829 op/s; 29395/277134 objects misplaced (10.607%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:09:30.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:29 smithi006 ceph-mon[37501]: osdmap e464: 8 total, 8 up, 4 in 2024-03-26T04:09:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:29 smithi006 ceph-mon[29446]: pgmap v996: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 52 MiB/s wr, 829 op/s; 29395/277134 objects misplaced (10.607%); 28 MiB/s, 450 objects/s recovering 2024-03-26T04:09:30.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:29 smithi006 ceph-mon[29446]: osdmap e464: 8 total, 8 up, 4 in 2024-03-26T04:09:30.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:30.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:30.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:30.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:30.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:30.749 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:30.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:30.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:30.877 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:30.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:30.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:30.979 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:31.032 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:31.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 78 16 96023 96007 76.9143 55.4375 0.0138606 0.0129964 2024-03-26T04:09:31.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:31.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.159 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:31.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:31.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:31.290 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:30 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:30 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:30 smithi086 ceph-mon[32965]: osdmap e465: 8 total, 8 up, 4 in 2024-03-26T04:09:31.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:31.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:31.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[37501]: osdmap e465: 8 total, 8 up, 4 in 2024-03-26T04:09:31.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:31.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:31.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:30 smithi006 ceph-mon[29446]: osdmap e465: 8 total, 8 up, 4 in 2024-03-26T04:09:31.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:31.416 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:31.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:31.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:31.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:31.687 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:31.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:31.815 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:31.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:31.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:31.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:31.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:32.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:32.068 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:32.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:32.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 79 16 96872 96856 76.6124 53.0625 0.0115339 0.0130478 2024-03-26T04:09:32.170 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:32.211 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:32.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:32.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:31 smithi006 ceph-mon[37501]: pgmap v999: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s; 29644/277881 objects misplaced (10.668%); 21 MiB/s, 332 objects/s recovering 2024-03-26T04:09:32.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:31 smithi006 ceph-mon[29446]: pgmap v999: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s; 29644/277881 objects misplaced (10.668%); 21 MiB/s, 332 objects/s recovering 2024-03-26T04:09:32.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:32.336 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:32.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:31 smithi086 ceph-mon[32965]: pgmap v999: 158 pgs: 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 150 active+clean; 5.6 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 852 B/s rd, 46 MiB/s wr, 744 op/s; 29644/277881 objects misplaced (10.668%); 21 MiB/s, 332 objects/s recovering 2024-03-26T04:09:32.438 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:32.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.495 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:32.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:32.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:32.619 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:32.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:32.721 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.760 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:32.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:32.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:32.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:32.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:32.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:32.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.016 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:33.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:33.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:33.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:2024-03-26T04:09:33.122186+0000 min lat: 0.00121124 max lat: 2.33208 avg lat: 0.0131059 2024-03-26T04:09:33.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-26T04:09:33.122 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 80 16 97659 97643 76.2696 49.1875 0.0116428 0.0131059 2024-03-26T04:09:33.143 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:33.245 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:33.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:32 smithi086 ceph-mon[32965]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:33.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:32 smithi006 ceph-mon[37501]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:32 smithi006 ceph-mon[29446]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-26T04:09:33.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:33.512 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:33.556 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:33.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:33.700 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:33.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:33.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:33.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:33.927 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:33.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:33.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:34.069 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:34.106 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:34.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:34.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 81 16 98424 98408 75.9182 47.8125 0.0127182 0.0131673 2024-03-26T04:09:34.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:34.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:34.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:34.318 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:33 smithi006 ceph-mon[37501]: pgmap v1000: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 28979/287598 objects misplaced (10.076%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:09:34.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:33 smithi006 ceph-mon[29446]: pgmap v1000: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 28979/287598 objects misplaced (10.076%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:09:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:33 smithi086 ceph-mon[32965]: pgmap v1000: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.4 KiB/s rd, 65 MiB/s wr, 1.04k op/s; 28979/287598 objects misplaced (10.076%); 30 MiB/s, 486 objects/s recovering 2024-03-26T04:09:34.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:34.379 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:34.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:34.482 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:34.517 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:34.619 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:34.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:34.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:34.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:34.761 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:34.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:34.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:34.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:35.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:35.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:35.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:35.121 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 82 16 99314 99298 75.6707 55.625 0.0131972 0.0132101 2024-03-26T04:09:35.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:35.188 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:35.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:35.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:35.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:35.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:35.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:35.456 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:35.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:35.558 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:35.595 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:35.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:35.726 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:09:35.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:35.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:35 smithi086 ceph-mon[32965]: osdmap e466: 8 total, 8 up, 4 in 2024-03-26T04:09:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:35 smithi086 ceph-mon[32965]: pgmap v1002: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 852 B/s rd, 44 MiB/s wr, 711 op/s; 28526/289941 objects misplaced (9.839%); 22 MiB/s, 358 objects/s recovering 2024-03-26T04:09:35.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:35.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:35.862 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:35.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:35.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:35 smithi006 ceph-mon[37501]: osdmap e466: 8 total, 8 up, 4 in 2024-03-26T04:09:35.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:35 smithi006 ceph-mon[37501]: pgmap v1002: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 852 B/s rd, 44 MiB/s wr, 711 op/s; 28526/289941 objects misplaced (9.839%); 22 MiB/s, 358 objects/s recovering 2024-03-26T04:09:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:35 smithi006 ceph-mon[29446]: osdmap e466: 8 total, 8 up, 4 in 2024-03-26T04:09:35.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:35 smithi006 ceph-mon[29446]: pgmap v1002: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 852 B/s rd, 44 MiB/s wr, 711 op/s; 28526/289941 objects misplaced (9.839%); 22 MiB/s, 358 objects/s recovering 2024-03-26T04:09:35.964 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:36.031 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:36.122 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 83 16 100088 100072 75.3414 48.375 0.0155183 0.0132678 2024-03-26T04:09:36.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:36.181 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:36.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:36.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.324 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:36.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:36.466 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:36.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.611 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:36.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:36.736 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:36.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:36.883 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:36.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:36.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:36.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:37.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:37.122 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 84 15 100886 100871 75.0389 49.9375 0.0175976 0.0133216 2024-03-26T04:09:37.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:37.150 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:37.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:37.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:37.292 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:37.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:37.520 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:37.561 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:37.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:37.707 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:37.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:37.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:37.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:37.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:37 smithi006 ceph-mon[37501]: pgmap v1003: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 27979/299712 objects misplaced (9.335%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:09:37.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:37 smithi006 ceph-mon[29446]: pgmap v1003: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 27979/299712 objects misplaced (9.335%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:09:37.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:37.976 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:37.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:38.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:37 smithi086 ceph-mon[32965]: pgmap v1003: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 27979/299712 objects misplaced (9.335%); 32 MiB/s, 512 objects/s recovering 2024-03-26T04:09:38.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:38.122 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 85 16 101513 101497 74.6164 39.125 0.0127768 0.0133964 2024-03-26T04:09:38.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:38.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:38.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:38.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:38.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.370 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:38.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.497 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:38.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:38.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:38.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:38.723 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:38.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:38.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:38.895 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:38.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:38.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:39.024 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:39.122 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 86 16 102313 102297 74.3301 50 0.0118709 0.0134481 2024-03-26T04:09:39.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:39.151 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:39.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:39.385 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:39.429 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:39.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:39.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:39.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:39.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:39.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:39.814 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:39.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:39.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:39.942 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:39.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:40.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:40.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:40.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:39 smithi086 ceph-mon[32965]: pgmap v1004: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 983 op/s; 27526/301905 objects misplaced (9.117%); 33 MiB/s, 530 objects/s recovering 2024-03-26T04:09:40.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 87 16 103225 103209 74.1307 57 0.00902376 0.013485 2024-03-26T04:09:40.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:39 smithi006 ceph-mon[37501]: pgmap v1004: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 983 op/s; 27526/301905 objects misplaced (9.117%); 33 MiB/s, 530 objects/s recovering 2024-03-26T04:09:40.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:39 smithi006 ceph-mon[29446]: pgmap v1004: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 61 MiB/s wr, 983 op/s; 27526/301905 objects misplaced (9.117%); 33 MiB/s, 530 objects/s recovering 2024-03-26T04:09:40.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:40.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:40.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:40.321 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:40.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:40.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:40.449 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:40.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:40.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:40.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:40.691 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:40.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:40.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:40.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:40.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:41.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:41.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:41.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 88 16 103988 103972 73.8302 47.6875 0.0127775 0.0135268 2024-03-26T04:09:41.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:41.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:41.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:41.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:41.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:41.260 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:41.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:41.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:41.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:41.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:41.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:41.646 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:41.748 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:41.786 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:41.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:41.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:41.911 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:41.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:41.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:42.013 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:42.059 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:42.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:41 smithi086 ceph-mon[32965]: pgmap v1005: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 52 MiB/s wr, 826 op/s; 27776/302655 objects misplaced (9.177%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:42.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout: 89 16 104849 104833 73.6052 53.8125 0.00843958 0.0135813 2024-03-26T04:09:42.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:41 smithi006 ceph-mon[29446]: pgmap v1005: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 52 MiB/s wr, 826 op/s; 27776/302655 objects misplaced (9.177%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:42.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:41 smithi006 ceph-mon[37501]: pgmap v1005: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 52 MiB/s wr, 826 op/s; 27776/302655 objects misplaced (9.177%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:42.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.215 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:42.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:42.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:42.342 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:42.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:42.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:42.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:42.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:42.596 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:42.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:42.698 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:42.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:42.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:42.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:42.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:42.880 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:42.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:42.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:43.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total time run: 90.0099 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Total writes made: 105674 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Write size: 65536 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Object size: 65536 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Bandwidth (MB/sec): 73.3766 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Bandwidth: 32.7681 2024-03-26T04:09:43.123 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max bandwidth (MB/sec): 189.438 2024-03-26T04:09:43.124 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min bandwidth (MB/sec): 0 2024-03-26T04:09:43.124 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average IOPS: 1174 2024-03-26T04:09:43.124 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev IOPS: 524.302 2024-03-26T04:09:43.124 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max IOPS: 3031 2024-03-26T04:09:43.124 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min IOPS: 0 2024-03-26T04:09:43.125 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Average Latency(s): 0.0136239 2024-03-26T04:09:43.125 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Stddev Latency(s): 0.029754 2024-03-26T04:09:43.125 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Max latency(s): 2.33208 2024-03-26T04:09:43.125 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Min latency(s): 0.00121124 2024-03-26T04:09:43.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:43.129 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Cleaning up (deleting benchmark objects) 2024-03-26T04:09:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.163 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:43.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:43.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:43.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:43.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:43.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:43.428 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:43.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:43.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:43.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:43.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:43.681 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:43.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:43.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:43.807 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:43.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:43.934 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:43.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:44.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:44.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:44.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:43 smithi086 ceph-mon[32965]: pgmap v1006: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 50 MiB/s wr, 800 op/s; 26870/311583 objects misplaced (8.624%); 27 MiB/s, 437 objects/s recovering 2024-03-26T04:09:44.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:43 smithi006 ceph-mon[37501]: pgmap v1006: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 50 MiB/s wr, 800 op/s; 26870/311583 objects misplaced (8.624%); 27 MiB/s, 437 objects/s recovering 2024-03-26T04:09:44.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:43 smithi006 ceph-mon[29446]: pgmap v1006: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 921 B/s rd, 50 MiB/s wr, 800 op/s; 26870/311583 objects misplaced (8.624%); 27 MiB/s, 437 objects/s recovering 2024-03-26T04:09:44.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:44.185 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:44.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:44.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:44.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:44.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:44.439 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:44.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:44.541 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:44.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.591 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:44.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:44.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:44.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:44.717 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:44.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:44.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:44.947 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:09:44.962 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:44.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:44.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3204409048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:44 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:45.117 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:45.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3204409048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:44 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3204409048' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:44 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:09:45.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:45.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:45.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:45.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:45.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:45.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:45.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:45.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:45.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.641 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:45.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:45.767 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:45.809 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:45.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:45.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:45.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:45.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:45.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:46.020 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:46.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:45 smithi086 ceph-mon[32965]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.4 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1008 B/s rd, 49 MiB/s wr, 791 op/s; 26342/313932 objects misplaced (8.391%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:45 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:45 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:09:46.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:45 smithi086 ceph-mon[32965]: osdmap e467: 8 total, 8 up, 4 in 2024-03-26T04:09:46.121 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:46.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[29446]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.4 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1008 B/s rd, 49 MiB/s wr, 791 op/s; 26342/313932 objects misplaced (8.391%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:46.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:46.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:09:46.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[29446]: osdmap e467: 8 total, 8 up, 4 in 2024-03-26T04:09:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[37501]: pgmap v1007: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.4 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1008 B/s rd, 49 MiB/s wr, 791 op/s; 26342/313932 objects misplaced (8.391%); 27 MiB/s, 432 objects/s recovering 2024-03-26T04:09:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:09:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:09:46.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:45 smithi006 ceph-mon[37501]: osdmap e467: 8 total, 8 up, 4 in 2024-03-26T04:09:46.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.161 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:46.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:46.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:46.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:46.389 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:46.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:46.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:46.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.559 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:46.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:46.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:46.685 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:46.786 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:46.828 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:46.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:46.930 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:46.969 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:46.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:46.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:47.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:47.111 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:47.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:47.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:47.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:47.184 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:47.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:47.236 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:47.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:47.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:47.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:47.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:47.465 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:47.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:47.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:47.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:47.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:47.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:47.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:47.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:47.895 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:47.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:47.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:47.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:48.035 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:47 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T04:09:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:47 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T04:09:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:47 smithi086 ceph-mon[32965]: pgmap v1009: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 808 op/s; 24735/311502 objects misplaced (7.941%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:09:48.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[29446]: pgmap v1009: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 808 op/s; 24735/311502 objects misplaced (7.941%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T04:09:48.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:47 smithi006 ceph-mon[37501]: pgmap v1009: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 336 GiB / 358 GiB avail; 1.1 KiB/s rd, 37 MiB/s wr, 808 op/s; 24735/311502 objects misplaced (7.941%); 27 MiB/s, 438 objects/s recovering 2024-03-26T04:09:48.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:48.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:48.288 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:48.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:48.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:48.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:48.539 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:48.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:48.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:48.666 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:48.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:48.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.792 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:48.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:48.918 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:48.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:49.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:49.044 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:49.146 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:49.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.170 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:49.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:49.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:49.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:49.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:49.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:49.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.432 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:49.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:49.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:49.558 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:49.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:49.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:49.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:49.789 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:49.832 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:49.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:49.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:49.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:49.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:49.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:50.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:49 smithi086 ceph-mon[32965]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 807 op/s; 23915/309327 objects misplaced (7.731%); 28 MiB/s, 441 objects/s recovering 2024-03-26T04:09:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:49 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:09:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:49 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:09:50.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.116 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:50.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[29446]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 807 op/s; 23915/309327 objects misplaced (7.731%); 28 MiB/s, 441 objects/s recovering 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[37501]: pgmap v1010: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 807 op/s; 23915/309327 objects misplaced (7.731%); 28 MiB/s, 441 objects/s recovering 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:09:50.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:49 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:09:50.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:50.240 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:50.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:50.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:50.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:50.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.501 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:50.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:50.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:50.627 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:50.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:50.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:50.767 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:50.869 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:50.909 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:50.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:51.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:51.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:51.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:51.182 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:51.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:51.308 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:51.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:51.410 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:51.451 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:51.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:51.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:51.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:51.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:51.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:51.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:51.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:51.845 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:51.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:51.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[37501]: osdmap e468: 8 total, 8 up, 4 in 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[37501]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 767 B/s rd, 16 MiB/s wr, 636 op/s; 23666/308580 objects misplaced (7.669%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[29446]: osdmap e468: 8 total, 8 up, 4 in 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T04:09:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:51 smithi006 ceph-mon[29446]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 767 B/s rd, 16 MiB/s wr, 636 op/s; 23666/308580 objects misplaced (7.669%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:09:51.947 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:51.986 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:51.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:51 smithi086 ceph-mon[32965]: osdmap e468: 8 total, 8 up, 4 in 2024-03-26T04:09:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:51 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T04:09:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:51 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T04:09:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:51 smithi086 ceph-mon[32965]: pgmap v1012: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.3 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 767 B/s rd, 16 MiB/s wr, 636 op/s; 23666/308580 objects misplaced (7.669%); 23 MiB/s, 360 objects/s recovering 2024-03-26T04:09:52.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:52.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:52.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:52.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.256 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:52.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:52.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:52.380 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:52.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:52.509 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:52.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:52.636 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:52.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:52.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:52.763 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:52.865 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:52.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:52 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T04:09:52.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:52 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T04:09:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:52 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T04:09:52.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:52 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T04:09:52.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:52.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:52.905 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:09:53.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:53.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:52 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T04:09:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:52 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T04:09:53.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:53.155 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:53.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:53.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:53.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:53.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:53.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:53.541 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:53.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:53.667 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:53.769 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:53.811 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:53.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:53 smithi006 ceph-mon[37501]: pgmap v1013: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.2 KiB/s rd, 9.7 MiB/s wr, 935 op/s; 21579/298962 objects misplaced (7.218%); 29 MiB/s, 464 objects/s recovering 2024-03-26T04:09:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:53 smithi006 ceph-mon[29446]: pgmap v1013: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.2 KiB/s rd, 9.7 MiB/s wr, 935 op/s; 21579/298962 objects misplaced (7.218%); 29 MiB/s, 464 objects/s recovering 2024-03-26T04:09:53.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:53.938 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:53.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:54.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:54.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:54.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:53 smithi086 ceph-mon[32965]: pgmap v1013: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.1 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.2 KiB/s rd, 9.7 MiB/s wr, 935 op/s; 21579/298962 objects misplaced (7.218%); 29 MiB/s, 464 objects/s recovering 2024-03-26T04:09:54.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:54.209 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:54.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:54.337 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:54.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:54.475 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:54.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:54.603 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:54.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:54.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:09:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:09:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:09:54.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:54.731 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:54.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:54.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:54.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:54.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:54.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:09:54.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:55.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:55.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:55.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:55.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:55.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:55.353 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:55.400 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:55.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:55.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:55.530 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:55.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:55.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:55.657 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:55.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:55.784 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:55.802 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:09:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:09:55.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:55 smithi006 ceph-mon[37501]: pgmap v1014: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.0 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.8 MiB/s wr, 803 op/s; 20709/296781 objects misplaced (6.978%); 28 MiB/s, 451 objects/s recovering 2024-03-26T04:09:55.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:55 smithi006 ceph-mon[29446]: pgmap v1014: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.0 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.8 MiB/s wr, 803 op/s; 20709/296781 objects misplaced (6.978%); 28 MiB/s, 451 objects/s recovering 2024-03-26T04:09:55.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:55.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:55.912 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:55.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:56.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:56.038 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:56.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:55 smithi086 ceph-mon[32965]: pgmap v1014: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 6.0 GiB data, 21 GiB used, 337 GiB / 358 GiB avail; 1.0 KiB/s rd, 6.8 MiB/s wr, 803 op/s; 20709/296781 objects misplaced (6.978%); 28 MiB/s, 451 objects/s recovering 2024-03-26T04:09:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:09:56.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:56.165 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:56.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:56.303 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:56.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:56.430 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:56.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:56.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:56.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.572 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:56.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:56.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:56.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:56.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:09:56.823 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:09:56.824 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:56.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:56.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:56.981 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:56.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:57.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:56 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:57.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:57.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:56 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:09:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:56 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:09:57.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:57.233 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:57.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:57.335 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:09:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:57.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:09:57.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:57.504 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:57.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:57.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:57.632 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:57.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:57.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:57.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:57.859 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:57.896 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:57.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:57.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:58.023 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:57 smithi086 ceph-mon[32965]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 703 op/s; 19087/290433 objects misplaced (6.572%); 25 MiB/s, 397 objects/s recovering 2024-03-26T04:09:58.125 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:09:58.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:57 smithi006 ceph-mon[37501]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 703 op/s; 19087/290433 objects misplaced (6.572%); 25 MiB/s, 397 objects/s recovering 2024-03-26T04:09:58.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:57 smithi006 ceph-mon[29446]: pgmap v1015: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.9 GiB data, 20 GiB used, 337 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 703 op/s; 19087/290433 objects misplaced (6.572%); 25 MiB/s, 397 objects/s recovering 2024-03-26T04:09:58.155 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:09:58.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:58.283 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:58.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:58.410 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:58.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:58.537 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:58.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:58.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:58.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:09:58.807 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:09:58.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:09:58.935 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:58.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:58.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:09:59.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:59.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:59.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:09:59.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:59.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:09:59.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:59.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:59.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:59.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:59.458 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:59.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:09:59.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:59.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:09:59.686 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:59.727 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:59.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:09:59.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:09:59.853 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:09:59.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:09:59.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:09:59.994 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:09:59.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:09:59.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:09:59 smithi086 ceph-mon[32965]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s; 17442/283905 objects misplaced (6.144%); 30 MiB/s, 472 objects/s recovering 2024-03-26T04:10:00.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:00.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:00.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:09:59 smithi006 ceph-mon[29446]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s; 17442/283905 objects misplaced (6.144%); 30 MiB/s, 472 objects/s recovering 2024-03-26T04:10:00.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:09:59 smithi006 ceph-mon[37501]: pgmap v1016: 158 pgs: 1 active+remapped+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.8 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 848 op/s; 17442/283905 objects misplaced (6.144%); 30 MiB/s, 472 objects/s recovering 2024-03-26T04:10:00.224 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:00.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:00.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:00.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:00.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:00.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.514 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:00.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:00.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.659 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:00.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:00.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:00.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.802 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:00.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:00.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:00.929 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:00.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:01.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:01.057 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:01.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: daemon osd.2 on smithi006 is in unknown state 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:00 smithi086 ceph-mon[32965]: pg 11.e is active+remapped+backfill_toofull, acting [4,1,7] 2024-03-26T04:10:01.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: daemon osd.2 on smithi006 is in unknown state 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[37501]: pg 11.e is active+remapped+backfill_toofull, acting [4,1,7] 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: Health detail: HEALTH_WARN 1 failed cephadm daemon(s); nodeep-scrub flag(s) set; Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: [WRN] CEPHADM_FAILED_DAEMON: 1 failed cephadm daemon(s) 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: daemon osd.2 on smithi006 is in unknown state 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull 2024-03-26T04:10:01.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:00 smithi006 ceph-mon[29446]: pg 11.e is active+remapped+backfill_toofull, acting [4,1,7] 2024-03-26T04:10:01.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:01.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:01.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:01.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:01.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:01.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:01.439 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:01.468 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:01.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:01.569 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:01.603 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:01.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:01.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:01.730 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:01.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:01.832 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:01.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:01.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:01.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:01.976 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:02.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:02.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:01 smithi086 ceph-mon[32965]: pgmap v1017: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.7 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1008 B/s rd, 0 B/s wr, 838 op/s; 17167/283080 objects misplaced (6.064%); 29 MiB/s, 465 objects/s recovering 2024-03-26T04:10:02.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:02.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:01 smithi006 ceph-mon[29446]: pgmap v1017: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.7 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1008 B/s rd, 0 B/s wr, 838 op/s; 17167/283080 objects misplaced (6.064%); 29 MiB/s, 465 objects/s recovering 2024-03-26T04:10:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:01 smithi006 ceph-mon[37501]: pgmap v1017: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.7 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1008 B/s rd, 0 B/s wr, 838 op/s; 17167/283080 objects misplaced (6.064%); 29 MiB/s, 465 objects/s recovering 2024-03-26T04:10:02.161 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:02.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:02.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:02.412 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:02.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:02.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:02.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:02.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:02.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:02.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:02.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:02.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:02.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:02.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:03.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:03.042 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:03.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:03.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:03.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:03.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:03.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:02 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:03.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:03.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.187 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:03.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:03.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:03.313 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:03.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:10:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:10:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:02 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:03.415 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:03.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.444 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:03.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:03.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:03.571 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:03.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:03.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:03.698 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:03.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:03.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:03.825 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:03.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:03.926 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:03.950 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:03.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:03.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:04.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:04.076 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:03 smithi006 ceph-mon[29446]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 14972/273255 objects misplaced (5.479%); 35 MiB/s, 552 objects/s recovering 2024-03-26T04:10:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:03 smithi006 ceph-mon[29446]: pgmap v1019: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 878 op/s; 14972/273255 objects misplaced (5.479%); 32 MiB/s, 509 objects/s recovering 2024-03-26T04:10:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:03 smithi006 ceph-mon[37501]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 14972/273255 objects misplaced (5.479%); 35 MiB/s, 552 objects/s recovering 2024-03-26T04:10:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:03 smithi006 ceph-mon[37501]: pgmap v1019: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 878 op/s; 14972/273255 objects misplaced (5.479%); 32 MiB/s, 509 objects/s recovering 2024-03-26T04:10:04.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:04.208 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:04.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:04.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:03 smithi086 ceph-mon[32965]: pgmap v1018: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.00k op/s; 14972/273255 objects misplaced (5.479%); 35 MiB/s, 552 objects/s recovering 2024-03-26T04:10:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:03 smithi086 ceph-mon[32965]: pgmap v1019: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 20 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 878 op/s; 14972/273255 objects misplaced (5.479%); 32 MiB/s, 509 objects/s recovering 2024-03-26T04:10:04.351 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:04.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:04.478 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:04.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:04.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.606 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:04.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:04.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:04.707 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:04.730 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:04.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:04.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:04.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:04.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:04.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:04.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:04.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:04.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:05.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:05.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:05.228 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.272 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:05.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:05.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:05.406 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:05.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:05.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:05.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:05.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:05.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:05.670 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:05.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:05.797 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:05.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:05.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:05.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:05.939 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:05.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:05.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:06.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:06.066 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:06.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:05 smithi006 ceph-mon[37501]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 893 op/s; 14130/270648 objects misplaced (5.221%); 32 MiB/s, 510 objects/s recovering 2024-03-26T04:10:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:05 smithi006 ceph-mon[29446]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 893 op/s; 14130/270648 objects misplaced (5.221%); 32 MiB/s, 510 objects/s recovering 2024-03-26T04:10:06.167 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:06.201 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:06.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:06.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:06.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:06.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:05 smithi086 ceph-mon[32965]: pgmap v1020: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 893 op/s; 14130/270648 objects misplaced (5.221%); 32 MiB/s, 510 objects/s recovering 2024-03-26T04:10:06.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:06.453 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:06.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:06.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:06.580 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:06.682 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:06.715 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:06.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:06.817 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:06.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:06.953 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:06.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:06.988 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:06.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:07.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:07.112 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:07.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:07.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:06 smithi006 ceph-mon[37501]: osdmap e469: 8 total, 8 up, 4 in 2024-03-26T04:10:07.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:06 smithi006 ceph-mon[29446]: osdmap e469: 8 total, 8 up, 4 in 2024-03-26T04:10:07.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:07.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:06 smithi086 ceph-mon[32965]: osdmap e469: 8 total, 8 up, 4 in 2024-03-26T04:10:07.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:07.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:07.475 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:07.503 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:07.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:07.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:07.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:07.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:07.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:07.764 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:07.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:07.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:07.892 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:07.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:07.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:08.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:08.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:08.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:07 smithi006 ceph-mon[37501]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 659 B/s rd, 0 B/s wr, 604 op/s; 13861/269841 objects misplaced (5.137%); 20 MiB/s, 327 objects/s recovering 2024-03-26T04:10:08.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:07 smithi006 ceph-mon[37501]: osdmap e470: 8 total, 8 up, 4 in 2024-03-26T04:10:08.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:07 smithi006 ceph-mon[29446]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 659 B/s rd, 0 B/s wr, 604 op/s; 13861/269841 objects misplaced (5.137%); 20 MiB/s, 327 objects/s recovering 2024-03-26T04:10:08.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:07 smithi006 ceph-mon[29446]: osdmap e470: 8 total, 8 up, 4 in 2024-03-26T04:10:08.150 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:08.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:08.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:08.280 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:07 smithi086 ceph-mon[32965]: pgmap v1022: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 7 active+remapped+backfilling, 150 active+clean; 5.5 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 659 B/s rd, 0 B/s wr, 604 op/s; 13861/269841 objects misplaced (5.137%); 20 MiB/s, 327 objects/s recovering 2024-03-26T04:10:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:07 smithi086 ceph-mon[32965]: osdmap e470: 8 total, 8 up, 4 in 2024-03-26T04:10:08.381 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:08.409 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:08.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:08.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:08.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:08.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:08.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:08.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:08.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:08.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:08.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:08.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:08.916 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:08.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:09.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:09.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:09.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:09.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:08 smithi006 ceph-mon[37501]: osdmap e471: 8 total, 8 up, 4 in 2024-03-26T04:10:09.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:08 smithi006 ceph-mon[29446]: osdmap e471: 8 total, 8 up, 4 in 2024-03-26T04:10:09.144 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:09.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.178 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:09.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:09.279 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:09.315 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:09.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:08 smithi086 ceph-mon[32965]: osdmap e471: 8 total, 8 up, 4 in 2024-03-26T04:10:09.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:09.441 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:09.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:09.543 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:09.566 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:09.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:09.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:09.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:09.828 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:09.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:09.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:09.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:09.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:10.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:10.081 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:10.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:10.206 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:10.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:10.332 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:09 smithi086 ceph-mon[32965]: pgmap v1025: 158 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 749 op/s; 10609/259770 objects misplaced (4.084%); 39 MiB/s, 619 objects/s recovering 2024-03-26T04:10:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:09 smithi086 ceph-mon[32965]: osdmap e472: 8 total, 8 up, 4 in 2024-03-26T04:10:10.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:09 smithi006 ceph-mon[37501]: pgmap v1025: 158 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 749 op/s; 10609/259770 objects misplaced (4.084%); 39 MiB/s, 619 objects/s recovering 2024-03-26T04:10:10.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:09 smithi006 ceph-mon[37501]: osdmap e472: 8 total, 8 up, 4 in 2024-03-26T04:10:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:09 smithi006 ceph-mon[29446]: pgmap v1025: 158 pgs: 1 peering, 1 active+clean+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 749 op/s; 10609/259770 objects misplaced (4.084%); 39 MiB/s, 619 objects/s recovering 2024-03-26T04:10:10.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:09 smithi006 ceph-mon[29446]: osdmap e472: 8 total, 8 up, 4 in 2024-03-26T04:10:10.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:10.458 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:10.560 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:10.587 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:10.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:10.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:10.712 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:10.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:10.837 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:10.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:10.965 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:10.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:10.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:11.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:11.092 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:11.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:11.193 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:11.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.235 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:11.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:11.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:10 smithi086 ceph-mon[32965]: osdmap e473: 8 total, 8 up, 4 in 2024-03-26T04:10:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:10 smithi086 ceph-mon[32965]: 1.9 scrub starts 2024-03-26T04:10:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:10 smithi086 ceph-mon[32965]: 1.9 scrub ok 2024-03-26T04:10:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:11.360 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:11.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[37501]: osdmap e473: 8 total, 8 up, 4 in 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[37501]: 1.9 scrub starts 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[37501]: 1.9 scrub ok 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[29446]: osdmap e473: 8 total, 8 up, 4 in 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[29446]: 1.9 scrub starts 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[29446]: 1.9 scrub ok 2024-03-26T04:10:11.368 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:11.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:11.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.491 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:11.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:11.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:11.617 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:11.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:11.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:11.744 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:11.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:11.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:11.870 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:11.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:11.972 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.011 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:12.112 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:12.156 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:12.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:12.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:11 smithi086 ceph-mon[32965]: pgmap v1028: 158 pgs: 1 peering, 1 active+clean+remapped, 5 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 865 op/s; 8910/259473 objects misplaced (3.434%); 71 MiB/s, 1.14k objects/s recovering 2024-03-26T04:10:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:11 smithi086 ceph-mon[32965]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:10:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:11 smithi086 ceph-mon[32965]: osdmap e474: 8 total, 8 up, 4 in 2024-03-26T04:10:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:11 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T04:10:12.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:11 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[29446]: pgmap v1028: 158 pgs: 1 peering, 1 active+clean+remapped, 5 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 865 op/s; 8910/259473 objects misplaced (3.434%); 71 MiB/s, 1.14k objects/s recovering 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[29446]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[29446]: osdmap e474: 8 total, 8 up, 4 in 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[37501]: pgmap v1028: 158 pgs: 1 peering, 1 active+clean+remapped, 5 active+remapped+backfilling, 151 active+clean; 5.3 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 865 op/s; 8910/259473 objects misplaced (3.434%); 71 MiB/s, 1.14k objects/s recovering 2024-03-26T04:10:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[37501]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-26T04:10:12.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[37501]: osdmap e474: 8 total, 8 up, 4 in 2024-03-26T04:10:12.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T04:10:12.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:11 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T04:10:12.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:12.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:12.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:12.562 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:12.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:12.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.843 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:12.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:12.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:12.969 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:12.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:12.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:13.070 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:13.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:13.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:13.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:13.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:13.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:13.337 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:13.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:13.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:13.503 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:13.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:13.630 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:13.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:13.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:13.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:13.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:13.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:13.986 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:14.010 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:14.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:14.136 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:14.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:14.262 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:14.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:14.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:13 smithi006 ceph-mon[37501]: pgmap v1030: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 381 op/s; 5497/255954 objects misplaced (2.148%); 78 MiB/s, 1.25k objects/s recovering 2024-03-26T04:10:14.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:13 smithi006 ceph-mon[29446]: pgmap v1030: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 381 op/s; 5497/255954 objects misplaced (2.148%); 78 MiB/s, 1.25k objects/s recovering 2024-03-26T04:10:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:13 smithi086 ceph-mon[32965]: pgmap v1030: 158 pgs: 1 peering, 2 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 19 GiB used, 338 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 381 op/s; 5497/255954 objects misplaced (2.148%); 78 MiB/s, 1.25k objects/s recovering 2024-03-26T04:10:14.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:14.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.408 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:14.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:14.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:14.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:14.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:14.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:14.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:14.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-26T04:10:14.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:14.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:14.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:14.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:14.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:14.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:15.049 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:15.093 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:15.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:15.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:15.239 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:15.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:15.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:15.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:15.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:15.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:15.525 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:15.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:15.651 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:15.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:15.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:15.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:15.904 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:15.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:15.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:16.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:16.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:15 smithi006 ceph-mon[37501]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.2 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 310 op/s; 5497/254139 objects misplaced (2.163%); 52 MiB/s, 828 objects/s recovering 2024-03-26T04:10:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:15 smithi006 ceph-mon[29446]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.2 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 310 op/s; 5497/254139 objects misplaced (2.163%); 52 MiB/s, 828 objects/s recovering 2024-03-26T04:10:16.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:16.184 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:16.286 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.330 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:16.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:15 smithi086 ceph-mon[32965]: pgmap v1031: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.2 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 310 op/s; 5497/254139 objects misplaced (2.163%); 52 MiB/s, 828 objects/s recovering 2024-03-26T04:10:16.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:16.454 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:16.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:16.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:16.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:16.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:16.707 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:16.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:16.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:16.851 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:16.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:16.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:16.978 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:16.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:17.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:17.107 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:17.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:17.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:17.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:17.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:17.338 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:10:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=cleanup t=2024-03-26T04:10:17.00780338Z level=info msg="Completed cleanup jobs" duration=41.926076ms 2024-03-26T04:10:17.338 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:17.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:17.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:17.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:17.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:17.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:17.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:17.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:17.637 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:17.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:17.765 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:17.867 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:17.905 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:17.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:18.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.034 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:18.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:18.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:18.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:17 smithi006 ceph-mon[37501]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.1 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 284 op/s; 4222/253209 objects misplaced (1.667%); 50 MiB/s, 798 objects/s recovering 2024-03-26T04:10:18.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:17 smithi006 ceph-mon[29446]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.1 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 284 op/s; 4222/253209 objects misplaced (1.667%); 50 MiB/s, 798 objects/s recovering 2024-03-26T04:10:18.161 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:18.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:18.263 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:18.306 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:17 smithi086 ceph-mon[32965]: pgmap v1032: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 5.1 GiB data, 19 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 284 op/s; 4222/253209 objects misplaced (1.667%); 50 MiB/s, 798 objects/s recovering 2024-03-26T04:10:18.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:18.432 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:18.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:18.559 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:18.661 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:18.720 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:18.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:18.847 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:18.894 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:10:18.943 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:10:18.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:18.988 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:10:18.991 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:18.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:19.028 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:10:19.066 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:10:19.094 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:19.114 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:10:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.138 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:19.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:19.142 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/811867369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T04:10:19.154 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:18 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/811867369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:19.159 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:10:19.235 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:10:19.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:19.239 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:10:19.240 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:10:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:18 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T04:10:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:18 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T04:10:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:18 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/811867369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:19.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:19.393 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:19.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:19.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:19.519 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:19.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:19.673 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:19.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:19.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:19.813 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:19.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:19.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:19.935 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:10:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:19.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:19.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:19.947 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:10:20.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:20.069 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:20.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:20.196 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:20.298 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:19 smithi086 ceph-mon[32965]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 902 op/s; 3733/236130 objects misplaced (1.581%); 35 MiB/s, 564 objects/s recovering 2024-03-26T04:10:20.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:20.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:19 smithi006 ceph-mon[29446]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 902 op/s; 3733/236130 objects misplaced (1.581%); 35 MiB/s, 564 objects/s recovering 2024-03-26T04:10:20.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:19 smithi006 ceph-mon[37501]: pgmap v1033: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 339 GiB / 358 GiB avail; 0 B/s wr, 902 op/s; 3733/236130 objects misplaced (1.581%); 35 MiB/s, 564 objects/s recovering 2024-03-26T04:10:20.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:20.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.465 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:20.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:20.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:20.590 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:20.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:20.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:20.729 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:20.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:20.879 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:20.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:20.981 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:21.022 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:21.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:21.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:21.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:20 smithi006 ceph-mon[29446]: from='client.25787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:10:21.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:20 smithi006 ceph-mon[37501]: from='client.25787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:10:21.149 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:21.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:21.251 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:21.290 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:21.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:20 smithi086 ceph-mon[32965]: from='client.25787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:10:21.391 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:21.430 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:21.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:21.556 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:21.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:21.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:21.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:21.711 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1033,"stamp":"2024-03-26T04:10:18.506228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5143849829,"num_objects":78710,"num_object_clones":0,"num_object_copies":236130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3733,"num_objects_unfound":0,"num_objects_dirty":78710,"num_whiteouts":0,"num_read":1952,"num_read_kb":1680,"num_write":134086,"num_write_kb":6763401,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40108,"num_bytes_recovered":2588850192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43223,"ondisk_log_size":43223,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":18999576,"kb_used_data":14967352,"kb_used_omap":11,"kb_used_meta":4032180,"kb_avail":355981032,"statfs":{"total":383980142592,"available":364524576768,"internally_reserved":0,"allocated":15326568448,"data_stored":15323844288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11660,"internal_metadata":4128952948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-509989286,"num_objects":-7781,"num_object_clones":0,"num_object_copies":-23343,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-5177,"num_objects_unfound":0,"num_objects_dirty":-7781,"num_whiteouts":0,"num_read":-9,"num_read_kb":-18,"num_write":7765,"num_write_kb":-219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4856,"num_bytes_recovered":318028137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.606261"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":884,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905374+0000","last_change":"2024-03-26T04:08:56.865789+0000","last_active":"2024-03-26T04:10:10.905374+0000","last_peered":"2024-03-26T04:10:10.905374+0000","last_clean":"2024-03-26T04:10:10.905374+0000","last_became_active":"2024-03-26T04:08:56.862531+0000","last_became_peered":"2024-03-26T04:08:56.862531+0000","last_unstale":"2024-03-26T04:10:10.905374+0000","last_undegraded":"2024-03-26T04:10:10.905374+0000","last_fullsized":"2024-03-26T04:10:10.905374+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:03.632384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:03.632384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":894,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910325+0000","last_change":"2024-03-26T04:08:56.863479+0000","last_active":"2024-03-26T04:10:10.910325+0000","last_peered":"2024-03-26T04:10:10.910325+0000","last_clean":"2024-03-26T04:10:10.910325+0000","last_became_active":"2024-03-26T04:08:56.863125+0000","last_became_peered":"2024-03-26T04:08:56.863125+0000","last_unstale":"2024-03-26T04:10:10.910325+0000","last_undegraded":"2024-03-26T04:10:10.910325+0000","last_fullsized":"2024-03-26T04:10:10.910325+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:55.521958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:58:37.457112+0000","last_clean_scrub_stamp":"2024-03-26T04:07:55.521958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":901,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913481+0000","last_change":"2024-03-26T04:05:29.539024+0000","last_active":"2024-03-26T04:10:10.913481+0000","last_peered":"2024-03-26T04:10:10.913481+0000","last_clean":"2024-03-26T04:10:10.913481+0000","last_became_active":"2024-03-26T04:05:29.538364+0000","last_became_peered":"2024-03-26T04:05:29.538364+0000","last_unstale":"2024-03-26T04:10:10.913481+0000","last_undegraded":"2024-03-26T04:10:10.913481+0000","last_fullsized":"2024-03-26T04:10:10.913481+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:52.117692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:02:52.117692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"448'27","reported_seq":998,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905072+0000","last_change":"2024-03-26T04:08:31.314138+0000","last_active":"2024-03-26T04:10:10.905072+0000","last_peered":"2024-03-26T04:10:10.905072+0000","last_clean":"2024-03-26T04:10:10.905072+0000","last_became_active":"2024-03-26T04:05:29.520084+0000","last_became_peered":"2024-03-26T04:05:29.520084+0000","last_unstale":"2024-03-26T04:10:10.905072+0000","last_undegraded":"2024-03-26T04:10:10.905072+0000","last_fullsized":"2024-03-26T04:10:10.905072+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:08:31.313993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.313993+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":266,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909983+0000","last_change":"2024-03-26T04:08:56.869793+0000","last_active":"2024-03-26T04:10:10.909983+0000","last_peered":"2024-03-26T04:10:10.909983+0000","last_clean":"2024-03-26T04:10:10.909983+0000","last_became_active":"2024-03-26T04:08:56.869461+0000","last_became_peered":"2024-03-26T04:08:56.869461+0000","last_unstale":"2024-03-26T04:10:10.909983+0000","last_undegraded":"2024-03-26T04:10:10.909983+0000","last_fullsized":"2024-03-26T04:10:10.909983+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_clean_scrub_stamp":"2024-03-26T04:08:35.321382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":890,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.904964+0000","last_change":"2024-03-26T04:09:46.678800+0000","last_active":"2024-03-26T04:10:10.904964+0000","last_peered":"2024-03-26T04:10:10.904964+0000","last_clean":"2024-03-26T04:10:10.904964+0000","last_became_active":"2024-03-26T04:05:29.519486+0000","last_became_peered":"2024-03-26T04:05:29.519486+0000","last_unstale":"2024-03-26T04:10:10.904964+0000","last_undegraded":"2024-03-26T04:10:10.904964+0000","last_fullsized":"2024-03-26T04:10:10.904964+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:09:46.678694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:09:46.678694+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:11:05.052434+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":446,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907618+0000","last_change":"2024-03-26T04:08:56.855912+0000","last_active":"2024-03-26T04:10:10.907618+0000","last_peered":"2024-03-26T04:10:10.907618+0000","last_clean":"2024-03-26T04:10:10.907618+0000","last_became_active":"2024-03-26T04:08:56.854666+0000","last_became_peered":"2024-03-26T04:08:56.854666+0000","last_unstale":"2024-03-26T04:10:10.907618+0000","last_undegraded":"2024-03-26T04:10:10.907618+0000","last_fullsized":"2024-03-26T04:10:10.907618+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.526536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.526536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"448'63","reported_seq":369,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906269+0000","last_change":"2024-03-26T04:07:44.566663+0000","last_active":"2024-03-26T04:10:10.906269+0000","last_peered":"2024-03-26T04:10:10.906269+0000","last_clean":"2024-03-26T04:10:10.906269+0000","last_became_active":"2024-03-26T04:05:29.533552+0000","last_became_peered":"2024-03-26T04:05:29.533552+0000","last_unstale":"2024-03-26T04:10:10.906269+0000","last_undegraded":"2024-03-26T04:10:10.906269+0000","last_fullsized":"2024-03-26T04:10:10.906269+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'51","last_scrub_stamp":"2024-03-26T04:07:44.566551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:44.566551+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":96,"num_read_kb":83,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":869,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911528+0000","last_change":"2024-03-26T04:08:56.863386+0000","last_active":"2024-03-26T04:10:10.911528+0000","last_peered":"2024-03-26T04:10:10.911528+0000","last_clean":"2024-03-26T04:10:10.911528+0000","last_became_active":"2024-03-26T04:08:56.863103+0000","last_became_peered":"2024-03-26T04:08:56.863103+0000","last_unstale":"2024-03-26T04:10:10.911528+0000","last_undegraded":"2024-03-26T04:10:10.911528+0000","last_fullsized":"2024-03-26T04:10:10.911528+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:21.537431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:21.537431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":630,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911820+0000","last_change":"2024-03-26T04:08:56.865865+0000","last_active":"2024-03-26T04:10:10.911820+0000","last_peered":"2024-03-26T04:10:10.911820+0000","last_clean":"2024-03-26T04:10:10.911820+0000","last_became_active":"2024-03-26T04:08:56.865530+0000","last_became_peered":"2024-03-26T04:08:56.865530+0000","last_unstale":"2024-03-26T04:10:10.911820+0000","last_undegraded":"2024-03-26T04:10:10.911820+0000","last_fullsized":"2024-03-26T04:10:10.911820+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:53.342172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.342172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":586,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906440+0000","last_change":"2024-03-26T04:08:56.866879+0000","last_active":"2024-03-26T04:10:10.906440+0000","last_peered":"2024-03-26T04:10:10.906440+0000","last_clean":"2024-03-26T04:10:10.906440+0000","last_became_active":"2024-03-26T04:08:56.865538+0000","last_became_peered":"2024-03-26T04:08:56.865538+0000","last_unstale":"2024-03-26T04:10:10.906440+0000","last_undegraded":"2024-03-26T04:10:10.906440+0000","last_fullsized":"2024-03-26T04:10:10.906440+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:55.527836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.527836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"448'34","reported_seq":999,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905546+0000","last_change":"2024-03-26T04:07:26.622146+0000","last_active":"2024-03-26T04:10:10.905546+0000","last_peered":"2024-03-26T04:10:10.905546+0000","last_clean":"2024-03-26T04:10:10.905546+0000","last_became_active":"2024-03-26T04:05:29.521715+0000","last_became_peered":"2024-03-26T04:05:29.521715+0000","last_unstale":"2024-03-26T04:10:10.905546+0000","last_undegraded":"2024-03-26T04:10:10.905546+0000","last_fullsized":"2024-03-26T04:10:10.905546+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'28","last_scrub_stamp":"2024-03-26T04:07:26.622024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.622024+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":682,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910516+0000","last_change":"2024-03-26T04:07:38.459491+0000","last_active":"2024-03-26T04:10:10.910516+0000","last_peered":"2024-03-26T04:10:10.910516+0000","last_clean":"2024-03-26T04:10:10.910516+0000","last_became_active":"2024-03-26T04:05:29.532847+0000","last_became_peered":"2024-03-26T04:05:29.532847+0000","last_unstale":"2024-03-26T04:10:10.910516+0000","last_undegraded":"2024-03-26T04:10:10.910516+0000","last_fullsized":"2024-03-26T04:10:10.910516+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.459399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.459399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":60,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905787+0000","last_change":"2024-03-26T04:08:56.866656+0000","last_active":"2024-03-26T04:10:10.905787+0000","last_peered":"2024-03-26T04:10:10.905787+0000","last_clean":"2024-03-26T04:10:10.905787+0000","last_became_active":"2024-03-26T04:08:56.862969+0000","last_became_peered":"2024-03-26T04:08:56.862969+0000","last_unstale":"2024-03-26T04:10:10.905787+0000","last_undegraded":"2024-03-26T04:10:10.905787+0000","last_fullsized":"2024-03-26T04:10:10.905787+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.766403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:35.757523+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.766403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":59,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905552+0000","last_change":"2024-03-26T04:08:56.863129+0000","last_active":"2024-03-26T04:10:10.905552+0000","last_peered":"2024-03-26T04:10:10.905552+0000","last_clean":"2024-03-26T04:10:10.905552+0000","last_became_active":"2024-03-26T04:08:56.862318+0000","last_became_peered":"2024-03-26T04:08:56.862318+0000","last_unstale":"2024-03-26T04:10:10.905552+0000","last_undegraded":"2024-03-26T04:10:10.905552+0000","last_fullsized":"2024-03-26T04:10:10.905552+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:01.292182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:01.292182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"448'18","reported_seq":953,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905891+0000","last_change":"2024-03-26T04:08:33.351243+0000","last_active":"2024-03-26T04:10:10.905891+0000","last_peered":"2024-03-26T04:10:10.905891+0000","last_clean":"2024-03-26T04:10:10.905891+0000","last_became_active":"2024-03-26T04:05:29.519855+0000","last_became_peered":"2024-03-26T04:05:29.519855+0000","last_unstale":"2024-03-26T04:10:10.905891+0000","last_undegraded":"2024-03-26T04:10:10.905891+0000","last_fullsized":"2024-03-26T04:10:10.905891+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'16","last_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_deep_scrub":"241'16","last_deep_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_clean_scrub_stamp":"2024-03-26T04:08:33.351097+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":910,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906917+0000","last_change":"2024-03-26T04:08:56.847258+0000","last_active":"2024-03-26T04:10:10.906917+0000","last_peered":"2024-03-26T04:10:10.906917+0000","last_clean":"2024-03-26T04:10:10.906917+0000","last_became_active":"2024-03-26T04:08:56.845524+0000","last_became_peered":"2024-03-26T04:08:56.845524+0000","last_unstale":"2024-03-26T04:10:10.906917+0000","last_undegraded":"2024-03-26T04:10:10.906917+0000","last_fullsized":"2024-03-26T04:10:10.906917+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:02.671615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:05:02.671615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":872,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912991+0000","last_change":"2024-03-26T04:04:51.186722+0000","last_active":"2024-03-26T04:10:10.912991+0000","last_peered":"2024-03-26T04:10:10.912991+0000","last_clean":"2024-03-26T04:10:10.912991+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:10:10.912991+0000","last_undegraded":"2024-03-26T04:10:10.912991+0000","last_fullsized":"2024-03-26T04:10:10.912991+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.186605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.186605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":889,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912158+0000","last_change":"2024-03-26T04:08:56.866535+0000","last_active":"2024-03-26T04:10:10.912158+0000","last_peered":"2024-03-26T04:10:10.912158+0000","last_clean":"2024-03-26T04:10:10.912158+0000","last_became_active":"2024-03-26T04:08:56.866193+0000","last_became_peered":"2024-03-26T04:08:56.866193+0000","last_unstale":"2024-03-26T04:10:10.912158+0000","last_undegraded":"2024-03-26T04:10:10.912158+0000","last_fullsized":"2024-03-26T04:10:10.912158+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:54.321548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:54.321548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"448'39","reported_seq":923,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907737+0000","last_change":"2024-03-26T04:05:29.533402+0000","last_active":"2024-03-26T04:10:10.907737+0000","last_peered":"2024-03-26T04:10:10.907737+0000","last_clean":"2024-03-26T04:10:10.907737+0000","last_became_active":"2024-03-26T04:05:29.525450+0000","last_became_peered":"2024-03-26T04:05:29.525450+0000","last_unstale":"2024-03-26T04:10:10.907737+0000","last_undegraded":"2024-03-26T04:10:10.907737+0000","last_fullsized":"2024-03-26T04:10:10.907737+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:02:45.014541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:02:45.014541+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":513,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914419+0000","last_change":"2024-03-26T04:08:24.139810+0000","last_active":"2024-03-26T04:10:10.914419+0000","last_peered":"2024-03-26T04:10:10.914419+0000","last_clean":"2024-03-26T04:10:10.914419+0000","last_became_active":"2024-03-26T04:05:29.533287+0000","last_became_peered":"2024-03-26T04:05:29.533287+0000","last_unstale":"2024-03-26T04:10:10.914419+0000","last_undegraded":"2024-03-26T04:10:10.914419+0000","last_fullsized":"2024-03-26T04:10:10.914419+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:24.139706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.139706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":360,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":900,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906107+0000","last_change":"2024-03-26T04:08:56.863537+0000","last_active":"2024-03-26T04:10:10.906107+0000","last_peered":"2024-03-26T04:10:10.906107+0000","last_clean":"2024-03-26T04:10:10.906107+0000","last_became_active":"2024-03-26T04:08:56.862795+0000","last_became_peered":"2024-03-26T04:08:56.862795+0000","last_unstale":"2024-03-26T04:10:10.906107+0000","last_undegraded":"2024-03-26T04:10:10.906107+0000","last_fullsized":"2024-03-26T04:10:10.906107+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.463158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.463158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":884,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905566+0000","last_change":"2024-03-26T04:07:24.624918+0000","last_active":"2024-03-26T04:10:10.905566+0000","last_peered":"2024-03-26T04:10:10.905566+0000","last_clean":"2024-03-26T04:10:10.905566+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:10:10.905566+0000","last_undegraded":"2024-03-26T04:10:10.905566+0000","last_fullsized":"2024-03-26T04:10:10.905566+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:24.624793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:24.624793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"448'30","reported_seq":437,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912636+0000","last_change":"2024-03-26T04:07:28.446713+0000","last_active":"2024-03-26T04:10:10.912636+0000","last_peered":"2024-03-26T04:10:10.912636+0000","last_clean":"2024-03-26T04:10:10.912636+0000","last_became_active":"2024-03-26T04:05:29.521493+0000","last_became_peered":"2024-03-26T04:05:29.521493+0000","last_unstale":"2024-03-26T04:10:10.912636+0000","last_undegraded":"2024-03-26T04:10:10.912636+0000","last_fullsized":"2024-03-26T04:10:10.912636+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'24","last_scrub_stamp":"2024-03-26T04:07:28.446562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:28.446562+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":353,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.908363+0000","last_change":"2024-03-26T04:08:56.880502+0000","last_active":"2024-03-26T04:10:10.908363+0000","last_peered":"2024-03-26T04:10:10.908363+0000","last_clean":"2024-03-26T04:10:10.908363+0000","last_became_active":"2024-03-26T04:08:56.861772+0000","last_became_peered":"2024-03-26T04:08:56.861772+0000","last_unstale":"2024-03-26T04:10:10.908363+0000","last_undegraded":"2024-03-26T04:10:10.908363+0000","last_fullsized":"2024-03-26T04:10:10.908363+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:02:40.908630+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:02:40.908630+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"450'12","reported_seq":1098,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906838+0000","last_change":"2024-03-26T04:08:56.893670+0000","last_active":"2024-03-26T04:10:10.906838+0000","last_peered":"2024-03-26T04:10:10.906838+0000","last_clean":"2024-03-26T04:10:10.906838+0000","last_became_active":"2024-03-26T04:08:56.862824+0000","last_became_peered":"2024-03-26T04:08:56.862824+0000","last_unstale":"2024-03-26T04:10:10.906838+0000","last_undegraded":"2024-03-26T04:10:10.906838+0000","last_fullsized":"2024-03-26T04:10:10.906838+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'11","last_scrub_stamp":"2024-03-26T04:08:27.362944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.362944+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":922,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910214+0000","last_change":"2024-03-26T04:08:56.885318+0000","last_active":"2024-03-26T04:10:10.910214+0000","last_peered":"2024-03-26T04:10:10.910214+0000","last_clean":"2024-03-26T04:10:10.910214+0000","last_became_active":"2024-03-26T04:08:56.861744+0000","last_became_peered":"2024-03-26T04:08:56.861744+0000","last_unstale":"2024-03-26T04:10:10.910214+0000","last_undegraded":"2024-03-26T04:10:10.910214+0000","last_fullsized":"2024-03-26T04:10:10.910214+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:04:18.067462+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.067462+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"448'42","reported_seq":1016,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914250+0000","last_change":"2024-03-26T04:08:56.917670+0000","last_active":"2024-03-26T04:10:10.914250+0000","last_peered":"2024-03-26T04:10:10.914250+0000","last_clean":"2024-03-26T04:10:10.914250+0000","last_became_active":"2024-03-26T04:08:56.864036+0000","last_became_peered":"2024-03-26T04:08:56.864036+0000","last_unstale":"2024-03-26T04:10:10.914250+0000","last_undegraded":"2024-03-26T04:10:10.914250+0000","last_fullsized":"2024-03-26T04:10:10.914250+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'32","last_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:04:05.344849+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":884,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912538+0000","last_change":"2024-03-26T04:07:47.538830+0000","last_active":"2024-03-26T04:10:10.912538+0000","last_peered":"2024-03-26T04:10:10.912538+0000","last_clean":"2024-03-26T04:10:10.912538+0000","last_became_active":"2024-03-26T04:05:29.520352+0000","last_became_peered":"2024-03-26T04:05:29.520352+0000","last_unstale":"2024-03-26T04:10:10.912538+0000","last_undegraded":"2024-03-26T04:10:10.912538+0000","last_fullsized":"2024-03-26T04:10:10.912538+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:47.538728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:47.538728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":884,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914374+0000","last_change":"2024-03-26T04:08:56.864071+0000","last_active":"2024-03-26T04:10:10.914374+0000","last_peered":"2024-03-26T04:10:10.914374+0000","last_clean":"2024-03-26T04:10:10.914374+0000","last_became_active":"2024-03-26T04:08:56.863159+0000","last_became_peered":"2024-03-26T04:08:56.863159+0000","last_unstale":"2024-03-26T04:10:10.914374+0000","last_undegraded":"2024-03-26T04:10:10.914374+0000","last_fullsized":"2024-03-26T04:10:10.914374+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:29.533765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.533765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":470,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911862+0000","last_change":"2024-03-26T04:07:31.484578+0000","last_active":"2024-03-26T04:10:10.911862+0000","last_peered":"2024-03-26T04:10:10.911862+0000","last_clean":"2024-03-26T04:10:10.911862+0000","last_became_active":"2024-03-26T04:05:29.540582+0000","last_became_peered":"2024-03-26T04:05:29.540582+0000","last_unstale":"2024-03-26T04:10:10.911862+0000","last_undegraded":"2024-03-26T04:10:10.911862+0000","last_fullsized":"2024-03-26T04:10:10.911862+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:31.484355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.484355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"448'35","reported_seq":73,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914440+0000","last_change":"2024-03-26T04:08:57.325236+0000","last_active":"2024-03-26T04:10:10.914440+0000","last_peered":"2024-03-26T04:10:10.914440+0000","last_clean":"2024-03-26T04:10:10.914440+0000","last_became_active":"2024-03-26T04:08:56.871545+0000","last_became_peered":"2024-03-26T04:08:56.871545+0000","last_unstale":"2024-03-26T04:10:10.914440+0000","last_undegraded":"2024-03-26T04:10:10.914440+0000","last_fullsized":"2024-03-26T04:10:10.914440+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:01:36.230509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:01:36.230509+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":738,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905492+0000","last_change":"2024-03-26T04:08:56.860375+0000","last_active":"2024-03-26T04:10:10.905492+0000","last_peered":"2024-03-26T04:10:10.905492+0000","last_clean":"2024-03-26T04:10:10.905492+0000","last_became_active":"2024-03-26T04:08:56.859656+0000","last_became_peered":"2024-03-26T04:08:56.859656+0000","last_unstale":"2024-03-26T04:10:10.905492+0000","last_undegraded":"2024-03-26T04:10:10.905492+0000","last_fullsized":"2024-03-26T04:10:10.905492+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.011134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":812,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905046+0000","last_change":"2024-03-26T04:08:56.866943+0000","last_active":"2024-03-26T04:10:10.905046+0000","last_peered":"2024-03-26T04:10:10.905046+0000","last_clean":"2024-03-26T04:10:10.905046+0000","last_became_active":"2024-03-26T04:08:56.866332+0000","last_became_peered":"2024-03-26T04:08:56.866332+0000","last_unstale":"2024-03-26T04:10:10.905046+0000","last_undegraded":"2024-03-26T04:10:10.905046+0000","last_fullsized":"2024-03-26T04:10:10.905046+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:17.385765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.385765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":917,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913692+0000","last_change":"2024-03-26T04:07:37.490184+0000","last_active":"2024-03-26T04:10:10.913692+0000","last_peered":"2024-03-26T04:10:10.913692+0000","last_clean":"2024-03-26T04:10:10.913692+0000","last_became_active":"2024-03-26T04:05:29.538915+0000","last_became_peered":"2024-03-26T04:05:29.538915+0000","last_unstale":"2024-03-26T04:10:10.913692+0000","last_undegraded":"2024-03-26T04:10:10.913692+0000","last_fullsized":"2024-03-26T04:10:10.913692+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.490105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.490105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"448'14","reported_seq":419,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909273+0000","last_change":"2024-03-26T04:08:17.636274+0000","last_active":"2024-03-26T04:10:10.909273+0000","last_peered":"2024-03-26T04:10:10.909273+0000","last_clean":"2024-03-26T04:10:10.909273+0000","last_became_active":"2024-03-26T04:05:29.520584+0000","last_became_peered":"2024-03-26T04:05:29.520584+0000","last_unstale":"2024-03-26T04:10:10.909273+0000","last_undegraded":"2024-03-26T04:10:10.909273+0000","last_fullsized":"2024-03-26T04:10:10.909273+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'10","last_scrub_stamp":"2024-03-26T04:08:17.636015+0000","last_deep_scrub":"140'6","last_deep_scrub_stamp":"2024-03-26T03:53:32.271368+0000","last_clean_scrub_stamp":"2024-03-26T04:08:17.636015+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":536,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911473+0000","last_change":"2024-03-26T04:08:19.648443+0000","last_active":"2024-03-26T04:10:10.911473+0000","last_peered":"2024-03-26T04:10:10.911473+0000","last_clean":"2024-03-26T04:10:10.911473+0000","last_became_active":"2024-03-26T04:05:29.536699+0000","last_became_peered":"2024-03-26T04:05:29.536699+0000","last_unstale":"2024-03-26T04:10:10.911473+0000","last_undegraded":"2024-03-26T04:10:10.911473+0000","last_fullsized":"2024-03-26T04:10:10.911473+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:19.648229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:08:19.648229+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":130,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905177+0000","last_change":"2024-03-26T04:08:56.845579+0000","last_active":"2024-03-26T04:10:10.905177+0000","last_peered":"2024-03-26T04:10:10.905177+0000","last_clean":"2024-03-26T04:10:10.905177+0000","last_became_active":"2024-03-26T04:08:56.845214+0000","last_became_peered":"2024-03-26T04:08:56.845214+0000","last_unstale":"2024-03-26T04:10:10.905177+0000","last_undegraded":"2024-03-26T04:10:10.905177+0000","last_fullsized":"2024-03-26T04:10:10.905177+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:27.148413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.148413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":627,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911378+0000","last_change":"2024-03-26T04:08:56.869009+0000","last_active":"2024-03-26T04:10:10.911378+0000","last_peered":"2024-03-26T04:10:10.911378+0000","last_clean":"2024-03-26T04:10:10.911378+0000","last_became_active":"2024-03-26T04:08:56.868654+0000","last_became_peered":"2024-03-26T04:08:56.868654+0000","last_unstale":"2024-03-26T04:10:10.911378+0000","last_undegraded":"2024-03-26T04:10:10.911378+0000","last_fullsized":"2024-03-26T04:10:10.911378+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:39.423550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.423550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.11","version":"448'39","reported_seq":586,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913488+0000","last_change":"2024-03-26T04:08:56.882927+0000","last_active":"2024-03-26T04:10:10.913488+0000","last_peered":"2024-03-26T04:10:10.913488+0000","last_clean":"2024-03-26T04:10:10.913488+0000","last_became_active":"2024-03-26T04:08:56.839608+0000","last_became_peered":"2024-03-26T04:08:56.839608+0000","last_unstale":"2024-03-26T04:10:10.913488+0000","last_undegraded":"2024-03-26T04:10:10.913488+0000","last_fullsized":"2024-03-26T04:10:10.913488+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:08:24.491666+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.491666+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":885,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909259+0000","last_change":"2024-03-26T04:03:07.149962+0000","last_active":"2024-03-26T04:10:10.909259+0000","last_peered":"2024-03-26T04:10:10.909259+0000","last_clean":"2024-03-26T04:10:10.909259+0000","last_became_active":"2024-03-26T04:02:52.506113+0000","last_became_peered":"2024-03-26T04:02:52.506113+0000","last_unstale":"2024-03-26T04:10:10.909259+0000","last_undegraded":"2024-03-26T04:10:10.909259+0000","last_fullsized":"2024-03-26T04:10:10.909259+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:07.149881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.149881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":470,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911657+0000","last_change":"2024-03-26T04:04:51.740064+0000","last_active":"2024-03-26T04:10:10.911657+0000","last_peered":"2024-03-26T04:10:10.911657+0000","last_clean":"2024-03-26T04:10:10.911657+0000","last_became_active":"2024-03-26T04:02:52.489626+0000","last_became_peered":"2024-03-26T04:02:52.489626+0000","last_unstale":"2024-03-26T04:10:10.911657+0000","last_undegraded":"2024-03-26T04:10:10.911657+0000","last_fullsized":"2024-03-26T04:10:10.911657+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:51.739925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:04:51.739925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":649,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906196+0000","last_change":"2024-03-26T04:08:56.847038+0000","last_active":"2024-03-26T04:10:10.906196+0000","last_peered":"2024-03-26T04:10:10.906196+0000","last_clean":"2024-03-26T04:10:10.906196+0000","last_became_active":"2024-03-26T04:08:56.846241+0000","last_became_peered":"2024-03-26T04:08:56.846241+0000","last_unstale":"2024-03-26T04:10:10.906196+0000","last_undegraded":"2024-03-26T04:10:10.906196+0000","last_fullsized":"2024-03-26T04:10:10.906196+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.951520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.951520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"448'25","reported_seq":948,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910845+0000","last_change":"2024-03-26T04:08:27.876373+0000","last_active":"2024-03-26T04:10:10.910845+0000","last_peered":"2024-03-26T04:10:10.910845+0000","last_clean":"2024-03-26T04:10:10.910845+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:10:10.910845+0000","last_undegraded":"2024-03-26T04:10:10.910845+0000","last_fullsized":"2024-03-26T04:10:10.910845+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:08:27.876252+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.876252+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":514,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913968+0000","last_change":"2024-03-26T04:08:27.562884+0000","last_active":"2024-03-26T04:10:10.913968+0000","last_peered":"2024-03-26T04:10:10.913968+0000","last_clean":"2024-03-26T04:10:10.913968+0000","last_became_active":"2024-03-26T04:05:29.537974+0000","last_became_peered":"2024-03-26T04:05:29.537974+0000","last_unstale":"2024-03-26T04:10:10.913968+0000","last_undegraded":"2024-03-26T04:10:10.913968+0000","last_fullsized":"2024-03-26T04:10:10.913968+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:27.562766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:08:27.562766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":904,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910753+0000","last_change":"2024-03-26T04:08:56.867471+0000","last_active":"2024-03-26T04:10:10.910753+0000","last_peered":"2024-03-26T04:10:10.910753+0000","last_clean":"2024-03-26T04:10:10.910753+0000","last_became_active":"2024-03-26T04:08:56.865835+0000","last_became_peered":"2024-03-26T04:08:56.865835+0000","last_unstale":"2024-03-26T04:10:10.910753+0000","last_undegraded":"2024-03-26T04:10:10.910753+0000","last_fullsized":"2024-03-26T04:10:10.910753+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:57.362566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:57.362566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":905,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905906+0000","last_change":"2024-03-26T04:09:50.719383+0000","last_active":"2024-03-26T04:10:10.905906+0000","last_peered":"2024-03-26T04:10:10.905906+0000","last_clean":"2024-03-26T04:10:10.905906+0000","last_became_active":"2024-03-26T04:08:56.844954+0000","last_became_peered":"2024-03-26T04:08:56.844954+0000","last_unstale":"2024-03-26T04:10:10.905906+0000","last_undegraded":"2024-03-26T04:10:10.905906+0000","last_fullsized":"2024-03-26T04:10:10.905906+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:09:50.719261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:09:50.719261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:11:20.531731+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"448'27","reported_seq":969,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906018+0000","last_change":"2024-03-26T04:09:51.692089+0000","last_active":"2024-03-26T04:10:10.906018+0000","last_peered":"2024-03-26T04:10:10.906018+0000","last_clean":"2024-03-26T04:10:10.906018+0000","last_became_active":"2024-03-26T04:05:29.520817+0000","last_became_peered":"2024-03-26T04:05:29.520817+0000","last_unstale":"2024-03-26T04:10:10.906018+0000","last_undegraded":"2024-03-26T04:10:10.906018+0000","last_fullsized":"2024-03-26T04:10:10.906018+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'27","last_scrub_stamp":"2024-03-26T04:09:51.691987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:09:51.691987+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:10:59.216282+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":887,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913846+0000","last_change":"2024-03-26T04:08:56.864462+0000","last_active":"2024-03-26T04:10:10.913846+0000","last_peered":"2024-03-26T04:10:10.913846+0000","last_clean":"2024-03-26T04:10:10.913846+0000","last_became_active":"2024-03-26T04:08:56.863736+0000","last_became_peered":"2024-03-26T04:08:56.863736+0000","last_unstale":"2024-03-26T04:10:10.913846+0000","last_undegraded":"2024-03-26T04:10:10.913846+0000","last_fullsized":"2024-03-26T04:10:10.913846+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:04:58.704813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"325'12","reported_seq":562,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912937+0000","last_change":"2024-03-26T04:03:53.367019+0000","last_active":"2024-03-26T04:10:10.912937+0000","last_peered":"2024-03-26T04:10:10.912937+0000","last_clean":"2024-03-26T04:10:10.912937+0000","last_became_active":"2024-03-26T04:02:52.511718+0000","last_became_peered":"2024-03-26T04:02:52.511718+0000","last_unstale":"2024-03-26T04:10:10.912937+0000","last_undegraded":"2024-03-26T04:10:10.912937+0000","last_fullsized":"2024-03-26T04:10:10.912937+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:53.366906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:53.366906+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":750,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906016+0000","last_change":"2024-03-26T04:08:56.846589+0000","last_active":"2024-03-26T04:10:10.906016+0000","last_peered":"2024-03-26T04:10:10.906016+0000","last_clean":"2024-03-26T04:10:10.906016+0000","last_became_active":"2024-03-26T04:08:56.845018+0000","last_became_peered":"2024-03-26T04:08:56.845018+0000","last_unstale":"2024-03-26T04:10:10.906016+0000","last_undegraded":"2024-03-26T04:10:10.906016+0000","last_fullsized":"2024-03-26T04:10:10.906016+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:32.845486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:32.845486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"448'49","reported_seq":483,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910941+0000","last_change":"2024-03-26T04:08:56.918497+0000","last_active":"2024-03-26T04:10:10.910941+0000","last_peered":"2024-03-26T04:10:10.910941+0000","last_clean":"2024-03-26T04:10:10.910941+0000","last_became_active":"2024-03-26T04:08:56.871518+0000","last_became_peered":"2024-03-26T04:08:56.871518+0000","last_unstale":"2024-03-26T04:10:10.910941+0000","last_undegraded":"2024-03-26T04:10:10.910941+0000","last_fullsized":"2024-03-26T04:10:10.910941+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'39","last_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:07:42.462212+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":887,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906889+0000","last_change":"2024-03-26T04:08:29.657075+0000","last_active":"2024-03-26T04:10:10.906889+0000","last_peered":"2024-03-26T04:10:10.906889+0000","last_clean":"2024-03-26T04:10:10.906889+0000","last_became_active":"2024-03-26T04:05:29.521886+0000","last_became_peered":"2024-03-26T04:05:29.521886+0000","last_unstale":"2024-03-26T04:10:10.906889+0000","last_undegraded":"2024-03-26T04:10:10.906889+0000","last_fullsized":"2024-03-26T04:10:10.906889+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:29.656935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:29.656935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"450'15","reported_seq":346,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907683+0000","last_change":"2024-03-26T04:08:56.881240+0000","last_active":"2024-03-26T04:10:10.907683+0000","last_peered":"2024-03-26T04:10:10.907683+0000","last_clean":"2024-03-26T04:10:10.907683+0000","last_became_active":"2024-03-26T04:08:56.862068+0000","last_became_peered":"2024-03-26T04:08:56.862068+0000","last_unstale":"2024-03-26T04:10:10.907683+0000","last_undegraded":"2024-03-26T04:10:10.907683+0000","last_fullsized":"2024-03-26T04:10:10.907683+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"334'12","last_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_deep_scrub":"334'12","last_deep_scrub_stamp":"2024-03-26T04:03:07.907441+0000","last_clean_scrub_stamp":"2024-03-26T04:03:07.907441+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":938,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911970+0000","last_change":"2024-03-26T04:08:56.907028+0000","last_active":"2024-03-26T04:10:10.911970+0000","last_peered":"2024-03-26T04:10:10.911970+0000","last_clean":"2024-03-26T04:10:10.911970+0000","last_became_active":"2024-03-26T04:08:56.863922+0000","last_became_peered":"2024-03-26T04:08:56.863922+0000","last_unstale":"2024-03-26T04:10:10.911970+0000","last_undegraded":"2024-03-26T04:10:10.911970+0000","last_fullsized":"2024-03-26T04:10:10.911970+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.538770+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"448'32","reported_seq":880,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912463+0000","last_change":"2024-03-26T04:08:56.921574+0000","last_active":"2024-03-26T04:10:10.912463+0000","last_peered":"2024-03-26T04:10:10.912463+0000","last_clean":"2024-03-26T04:10:10.912463+0000","last_became_active":"2024-03-26T04:08:56.864919+0000","last_became_peered":"2024-03-26T04:08:56.864919+0000","last_unstale":"2024-03-26T04:10:10.912463+0000","last_undegraded":"2024-03-26T04:10:10.912463+0000","last_fullsized":"2024-03-26T04:10:10.912463+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:08:33.108111+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:08:33.108111+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":811,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907147+0000","last_change":"2024-03-26T04:08:56.848343+0000","last_active":"2024-03-26T04:10:10.907147+0000","last_peered":"2024-03-26T04:10:10.907147+0000","last_clean":"2024-03-26T04:10:10.907147+0000","last_became_active":"2024-03-26T04:08:56.833658+0000","last_became_peered":"2024-03-26T04:08:56.833658+0000","last_unstale":"2024-03-26T04:10:10.907147+0000","last_undegraded":"2024-03-26T04:10:10.907147+0000","last_fullsized":"2024-03-26T04:10:10.907147+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:08:21.812923+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:08:21.812923+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":466,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913921+0000","last_change":"2024-03-26T04:08:56.871548+0000","last_active":"2024-03-26T04:10:10.913921+0000","last_peered":"2024-03-26T04:10:10.913921+0000","last_clean":"2024-03-26T04:10:10.913921+0000","last_became_active":"2024-03-26T04:08:56.870098+0000","last_became_peered":"2024-03-26T04:08:56.870098+0000","last_unstale":"2024-03-26T04:10:10.913921+0000","last_undegraded":"2024-03-26T04:10:10.913921+0000","last_fullsized":"2024-03-26T04:10:10.913921+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:58.337496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:03:58.337496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":590,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905400+0000","last_change":"2024-03-26T04:08:56.847634+0000","last_active":"2024-03-26T04:10:10.905400+0000","last_peered":"2024-03-26T04:10:10.905400+0000","last_clean":"2024-03-26T04:10:10.905400+0000","last_became_active":"2024-03-26T04:08:56.845794+0000","last_became_peered":"2024-03-26T04:08:56.845794+0000","last_unstale":"2024-03-26T04:10:10.905400+0000","last_undegraded":"2024-03-26T04:10:10.905400+0000","last_fullsized":"2024-03-26T04:10:10.905400+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:57.522794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:57.522794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.16","version":"448'39","reported_seq":1024,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914276+0000","last_change":"2024-03-26T04:08:56.904108+0000","last_active":"2024-03-26T04:10:10.914276+0000","last_peered":"2024-03-26T04:10:10.914276+0000","last_clean":"2024-03-26T04:10:10.914276+0000","last_became_active":"2024-03-26T04:08:56.862596+0000","last_became_peered":"2024-03-26T04:08:56.862596+0000","last_unstale":"2024-03-26T04:10:10.914276+0000","last_undegraded":"2024-03-26T04:10:10.914276+0000","last_fullsized":"2024-03-26T04:10:10.914276+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'33","last_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.584130+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":572,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906079+0000","last_change":"2024-03-26T04:08:56.864440+0000","last_active":"2024-03-26T04:10:10.906079+0000","last_peered":"2024-03-26T04:10:10.906079+0000","last_clean":"2024-03-26T04:10:10.906079+0000","last_became_active":"2024-03-26T04:08:56.862509+0000","last_became_peered":"2024-03-26T04:08:56.862509+0000","last_unstale":"2024-03-26T04:10:10.906079+0000","last_undegraded":"2024-03-26T04:10:10.906079+0000","last_fullsized":"2024-03-26T04:10:10.906079+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:51.008927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:51.008927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":620,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911739+0000","last_change":"2024-03-26T04:08:56.868847+0000","last_active":"2024-03-26T04:10:10.911739+0000","last_peered":"2024-03-26T04:10:10.911739+0000","last_clean":"2024-03-26T04:10:10.911739+0000","last_became_active":"2024-03-26T04:08:56.868262+0000","last_became_peered":"2024-03-26T04:08:56.868262+0000","last_unstale":"2024-03-26T04:10:10.911739+0000","last_undegraded":"2024-03-26T04:10:10.911739+0000","last_fullsized":"2024-03-26T04:10:10.911739+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:04:10.223350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":856,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910267+0000","last_change":"2024-03-26T04:08:37.529848+0000","last_active":"2024-03-26T04:10:10.910267+0000","last_peered":"2024-03-26T04:10:10.910267+0000","last_clean":"2024-03-26T04:10:10.910267+0000","last_became_active":"2024-03-26T04:05:29.540070+0000","last_became_peered":"2024-03-26T04:05:29.540070+0000","last_unstale":"2024-03-26T04:10:10.910267+0000","last_undegraded":"2024-03-26T04:10:10.910267+0000","last_fullsized":"2024-03-26T04:10:10.910267+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:37.529261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:08:37.529261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"448'25","reported_seq":365,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913791+0000","last_change":"2024-03-26T04:08:56.903278+0000","last_active":"2024-03-26T04:10:10.913791+0000","last_peered":"2024-03-26T04:10:10.913791+0000","last_clean":"2024-03-26T04:10:10.913791+0000","last_became_active":"2024-03-26T04:08:56.868889+0000","last_became_peered":"2024-03-26T04:08:56.868889+0000","last_unstale":"2024-03-26T04:10:10.913791+0000","last_undegraded":"2024-03-26T04:10:10.913791+0000","last_fullsized":"2024-03-26T04:10:10.913791+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'19","last_scrub_stamp":"2024-03-26T04:07:48.515443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:48.515443+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.7","version":"474'8340","reported_seq":8444,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.567657+0000","last_change":"2024-03-26T04:08:02.634036+0000","last_active":"2024-03-26T04:10:16.567657+0000","last_peered":"2024-03-26T04:10:16.567657+0000","last_clean":"2024-03-26T04:10:16.567657+0000","last_became_active":"2024-03-26T04:08:02.633750+0000","last_became_peered":"2024-03-26T04:08:02.633750+0000","last_unstale":"2024-03-26T04:10:16.567657+0000","last_undegraded":"2024-03-26T04:10:16.567657+0000","last_fullsized":"2024-03-26T04:10:16.567657+0000","mapping_epoch":427,"log_start":"463'5911","ondisk_log_start":"463'5911","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2429,"log_dups_size":3000,"ondisk_log_size":2429,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8340,"num_write_kb":425472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":609,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912139+0000","last_change":"2024-03-26T04:08:36.535719+0000","last_active":"2024-03-26T04:10:10.912139+0000","last_peered":"2024-03-26T04:10:10.912139+0000","last_clean":"2024-03-26T04:10:10.912139+0000","last_became_active":"2024-03-26T04:05:29.537211+0000","last_became_peered":"2024-03-26T04:05:29.537211+0000","last_unstale":"2024-03-26T04:10:10.912139+0000","last_undegraded":"2024-03-26T04:10:10.912139+0000","last_fullsized":"2024-03-26T04:10:10.912139+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:36.535552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:36.535552+0000","last_clean_scrub_stamp":"2024-03-26T04:08:36.535552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"446'4","reported_seq":923,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906748+0000","last_change":"2024-03-26T04:08:56.908939+0000","last_active":"2024-03-26T04:10:10.906748+0000","last_peered":"2024-03-26T04:10:10.906748+0000","last_clean":"2024-03-26T04:10:10.906748+0000","last_became_active":"2024-03-26T04:08:56.863670+0000","last_became_peered":"2024-03-26T04:08:56.863670+0000","last_unstale":"2024-03-26T04:10:10.906748+0000","last_undegraded":"2024-03-26T04:10:10.906748+0000","last_fullsized":"2024-03-26T04:10:10.906748+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'4","last_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_deep_scrub":"446'4","last_deep_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.098553+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.8","version":"448'51","reported_seq":930,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.904923+0000","last_change":"2024-03-26T04:08:56.897366+0000","last_active":"2024-03-26T04:10:10.904923+0000","last_peered":"2024-03-26T04:10:10.904923+0000","last_clean":"2024-03-26T04:10:10.904923+0000","last_became_active":"2024-03-26T04:08:56.862064+0000","last_became_peered":"2024-03-26T04:08:56.862064+0000","last_unstale":"2024-03-26T04:10:10.904923+0000","last_undegraded":"2024-03-26T04:10:10.904923+0000","last_fullsized":"2024-03-26T04:10:10.904923+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'39","last_scrub_stamp":"2024-03-26T04:08:37.152100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:37.152100+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"450'10","reported_seq":106,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905097+0000","last_change":"2024-03-26T04:08:56.918039+0000","last_active":"2024-03-26T04:10:10.905097+0000","last_peered":"2024-03-26T04:10:10.905097+0000","last_clean":"2024-03-26T04:10:10.905097+0000","last_became_active":"2024-03-26T04:08:56.864436+0000","last_became_peered":"2024-03-26T04:08:56.864436+0000","last_unstale":"2024-03-26T04:10:10.905097+0000","last_undegraded":"2024-03-26T04:10:10.905097+0000","last_fullsized":"2024-03-26T04:10:10.905097+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"100'9","last_scrub_stamp":"2024-03-26T04:07:54.941219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.941219+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.6","version":"474'8190","reported_seq":8294,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.853762+0000","last_change":"2024-03-26T04:08:02.632658+0000","last_active":"2024-03-26T04:10:16.853762+0000","last_peered":"2024-03-26T04:10:16.853762+0000","last_clean":"2024-03-26T04:10:16.853762+0000","last_became_active":"2024-03-26T04:08:02.631983+0000","last_became_peered":"2024-03-26T04:08:02.631983+0000","last_unstale":"2024-03-26T04:10:16.853762+0000","last_undegraded":"2024-03-26T04:10:16.853762+0000","last_fullsized":"2024-03-26T04:10:16.853762+0000","mapping_epoch":427,"log_start":"461'5590","ondisk_log_start":"461'5590","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2600,"log_dups_size":3000,"ondisk_log_size":2600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323354624,"num_objects":4934,"num_object_clones":0,"num_object_copies":14802,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4934,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8190,"num_write_kb":419968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":201,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910044+0000","last_change":"2024-03-26T04:08:56.863588+0000","last_active":"2024-03-26T04:10:10.910044+0000","last_peered":"2024-03-26T04:10:10.910044+0000","last_clean":"2024-03-26T04:10:10.910044+0000","last_became_active":"2024-03-26T04:08:56.863164+0000","last_became_peered":"2024-03-26T04:08:56.863164+0000","last_unstale":"2024-03-26T04:10:10.910044+0000","last_undegraded":"2024-03-26T04:10:10.910044+0000","last_fullsized":"2024-03-26T04:10:10.910044+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:20.512660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:08:20.512660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":60,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906622+0000","last_change":"2024-03-26T04:08:56.867356+0000","last_active":"2024-03-26T04:10:10.906622+0000","last_peered":"2024-03-26T04:10:10.906622+0000","last_clean":"2024-03-26T04:10:10.906622+0000","last_became_active":"2024-03-26T04:08:56.865701+0000","last_became_peered":"2024-03-26T04:08:56.865701+0000","last_unstale":"2024-03-26T04:10:10.906622+0000","last_undegraded":"2024-03-26T04:10:10.906622+0000","last_fullsized":"2024-03-26T04:10:10.906622+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_clean_scrub_stamp":"2024-03-26T04:08:34.104519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"448'36","reported_seq":1003,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905549+0000","last_change":"2024-03-26T04:04:19.443761+0000","last_active":"2024-03-26T04:10:10.905549+0000","last_peered":"2024-03-26T04:10:10.905549+0000","last_clean":"2024-03-26T04:10:10.905549+0000","last_became_active":"2024-03-26T04:02:52.513309+0000","last_became_peered":"2024-03-26T04:02:52.513309+0000","last_unstale":"2024-03-26T04:10:10.905549+0000","last_undegraded":"2024-03-26T04:10:10.905549+0000","last_fullsized":"2024-03-26T04:10:10.905549+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'30","last_scrub_stamp":"2024-03-26T04:04:19.443608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.443608+0000","objects_scrubbed":7,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":803,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.904933+0000","last_change":"2024-03-26T04:07:53.660073+0000","last_active":"2024-03-26T04:10:10.904933+0000","last_peered":"2024-03-26T04:10:10.904933+0000","last_clean":"2024-03-26T04:10:10.904933+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:10:10.904933+0000","last_undegraded":"2024-03-26T04:10:10.904933+0000","last_fullsized":"2024-03-26T04:10:10.904933+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:53.659974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:53.659974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"474'8430","reported_seq":19423,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.852548+0000","last_change":"2024-03-26T04:10:09.173583+0000","last_active":"2024-03-26T04:10:16.852548+0000","last_peered":"2024-03-26T04:10:16.852548+0000","last_clean":"2024-03-26T04:10:16.852548+0000","last_became_active":"2024-03-26T04:10:09.150504+0000","last_became_peered":"2024-03-26T04:10:09.150504+0000","last_unstale":"2024-03-26T04:10:16.852548+0000","last_undegraded":"2024-03-26T04:10:16.852548+0000","last_fullsized":"2024-03-26T04:10:16.852548+0000","mapping_epoch":471,"log_start":"463'5822","ondisk_log_start":"463'5822","created":427,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2608,"log_dups_size":3000,"ondisk_log_size":2608,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324665344,"num_objects":4954,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8430,"num_write_kb":428288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5439,"num_bytes_recovered":356450304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":756,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905268+0000","last_change":"2024-03-26T04:08:56.856822+0000","last_active":"2024-03-26T04:10:10.905268+0000","last_peered":"2024-03-26T04:10:10.905268+0000","last_clean":"2024-03-26T04:10:10.905268+0000","last_became_active":"2024-03-26T04:08:56.855035+0000","last_became_peered":"2024-03-26T04:08:56.855035+0000","last_unstale":"2024-03-26T04:10:10.905268+0000","last_undegraded":"2024-03-26T04:10:10.905268+0000","last_fullsized":"2024-03-26T04:10:10.905268+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:37.891864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:02:37.891864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":802,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905769+0000","last_change":"2024-03-26T04:08:56.849260+0000","last_active":"2024-03-26T04:10:10.905769+0000","last_peered":"2024-03-26T04:10:10.905769+0000","last_clean":"2024-03-26T04:10:10.905769+0000","last_became_active":"2024-03-26T04:08:56.848815+0000","last_became_peered":"2024-03-26T04:08:56.848815+0000","last_unstale":"2024-03-26T04:10:10.905769+0000","last_undegraded":"2024-03-26T04:10:10.905769+0000","last_fullsized":"2024-03-26T04:10:10.905769+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:47.965300+0000","last_clean_scrub_stamp":"2024-03-26T04:02:47.965300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":689,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905822+0000","last_change":"2024-03-26T04:08:56.848363+0000","last_active":"2024-03-26T04:10:10.905822+0000","last_peered":"2024-03-26T04:10:10.905822+0000","last_clean":"2024-03-26T04:10:10.905822+0000","last_became_active":"2024-03-26T04:08:56.847365+0000","last_became_peered":"2024-03-26T04:08:56.847365+0000","last_unstale":"2024-03-26T04:10:10.905822+0000","last_undegraded":"2024-03-26T04:10:10.905822+0000","last_fullsized":"2024-03-26T04:10:10.905822+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:14.382655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:14.382655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.a","version":"448'67","reported_seq":1019,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910577+0000","last_change":"2024-03-26T04:08:56.893591+0000","last_active":"2024-03-26T04:10:10.910577+0000","last_peered":"2024-03-26T04:10:10.910577+0000","last_clean":"2024-03-26T04:10:10.910577+0000","last_became_active":"2024-03-26T04:08:56.867218+0000","last_became_peered":"2024-03-26T04:08:56.867218+0000","last_unstale":"2024-03-26T04:10:10.910577+0000","last_undegraded":"2024-03-26T04:10:10.910577+0000","last_fullsized":"2024-03-26T04:10:10.910577+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'51","last_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.371335+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.4","version":"474'8402","reported_seq":19359,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:18.372765+0000","last_change":"2024-03-26T04:10:06.895107+0000","last_active":"2024-03-26T04:10:18.372765+0000","last_peered":"2024-03-26T04:10:18.372765+0000","last_clean":"2024-03-26T04:10:18.372765+0000","last_became_active":"2024-03-26T04:10:06.884538+0000","last_became_peered":"2024-03-26T04:10:06.884538+0000","last_unstale":"2024-03-26T04:10:18.372765+0000","last_undegraded":"2024-03-26T04:10:18.372765+0000","last_fullsized":"2024-03-26T04:10:18.372765+0000","mapping_epoch":469,"log_start":"463'5815","ondisk_log_start":"463'5815","created":427,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2587,"log_dups_size":3000,"ondisk_log_size":2587,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":318242816,"num_objects":4856,"num_object_clones":0,"num_object_copies":14568,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8402,"num_write_kb":424256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5420,"num_bytes_recovered":355205120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":879,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914102+0000","last_change":"2024-03-26T04:08:56.867601+0000","last_active":"2024-03-26T04:10:10.914102+0000","last_peered":"2024-03-26T04:10:10.914102+0000","last_clean":"2024-03-26T04:10:10.914102+0000","last_became_active":"2024-03-26T04:08:56.866823+0000","last_became_peered":"2024-03-26T04:08:56.866823+0000","last_unstale":"2024-03-26T04:10:10.914102+0000","last_undegraded":"2024-03-26T04:10:10.914102+0000","last_fullsized":"2024-03-26T04:10:10.914102+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:17.661725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:17.661725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":812,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906083+0000","last_change":"2024-03-26T04:07:37.592031+0000","last_active":"2024-03-26T04:10:10.906083+0000","last_peered":"2024-03-26T04:10:10.906083+0000","last_clean":"2024-03-26T04:10:10.906083+0000","last_became_active":"2024-03-26T04:05:29.524644+0000","last_became_peered":"2024-03-26T04:05:29.524644+0000","last_unstale":"2024-03-26T04:10:10.906083+0000","last_undegraded":"2024-03-26T04:10:10.906083+0000","last_fullsized":"2024-03-26T04:10:10.906083+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:37.591946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:07:37.591946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":645,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911661+0000","last_change":"2024-03-26T04:07:34.405175+0000","last_active":"2024-03-26T04:10:10.911661+0000","last_peered":"2024-03-26T04:10:10.911661+0000","last_clean":"2024-03-26T04:10:10.911661+0000","last_became_active":"2024-03-26T04:05:29.531951+0000","last_became_peered":"2024-03-26T04:05:29.531951+0000","last_unstale":"2024-03-26T04:10:10.911661+0000","last_undegraded":"2024-03-26T04:10:10.911661+0000","last_fullsized":"2024-03-26T04:10:10.911661+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.404972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.404972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"448'27","reported_seq":971,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912534+0000","last_change":"2024-03-26T04:04:55.743092+0000","last_active":"2024-03-26T04:10:10.912534+0000","last_peered":"2024-03-26T04:10:10.912534+0000","last_clean":"2024-03-26T04:10:10.912534+0000","last_became_active":"2024-03-26T04:01:35.803422+0000","last_became_peered":"2024-03-26T04:01:35.803422+0000","last_unstale":"2024-03-26T04:10:10.912534+0000","last_undegraded":"2024-03-26T04:10:10.912534+0000","last_fullsized":"2024-03-26T04:10:10.912534+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'21","last_scrub_stamp":"2024-03-26T04:04:55.742967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:55.742967+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"474'8273","reported_seq":8376,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.575687+0000","last_change":"2024-03-26T04:08:02.633493+0000","last_active":"2024-03-26T04:10:16.575687+0000","last_peered":"2024-03-26T04:10:16.575687+0000","last_clean":"2024-03-26T04:10:16.575687+0000","last_became_active":"2024-03-26T04:08:02.633214+0000","last_became_peered":"2024-03-26T04:08:02.633214+0000","last_unstale":"2024-03-26T04:10:16.575687+0000","last_undegraded":"2024-03-26T04:10:16.575687+0000","last_fullsized":"2024-03-26T04:10:16.575687+0000","mapping_epoch":427,"log_start":"463'5884","ondisk_log_start":"463'5884","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2389,"log_dups_size":3000,"ondisk_log_size":2389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":325779456,"num_objects":4971,"num_object_clones":0,"num_object_copies":14913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8273,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":695,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913602+0000","last_change":"2024-03-26T04:08:28.514002+0000","last_active":"2024-03-26T04:10:10.913602+0000","last_peered":"2024-03-26T04:10:10.913602+0000","last_clean":"2024-03-26T04:10:10.913602+0000","last_became_active":"2024-03-26T04:05:29.538433+0000","last_became_peered":"2024-03-26T04:05:29.538433+0000","last_unstale":"2024-03-26T04:10:10.913602+0000","last_undegraded":"2024-03-26T04:10:10.913602+0000","last_fullsized":"2024-03-26T04:10:10.913602+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:08:28.513772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:08:28.513772+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":477,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906609+0000","last_change":"2024-03-26T04:08:56.865114+0000","last_active":"2024-03-26T04:10:10.906609+0000","last_peered":"2024-03-26T04:10:10.906609+0000","last_clean":"2024-03-26T04:10:10.906609+0000","last_became_active":"2024-03-26T04:08:56.862781+0000","last_became_peered":"2024-03-26T04:08:56.862781+0000","last_unstale":"2024-03-26T04:10:10.906609+0000","last_undegraded":"2024-03-26T04:10:10.906609+0000","last_fullsized":"2024-03-26T04:10:10.906609+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:03.264133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:04:03.264133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":780,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910464+0000","last_change":"2024-03-26T04:08:56.894926+0000","last_active":"2024-03-26T04:10:10.910464+0000","last_peered":"2024-03-26T04:10:10.910464+0000","last_clean":"2024-03-26T04:10:10.910464+0000","last_became_active":"2024-03-26T04:08:56.867760+0000","last_became_peered":"2024-03-26T04:08:56.867760+0000","last_unstale":"2024-03-26T04:10:10.910464+0000","last_undegraded":"2024-03-26T04:10:10.910464+0000","last_fullsized":"2024-03-26T04:10:10.910464+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:04:17.110315+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:04:17.110315+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":883,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905655+0000","last_change":"2024-03-26T04:04:52.483535+0000","last_active":"2024-03-26T04:10:10.905655+0000","last_peered":"2024-03-26T04:10:10.905655+0000","last_clean":"2024-03-26T04:10:10.905655+0000","last_became_active":"2024-03-26T03:50:45.593662+0000","last_became_peered":"2024-03-26T03:50:45.593662+0000","last_unstale":"2024-03-26T04:10:10.905655+0000","last_undegraded":"2024-03-26T04:10:10.905655+0000","last_fullsized":"2024-03-26T04:10:10.905655+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:52.483429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:04:52.483429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"450'64","reported_seq":1365,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911113+0000","last_change":"2024-03-26T04:08:56.924513+0000","last_active":"2024-03-26T04:10:10.911113+0000","last_peered":"2024-03-26T04:10:10.911113+0000","last_clean":"2024-03-26T04:10:10.911113+0000","last_became_active":"2024-03-26T04:08:56.872084+0000","last_became_peered":"2024-03-26T04:08:56.872084+0000","last_unstale":"2024-03-26T04:10:10.911113+0000","last_undegraded":"2024-03-26T04:10:10.911113+0000","last_fullsized":"2024-03-26T04:10:10.911113+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'51","last_scrub_stamp":"2024-03-26T04:08:24.824658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.824658+0000","objects_scrubbed":11,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":877,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912566+0000","last_change":"2024-03-26T04:05:04.831962+0000","last_active":"2024-03-26T04:10:10.912566+0000","last_peered":"2024-03-26T04:10:10.912566+0000","last_clean":"2024-03-26T04:10:10.912566+0000","last_became_active":"2024-03-26T03:48:48.023646+0000","last_became_peered":"2024-03-26T03:48:48.023646+0000","last_unstale":"2024-03-26T04:10:10.912566+0000","last_undegraded":"2024-03-26T04:10:10.912566+0000","last_fullsized":"2024-03-26T04:10:10.912566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:04.831457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:05:04.831457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"474'8295","reported_seq":8400,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.851888+0000","last_change":"2024-03-26T04:08:02.632925+0000","last_active":"2024-03-26T04:10:16.851888+0000","last_peered":"2024-03-26T04:10:16.851888+0000","last_clean":"2024-03-26T04:10:16.851888+0000","last_became_active":"2024-03-26T04:08:02.631353+0000","last_became_peered":"2024-03-26T04:08:02.631353+0000","last_unstale":"2024-03-26T04:10:16.851888+0000","last_undegraded":"2024-03-26T04:10:16.851888+0000","last_fullsized":"2024-03-26T04:10:16.851888+0000","mapping_epoch":427,"log_start":"463'5716","ondisk_log_start":"463'5716","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2579,"log_dups_size":3000,"ondisk_log_size":2579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314769408,"num_objects":4803,"num_object_clones":0,"num_object_copies":14409,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4803,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8295,"num_write_kb":419136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":728,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911692+0000","last_change":"2024-03-26T04:08:59.851553+0000","last_active":"2024-03-26T04:10:10.911692+0000","last_peered":"2024-03-26T04:10:10.911692+0000","last_clean":"2024-03-26T04:10:10.911692+0000","last_became_active":"2024-03-26T04:08:59.851169+0000","last_became_peered":"2024-03-26T04:08:59.851169+0000","last_unstale":"2024-03-26T04:10:10.911692+0000","last_undegraded":"2024-03-26T04:10:10.911692+0000","last_fullsized":"2024-03-26T04:10:10.911692+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:11.458734+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:04:11.458734+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":763,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907852+0000","last_change":"2024-03-26T04:08:56.855459+0000","last_active":"2024-03-26T04:10:10.907852+0000","last_peered":"2024-03-26T04:10:10.907852+0000","last_clean":"2024-03-26T04:10:10.907852+0000","last_became_active":"2024-03-26T04:08:56.853944+0000","last_became_peered":"2024-03-26T04:08:56.853944+0000","last_unstale":"2024-03-26T04:10:10.907852+0000","last_undegraded":"2024-03-26T04:10:10.907852+0000","last_fullsized":"2024-03-26T04:10:10.907852+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:07:38.023066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4],"acting":[2,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":905,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909328+0000","last_change":"2024-03-26T04:08:29.947935+0000","last_active":"2024-03-26T04:10:10.909328+0000","last_peered":"2024-03-26T04:10:10.909328+0000","last_clean":"2024-03-26T04:10:10.909328+0000","last_became_active":"2024-03-26T04:05:29.533776+0000","last_became_peered":"2024-03-26T04:05:29.533776+0000","last_unstale":"2024-03-26T04:10:10.909328+0000","last_undegraded":"2024-03-26T04:10:10.909328+0000","last_fullsized":"2024-03-26T04:10:10.909328+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:29.947803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:08:29.947803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":829,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905806+0000","last_change":"2024-03-26T04:08:24.797749+0000","last_active":"2024-03-26T04:10:10.905806+0000","last_peered":"2024-03-26T04:10:10.905806+0000","last_clean":"2024-03-26T04:10:10.905806+0000","last_became_active":"2024-03-26T04:05:29.543967+0000","last_became_peered":"2024-03-26T04:05:29.543967+0000","last_unstale":"2024-03-26T04:10:10.905806+0000","last_undegraded":"2024-03-26T04:10:10.905806+0000","last_fullsized":"2024-03-26T04:10:10.905806+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:24.797631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:08:24.797631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"448'33","reported_seq":983,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914151+0000","last_change":"2024-03-26T04:08:56.913082+0000","last_active":"2024-03-26T04:10:10.914151+0000","last_peered":"2024-03-26T04:10:10.914151+0000","last_clean":"2024-03-26T04:10:10.914151+0000","last_became_active":"2024-03-26T04:08:56.870173+0000","last_became_peered":"2024-03-26T04:08:56.870173+0000","last_unstale":"2024-03-26T04:10:10.914151+0000","last_undegraded":"2024-03-26T04:10:10.914151+0000","last_fullsized":"2024-03-26T04:10:10.914151+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"371'29","last_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:04:59.217868+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":829,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:11.112666+0000","last_change":"2024-03-26T04:10:11.112701+0000","last_active":"2024-03-26T04:10:11.112666+0000","last_peered":"2024-03-26T04:10:11.112666+0000","last_clean":"2024-03-26T04:10:11.112666+0000","last_became_active":"2024-03-26T04:08:56.854146+0000","last_became_peered":"2024-03-26T04:08:56.854146+0000","last_unstale":"2024-03-26T04:10:11.112666+0000","last_undegraded":"2024-03-26T04:10:11.112666+0000","last_fullsized":"2024-03-26T04:10:11.112666+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:10:11.112591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:10:11.112591+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:11:23.902840+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"474'8198","reported_seq":8301,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.580044+0000","last_change":"2024-03-26T04:08:02.629892+0000","last_active":"2024-03-26T04:10:16.580044+0000","last_peered":"2024-03-26T04:10:16.580044+0000","last_clean":"2024-03-26T04:10:16.580044+0000","last_became_active":"2024-03-26T04:08:02.629608+0000","last_became_peered":"2024-03-26T04:08:02.629608+0000","last_unstale":"2024-03-26T04:10:16.580044+0000","last_undegraded":"2024-03-26T04:10:16.580044+0000","last_fullsized":"2024-03-26T04:10:16.580044+0000","mapping_epoch":427,"log_start":"463'5766","ondisk_log_start":"463'5766","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2432,"log_dups_size":3000,"ondisk_log_size":2432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319684608,"num_objects":4878,"num_object_clones":0,"num_object_copies":14634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8198,"num_write_kb":418432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":617,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906634+0000","last_change":"2024-03-26T04:08:56.866030+0000","last_active":"2024-03-26T04:10:10.906634+0000","last_peered":"2024-03-26T04:10:10.906634+0000","last_clean":"2024-03-26T04:10:10.906634+0000","last_became_active":"2024-03-26T04:08:56.862727+0000","last_became_peered":"2024-03-26T04:08:56.862727+0000","last_unstale":"2024-03-26T04:10:10.906634+0000","last_undegraded":"2024-03-26T04:10:10.906634+0000","last_fullsized":"2024-03-26T04:10:10.906634+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:04:02.403784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":346,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909467+0000","last_change":"2024-03-26T04:08:59.853118+0000","last_active":"2024-03-26T04:10:10.909467+0000","last_peered":"2024-03-26T04:10:10.909467+0000","last_clean":"2024-03-26T04:10:10.909467+0000","last_became_active":"2024-03-26T04:08:59.852745+0000","last_became_peered":"2024-03-26T04:08:59.852745+0000","last_unstale":"2024-03-26T04:10:10.909467+0000","last_undegraded":"2024-03-26T04:10:10.909467+0000","last_fullsized":"2024-03-26T04:10:10.909467+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:08:16.158354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:08:16.158354+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:10:15.446857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":161009,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":511,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905211+0000","last_change":"2024-03-26T04:08:56.866956+0000","last_active":"2024-03-26T04:10:10.905211+0000","last_peered":"2024-03-26T04:10:10.905211+0000","last_clean":"2024-03-26T04:10:10.905211+0000","last_became_active":"2024-03-26T04:08:56.866010+0000","last_became_peered":"2024-03-26T04:08:56.866010+0000","last_unstale":"2024-03-26T04:10:10.905211+0000","last_undegraded":"2024-03-26T04:10:10.905211+0000","last_fullsized":"2024-03-26T04:10:10.905211+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:56.545464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:56.545464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.5","version":"450'42","reported_seq":1163,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906269+0000","last_change":"2024-03-26T04:08:56.916941+0000","last_active":"2024-03-26T04:10:10.906269+0000","last_peered":"2024-03-26T04:10:10.906269+0000","last_clean":"2024-03-26T04:10:10.906269+0000","last_became_active":"2024-03-26T04:08:56.863726+0000","last_became_peered":"2024-03-26T04:08:56.863726+0000","last_unstale":"2024-03-26T04:10:10.906269+0000","last_undegraded":"2024-03-26T04:10:10.906269+0000","last_fullsized":"2024-03-26T04:10:10.906269+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'29","last_scrub_stamp":"2024-03-26T04:04:20.432551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:20.432551+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":711,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907267+0000","last_change":"2024-03-26T04:08:56.856134+0000","last_active":"2024-03-26T04:10:10.907267+0000","last_peered":"2024-03-26T04:10:10.907267+0000","last_clean":"2024-03-26T04:10:10.907267+0000","last_became_active":"2024-03-26T04:08:56.855807+0000","last_became_peered":"2024-03-26T04:08:56.855807+0000","last_unstale":"2024-03-26T04:10:10.907267+0000","last_undegraded":"2024-03-26T04:10:10.907267+0000","last_fullsized":"2024-03-26T04:10:10.907267+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:25.836032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:00.772625+0000","last_clean_scrub_stamp":"2024-03-26T04:08:25.836032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.9","version":"474'8256","reported_seq":19157,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.829815+0000","last_change":"2024-03-26T04:10:09.921192+0000","last_active":"2024-03-26T04:10:16.829815+0000","last_peered":"2024-03-26T04:10:16.829815+0000","last_clean":"2024-03-26T04:10:16.829815+0000","last_became_active":"2024-03-26T04:10:09.911586+0000","last_became_peered":"2024-03-26T04:10:09.911586+0000","last_unstale":"2024-03-26T04:10:16.829815+0000","last_undegraded":"2024-03-26T04:10:16.829815+0000","last_fullsized":"2024-03-26T04:10:16.829815+0000","mapping_epoch":472,"log_start":"462'5673","ondisk_log_start":"462'5673","created":427,"last_epoch_clean":473,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2583,"log_dups_size":3000,"ondisk_log_size":2583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321388544,"num_objects":4904,"num_object_clones":0,"num_object_copies":14712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8256,"num_write_kb":421120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5391,"num_bytes_recovered":353304576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":772,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910651+0000","last_change":"2024-03-26T04:04:07.285956+0000","last_active":"2024-03-26T04:10:10.910651+0000","last_peered":"2024-03-26T04:10:10.910651+0000","last_clean":"2024-03-26T04:10:10.910651+0000","last_became_active":"2024-03-26T04:01:38.811943+0000","last_became_peered":"2024-03-26T04:01:38.811943+0000","last_unstale":"2024-03-26T04:10:10.910651+0000","last_undegraded":"2024-03-26T04:10:10.910651+0000","last_fullsized":"2024-03-26T04:10:10.910651+0000","mapping_epoch":296,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":297,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:07.285700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:07.285700+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":904,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911566+0000","last_change":"2024-03-26T04:07:49.506391+0000","last_active":"2024-03-26T04:10:10.911566+0000","last_peered":"2024-03-26T04:10:10.911566+0000","last_clean":"2024-03-26T04:10:10.911566+0000","last_became_active":"2024-03-26T04:05:29.521425+0000","last_became_peered":"2024-03-26T04:05:29.521425+0000","last_unstale":"2024-03-26T04:10:10.911566+0000","last_undegraded":"2024-03-26T04:10:10.911566+0000","last_fullsized":"2024-03-26T04:10:10.911566+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:49.506266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.506266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":918,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913858+0000","last_change":"2024-03-26T04:08:56.867840+0000","last_active":"2024-03-26T04:10:10.913858+0000","last_peered":"2024-03-26T04:10:10.913858+0000","last_clean":"2024-03-26T04:10:10.913858+0000","last_became_active":"2024-03-26T04:08:56.867263+0000","last_became_peered":"2024-03-26T04:08:56.867263+0000","last_unstale":"2024-03-26T04:10:10.913858+0000","last_undegraded":"2024-03-26T04:10:10.913858+0000","last_fullsized":"2024-03-26T04:10:10.913858+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:56.329511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:56.329511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":186,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:14.661081+0000","last_change":"2024-03-26T04:08:56.883694+0000","last_active":"2024-03-26T04:10:14.661081+0000","last_peered":"2024-03-26T04:10:14.661081+0000","last_clean":"2024-03-26T04:10:14.661081+0000","last_became_active":"2024-03-26T04:08:56.867031+0000","last_became_peered":"2024-03-26T04:08:56.867031+0000","last_unstale":"2024-03-26T04:10:14.661081+0000","last_undegraded":"2024-03-26T04:10:14.661081+0000","last_fullsized":"2024-03-26T04:10:14.661081+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-26T04:07:31.850790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:41.993340+0000","last_clean_scrub_stamp":"2024-03-26T04:07:31.850790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"448'42","reported_seq":941,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914046+0000","last_change":"2024-03-26T04:07:49.546831+0000","last_active":"2024-03-26T04:10:10.914046+0000","last_peered":"2024-03-26T04:10:10.914046+0000","last_clean":"2024-03-26T04:10:10.914046+0000","last_became_active":"2024-03-26T04:05:29.538986+0000","last_became_peered":"2024-03-26T04:05:29.538986+0000","last_unstale":"2024-03-26T04:10:10.914046+0000","last_undegraded":"2024-03-26T04:10:10.914046+0000","last_fullsized":"2024-03-26T04:10:10.914046+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"422'30","last_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_deep_scrub":"422'30","last_deep_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_clean_scrub_stamp":"2024-03-26T04:07:49.546733+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":589,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913237+0000","last_change":"2024-03-26T04:05:01.160620+0000","last_active":"2024-03-26T04:10:10.913237+0000","last_peered":"2024-03-26T04:10:10.913237+0000","last_clean":"2024-03-26T04:10:10.913237+0000","last_became_active":"2024-03-26T04:01:35.802991+0000","last_became_peered":"2024-03-26T04:01:35.802991+0000","last_unstale":"2024-03-26T04:10:10.913237+0000","last_undegraded":"2024-03-26T04:10:10.913237+0000","last_fullsized":"2024-03-26T04:10:10.913237+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:05:01.160427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"474'8207","reported_seq":8310,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.567278+0000","last_change":"2024-03-26T04:08:02.631311+0000","last_active":"2024-03-26T04:10:16.567278+0000","last_peered":"2024-03-26T04:10:16.567278+0000","last_clean":"2024-03-26T04:10:16.567278+0000","last_became_active":"2024-03-26T04:08:02.630915+0000","last_became_peered":"2024-03-26T04:08:02.630915+0000","last_unstale":"2024-03-26T04:10:16.567278+0000","last_undegraded":"2024-03-26T04:10:16.567278+0000","last_fullsized":"2024-03-26T04:10:16.567278+0000","mapping_epoch":427,"log_start":"463'5745","ondisk_log_start":"463'5745","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2462,"log_dups_size":3000,"ondisk_log_size":2462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316604416,"num_objects":4831,"num_object_clones":0,"num_object_copies":14493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8207,"num_write_kb":417216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":737,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913742+0000","last_change":"2024-03-26T04:08:59.860837+0000","last_active":"2024-03-26T04:10:10.913742+0000","last_peered":"2024-03-26T04:10:10.913742+0000","last_clean":"2024-03-26T04:10:10.913742+0000","last_became_active":"2024-03-26T04:08:59.860447+0000","last_became_peered":"2024-03-26T04:08:59.860447+0000","last_unstale":"2024-03-26T04:10:10.913742+0000","last_undegraded":"2024-03-26T04:10:10.913742+0000","last_fullsized":"2024-03-26T04:10:10.913742+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:04:53.785762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:04:53.785762+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":891,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910418+0000","last_change":"2024-03-26T04:08:56.868416+0000","last_active":"2024-03-26T04:10:10.910418+0000","last_peered":"2024-03-26T04:10:10.910418+0000","last_clean":"2024-03-26T04:10:10.910418+0000","last_became_active":"2024-03-26T04:08:56.866371+0000","last_became_peered":"2024-03-26T04:08:56.866371+0000","last_unstale":"2024-03-26T04:10:10.910418+0000","last_undegraded":"2024-03-26T04:10:10.910418+0000","last_fullsized":"2024-03-26T04:10:10.910418+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:41.426604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:35.502151+0000","last_clean_scrub_stamp":"2024-03-26T04:07:41.426604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":906,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911305+0000","last_change":"2024-03-26T04:07:45.465265+0000","last_active":"2024-03-26T04:10:10.911305+0000","last_peered":"2024-03-26T04:10:10.911305+0000","last_clean":"2024-03-26T04:10:10.911305+0000","last_became_active":"2024-03-26T04:05:29.532029+0000","last_became_peered":"2024-03-26T04:05:29.532029+0000","last_unstale":"2024-03-26T04:10:10.911305+0000","last_undegraded":"2024-03-26T04:10:10.911305+0000","last_fullsized":"2024-03-26T04:10:10.911305+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.465179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.465179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"448'54","reported_seq":1055,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912252+0000","last_change":"2024-03-26T04:08:56.885200+0000","last_active":"2024-03-26T04:10:10.912252+0000","last_peered":"2024-03-26T04:10:10.912252+0000","last_clean":"2024-03-26T04:10:10.912252+0000","last_became_active":"2024-03-26T04:08:56.845070+0000","last_became_peered":"2024-03-26T04:08:56.845070+0000","last_unstale":"2024-03-26T04:10:10.912252+0000","last_undegraded":"2024-03-26T04:10:10.912252+0000","last_fullsized":"2024-03-26T04:10:10.912252+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"358'44","last_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_deep_scrub":"358'44","last_deep_scrub_stamp":"2024-03-26T04:04:19.571532+0000","last_clean_scrub_stamp":"2024-03-26T04:04:19.571532+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":845,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911876+0000","last_change":"2024-03-26T04:08:56.841172+0000","last_active":"2024-03-26T04:10:10.911876+0000","last_peered":"2024-03-26T04:10:10.911876+0000","last_clean":"2024-03-26T04:10:10.911876+0000","last_became_active":"2024-03-26T04:08:56.840828+0000","last_became_peered":"2024-03-26T04:08:56.840828+0000","last_unstale":"2024-03-26T04:10:10.911876+0000","last_undegraded":"2024-03-26T04:10:10.911876+0000","last_fullsized":"2024-03-26T04:10:10.911876+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_clean_scrub_stamp":"2024-03-26T04:08:34.576640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.d","version":"474'8405","reported_seq":8509,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:18.357065+0000","last_change":"2024-03-26T04:08:02.629735+0000","last_active":"2024-03-26T04:10:18.357065+0000","last_peered":"2024-03-26T04:10:18.357065+0000","last_clean":"2024-03-26T04:10:18.357065+0000","last_became_active":"2024-03-26T04:08:02.629370+0000","last_became_peered":"2024-03-26T04:08:02.629370+0000","last_unstale":"2024-03-26T04:10:18.357065+0000","last_undegraded":"2024-03-26T04:10:18.357065+0000","last_fullsized":"2024-03-26T04:10:18.357065+0000","mapping_epoch":427,"log_start":"463'5830","ondisk_log_start":"463'5830","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2575,"log_dups_size":3000,"ondisk_log_size":2575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":317456408,"num_objects":4845,"num_object_clones":0,"num_object_copies":14535,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4845,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8405,"num_write_kb":423937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":583,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906395+0000","last_change":"2024-03-26T04:08:59.865819+0000","last_active":"2024-03-26T04:10:10.906395+0000","last_peered":"2024-03-26T04:10:10.906395+0000","last_clean":"2024-03-26T04:10:10.906395+0000","last_became_active":"2024-03-26T04:08:59.865397+0000","last_became_peered":"2024-03-26T04:08:59.865397+0000","last_unstale":"2024-03-26T04:10:10.906395+0000","last_undegraded":"2024-03-26T04:10:10.906395+0000","last_fullsized":"2024-03-26T04:10:10.906395+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:29.526591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:29.526591+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":902,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909372+0000","last_change":"2024-03-26T04:08:21.494251+0000","last_active":"2024-03-26T04:10:10.909372+0000","last_peered":"2024-03-26T04:10:10.909372+0000","last_clean":"2024-03-26T04:10:10.909372+0000","last_became_active":"2024-03-26T04:05:29.538900+0000","last_became_peered":"2024-03-26T04:05:29.538900+0000","last_unstale":"2024-03-26T04:10:10.909372+0000","last_undegraded":"2024-03-26T04:10:10.909372+0000","last_fullsized":"2024-03-26T04:10:10.909372+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:21.494131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:08:21.494131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":542,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906344+0000","last_change":"2024-03-26T04:08:56.851376+0000","last_active":"2024-03-26T04:10:10.906344+0000","last_peered":"2024-03-26T04:10:10.906344+0000","last_clean":"2024-03-26T04:10:10.906344+0000","last_became_active":"2024-03-26T04:08:56.850460+0000","last_became_peered":"2024-03-26T04:08:56.850460+0000","last_unstale":"2024-03-26T04:10:10.906344+0000","last_undegraded":"2024-03-26T04:10:10.906344+0000","last_fullsized":"2024-03-26T04:10:10.906344+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:03.918739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:03.918739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.2","version":"448'36","reported_seq":100,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906271+0000","last_change":"2024-03-26T04:08:26.643613+0000","last_active":"2024-03-26T04:10:10.906271+0000","last_peered":"2024-03-26T04:10:10.906271+0000","last_clean":"2024-03-26T04:10:10.906271+0000","last_became_active":"2024-03-26T04:08:26.587565+0000","last_became_peered":"2024-03-26T04:08:26.587565+0000","last_unstale":"2024-03-26T04:10:10.906271+0000","last_undegraded":"2024-03-26T04:10:10.906271+0000","last_fullsized":"2024-03-26T04:10:10.906271+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'26","last_scrub_stamp":"2024-03-26T04:04:12.249693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:12.249693+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"450'16","reported_seq":1209,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911974+0000","last_change":"2024-03-26T04:08:56.882173+0000","last_active":"2024-03-26T04:10:10.911974+0000","last_peered":"2024-03-26T04:10:10.911974+0000","last_clean":"2024-03-26T04:10:10.911974+0000","last_became_active":"2024-03-26T04:08:56.862194+0000","last_became_peered":"2024-03-26T04:08:56.862194+0000","last_unstale":"2024-03-26T04:10:10.911974+0000","last_undegraded":"2024-03-26T04:10:10.911974+0000","last_fullsized":"2024-03-26T04:10:10.911974+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'15","last_scrub_stamp":"2024-03-26T04:08:31.583103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:08:31.583103+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.e","version":"474'8134","reported_seq":13686,"reported_epoch":474,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:10:14.814928+0000","last_change":"2024-03-26T04:10:05.522602+0000","last_active":"2024-03-26T04:10:14.814928+0000","last_peered":"2024-03-26T04:10:14.814928+0000","last_clean":"2024-03-26T04:08:55.792710+0000","last_became_active":"2024-03-26T04:08:57.888224+0000","last_became_peered":"2024-03-26T04:08:57.888224+0000","last_unstale":"2024-03-26T04:10:14.814928+0000","last_undegraded":"2024-03-26T04:10:14.814928+0000","last_fullsized":"2024-03-26T04:10:14.814928+0000","mapping_epoch":450,"log_start":"458'5331","ondisk_log_start":"458'5331","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2803,"log_dups_size":3000,"ondisk_log_size":2803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2495,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8134,"num_write_kb":422208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2717,"num_bytes_recovered":178061312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,7],"avail_no_missing":["4","1","7"],"object_location_counts":[{"shards":"1,4,7","objects":5060}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":407,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905902+0000","last_change":"2024-03-26T04:08:59.842731+0000","last_active":"2024-03-26T04:10:10.905902+0000","last_peered":"2024-03-26T04:10:10.905902+0000","last_clean":"2024-03-26T04:10:10.905902+0000","last_became_active":"2024-03-26T04:08:59.842458+0000","last_became_peered":"2024-03-26T04:08:59.842458+0000","last_unstale":"2024-03-26T04:10:10.905902+0000","last_undegraded":"2024-03-26T04:10:10.905902+0000","last_fullsized":"2024-03-26T04:10:10.905902+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:26.838741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:26.838741+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":744,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.907141+0000","last_change":"2024-03-26T04:08:56.849293+0000","last_active":"2024-03-26T04:10:10.907141+0000","last_peered":"2024-03-26T04:10:10.907141+0000","last_clean":"2024-03-26T04:10:10.907141+0000","last_became_active":"2024-03-26T04:08:56.848865+0000","last_became_peered":"2024-03-26T04:08:56.848865+0000","last_unstale":"2024-03-26T04:10:10.907141+0000","last_undegraded":"2024-03-26T04:10:10.907141+0000","last_fullsized":"2024-03-26T04:10:10.907141+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:40.042936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.934617+0000","last_clean_scrub_stamp":"2024-03-26T04:07:40.042936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":886,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914148+0000","last_change":"2024-03-26T04:05:07.862785+0000","last_active":"2024-03-26T04:10:10.914148+0000","last_peered":"2024-03-26T04:10:10.914148+0000","last_clean":"2024-03-26T04:10:10.914148+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:10:10.914148+0000","last_undegraded":"2024-03-26T04:10:10.914148+0000","last_fullsized":"2024-03-26T04:10:10.914148+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:05:07.862659+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:05:07.862659+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"448'48","reported_seq":1029,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905499+0000","last_change":"2024-03-26T04:08:56.881594+0000","last_active":"2024-03-26T04:10:10.905499+0000","last_peered":"2024-03-26T04:10:10.905499+0000","last_clean":"2024-03-26T04:10:10.905499+0000","last_became_active":"2024-03-26T04:08:56.846958+0000","last_became_peered":"2024-03-26T04:08:56.846958+0000","last_unstale":"2024-03-26T04:10:10.905499+0000","last_undegraded":"2024-03-26T04:10:10.905499+0000","last_fullsized":"2024-03-26T04:10:10.905499+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'34","last_scrub_stamp":"2024-03-26T04:04:16.357311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:16.357311+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"325'12","reported_seq":623,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909641+0000","last_change":"2024-03-26T04:03:51.382658+0000","last_active":"2024-03-26T04:10:10.909641+0000","last_peered":"2024-03-26T04:10:10.909641+0000","last_clean":"2024-03-26T04:10:10.909641+0000","last_became_active":"2024-03-26T04:02:52.515390+0000","last_became_peered":"2024-03-26T04:02:52.515390+0000","last_unstale":"2024-03-26T04:10:10.909641+0000","last_undegraded":"2024-03-26T04:10:10.909641+0000","last_fullsized":"2024-03-26T04:10:10.909641+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'12","last_scrub_stamp":"2024-03-26T04:03:51.382563+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:03:51.382563+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.f","version":"474'8292","reported_seq":19160,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.567401+0000","last_change":"2024-03-26T04:10:10.925026+0000","last_active":"2024-03-26T04:10:16.567401+0000","last_peered":"2024-03-26T04:10:16.567401+0000","last_clean":"2024-03-26T04:10:16.567401+0000","last_became_active":"2024-03-26T04:10:10.914963+0000","last_became_peered":"2024-03-26T04:10:10.914963+0000","last_unstale":"2024-03-26T04:10:16.567401+0000","last_undegraded":"2024-03-26T04:10:16.567401+0000","last_fullsized":"2024-03-26T04:10:16.567401+0000","mapping_epoch":473,"log_start":"464'5899","ondisk_log_start":"464'5899","created":427,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2393,"log_dups_size":3000,"ondisk_log_size":2393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":319815680,"num_objects":4880,"num_object_clones":0,"num_object_copies":14640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8292,"num_write_kb":421504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5375,"num_bytes_recovered":352256000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":732,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905874+0000","last_change":"2024-03-26T04:07:39.474303+0000","last_active":"2024-03-26T04:10:10.905874+0000","last_peered":"2024-03-26T04:10:10.905874+0000","last_clean":"2024-03-26T04:10:10.905874+0000","last_became_active":"2024-03-26T04:05:29.521761+0000","last_became_peered":"2024-03-26T04:05:29.521761+0000","last_unstale":"2024-03-26T04:10:10.905874+0000","last_undegraded":"2024-03-26T04:10:10.905874+0000","last_fullsized":"2024-03-26T04:10:10.905874+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:39.474219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:07:39.474219+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":890,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905420+0000","last_change":"2024-03-26T04:08:56.846387+0000","last_active":"2024-03-26T04:10:10.905420+0000","last_peered":"2024-03-26T04:10:10.905420+0000","last_clean":"2024-03-26T04:10:10.905420+0000","last_became_active":"2024-03-26T04:08:56.845955+0000","last_became_peered":"2024-03-26T04:08:56.845955+0000","last_unstale":"2024-03-26T04:10:10.905420+0000","last_undegraded":"2024-03-26T04:10:10.905420+0000","last_fullsized":"2024-03-26T04:10:10.905420+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:30.569498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:07:30.569498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":395,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.913580+0000","last_change":"2024-03-26T04:08:56.867628+0000","last_active":"2024-03-26T04:10:10.913580+0000","last_peered":"2024-03-26T04:10:10.913580+0000","last_clean":"2024-03-26T04:10:10.913580+0000","last_became_active":"2024-03-26T04:08:56.867210+0000","last_became_peered":"2024-03-26T04:08:56.867210+0000","last_unstale":"2024-03-26T04:10:10.913580+0000","last_undegraded":"2024-03-26T04:10:10.913580+0000","last_fullsized":"2024-03-26T04:10:10.913580+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:06.121917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:03:06.121917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"448'64","reported_seq":505,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911785+0000","last_change":"2024-03-26T04:08:56.906816+0000","last_active":"2024-03-26T04:10:10.911785+0000","last_peered":"2024-03-26T04:10:10.911785+0000","last_clean":"2024-03-26T04:10:10.911785+0000","last_became_active":"2024-03-26T04:08:56.862404+0000","last_became_peered":"2024-03-26T04:08:56.862404+0000","last_unstale":"2024-03-26T04:10:10.911785+0000","last_undegraded":"2024-03-26T04:10:10.911785+0000","last_fullsized":"2024-03-26T04:10:10.911785+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'52","last_scrub_stamp":"2024-03-26T04:07:35.968179+0000","last_deep_scrub":"85'40","last_deep_scrub_stamp":"2024-03-26T03:49:51.111316+0000","last_clean_scrub_stamp":"2024-03-26T04:07:35.968179+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":893,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905165+0000","last_change":"2024-03-26T04:08:56.848799+0000","last_active":"2024-03-26T04:10:10.905165+0000","last_peered":"2024-03-26T04:10:10.905165+0000","last_clean":"2024-03-26T04:10:10.905165+0000","last_became_active":"2024-03-26T04:08:56.847255+0000","last_became_peered":"2024-03-26T04:08:56.847255+0000","last_unstale":"2024-03-26T04:10:10.905165+0000","last_undegraded":"2024-03-26T04:10:10.905165+0000","last_fullsized":"2024-03-26T04:10:10.905165+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.297701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"474'8381","reported_seq":19366,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.457205+0000","last_change":"2024-03-26T04:10:07.900561+0000","last_active":"2024-03-26T04:10:16.457205+0000","last_peered":"2024-03-26T04:10:16.457205+0000","last_clean":"2024-03-26T04:10:16.457205+0000","last_became_active":"2024-03-26T04:10:07.887223+0000","last_became_peered":"2024-03-26T04:10:07.887223+0000","last_unstale":"2024-03-26T04:10:16.457205+0000","last_undegraded":"2024-03-26T04:10:16.457205+0000","last_fullsized":"2024-03-26T04:10:16.457205+0000","mapping_epoch":470,"log_start":"464'5966","ondisk_log_start":"464'5966","created":427,"last_epoch_clean":471,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2415,"log_dups_size":3000,"ondisk_log_size":2415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324075520,"num_objects":4945,"num_object_clones":0,"num_object_copies":14835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8381,"num_write_kb":426432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5433,"num_bytes_recovered":356057088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":569,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905375+0000","last_change":"2024-03-26T04:07:22.530369+0000","last_active":"2024-03-26T04:10:10.905375+0000","last_peered":"2024-03-26T04:10:10.905375+0000","last_clean":"2024-03-26T04:10:10.905375+0000","last_became_active":"2024-03-26T04:05:29.524691+0000","last_became_peered":"2024-03-26T04:05:29.524691+0000","last_unstale":"2024-03-26T04:10:10.905375+0000","last_undegraded":"2024-03-26T04:10:10.905375+0000","last_fullsized":"2024-03-26T04:10:10.905375+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:07:22.530143+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:07:22.530143+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":567,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912737+0000","last_change":"2024-03-26T04:08:20.860169+0000","last_active":"2024-03-26T04:10:10.912737+0000","last_peered":"2024-03-26T04:10:10.912737+0000","last_clean":"2024-03-26T04:10:10.912737+0000","last_became_active":"2024-03-26T04:05:29.532692+0000","last_became_peered":"2024-03-26T04:05:29.532692+0000","last_unstale":"2024-03-26T04:10:10.912737+0000","last_undegraded":"2024-03-26T04:10:10.912737+0000","last_fullsized":"2024-03-26T04:10:10.912737+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:20.860049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:08:20.860049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":163,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"379'13","reported_seq":1168,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906983+0000","last_change":"2024-03-26T04:07:54.544916+0000","last_active":"2024-03-26T04:10:10.906983+0000","last_peered":"2024-03-26T04:10:10.906983+0000","last_clean":"2024-03-26T04:10:10.906983+0000","last_became_active":"2024-03-26T04:05:29.521835+0000","last_became_peered":"2024-03-26T04:05:29.521835+0000","last_unstale":"2024-03-26T04:10:10.906983+0000","last_undegraded":"2024-03-26T04:10:10.906983+0000","last_fullsized":"2024-03-26T04:10:10.906983+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"379'13","last_scrub_stamp":"2024-03-26T04:07:54.544776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:07:54.544776+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":466,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.909161+0000","last_change":"2024-03-26T04:08:56.862714+0000","last_active":"2024-03-26T04:10:10.909161+0000","last_peered":"2024-03-26T04:10:10.909161+0000","last_clean":"2024-03-26T04:10:10.909161+0000","last_became_active":"2024-03-26T04:08:56.862406+0000","last_became_peered":"2024-03-26T04:08:56.862406+0000","last_unstale":"2024-03-26T04:10:10.909161+0000","last_undegraded":"2024-03-26T04:10:10.909161+0000","last_fullsized":"2024-03-26T04:10:10.909161+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:03:50.413099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:03:50.413099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"448'39","reported_seq":1026,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.914049+0000","last_change":"2024-03-26T04:08:56.905523+0000","last_active":"2024-03-26T04:10:10.914049+0000","last_peered":"2024-03-26T04:10:10.914049+0000","last_clean":"2024-03-26T04:10:10.914049+0000","last_became_active":"2024-03-26T04:08:56.864029+0000","last_became_peered":"2024-03-26T04:08:56.864029+0000","last_unstale":"2024-03-26T04:10:10.914049+0000","last_undegraded":"2024-03-26T04:10:10.914049+0000","last_fullsized":"2024-03-26T04:10:10.914049+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'27","last_scrub_stamp":"2024-03-26T04:04:06.281970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.281970+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"474'8267","reported_seq":8372,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:16.803582+0000","last_change":"2024-03-26T04:08:02.634056+0000","last_active":"2024-03-26T04:10:16.803582+0000","last_peered":"2024-03-26T04:10:16.803582+0000","last_clean":"2024-03-26T04:10:16.803582+0000","last_became_active":"2024-03-26T04:08:02.633211+0000","last_became_peered":"2024-03-26T04:08:02.633211+0000","last_unstale":"2024-03-26T04:10:16.803582+0000","last_undegraded":"2024-03-26T04:10:16.803582+0000","last_fullsized":"2024-03-26T04:10:16.803582+0000","mapping_epoch":427,"log_start":"462'5639","ondisk_log_start":"462'5639","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2628,"log_dups_size":3000,"ondisk_log_size":2628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":323289088,"num_objects":4933,"num_object_clones":0,"num_object_copies":14799,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4933,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8267,"num_write_kb":422400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":236,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.911889+0000","last_change":"2024-03-26T04:08:56.872658+0000","last_active":"2024-03-26T04:10:10.911889+0000","last_peered":"2024-03-26T04:10:10.911889+0000","last_clean":"2024-03-26T04:10:10.911889+0000","last_became_active":"2024-03-26T04:08:56.871809+0000","last_became_peered":"2024-03-26T04:08:56.871809+0000","last_unstale":"2024-03-26T04:10:10.911889+0000","last_undegraded":"2024-03-26T04:10:10.911889+0000","last_fullsized":"2024-03-26T04:10:10.911889+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_clean_scrub_stamp":"2024-03-26T04:08:30.133040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":610,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906865+0000","last_change":"2024-03-26T04:07:34.478534+0000","last_active":"2024-03-26T04:10:10.906865+0000","last_peered":"2024-03-26T04:10:10.906865+0000","last_clean":"2024-03-26T04:10:10.906865+0000","last_became_active":"2024-03-26T04:05:29.520457+0000","last_became_peered":"2024-03-26T04:05:29.520457+0000","last_unstale":"2024-03-26T04:10:10.906865+0000","last_undegraded":"2024-03-26T04:10:10.906865+0000","last_fullsized":"2024-03-26T04:10:10.906865+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:34.478427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:07:34.478427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":891,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905262+0000","last_change":"2024-03-26T04:08:56.865804+0000","last_active":"2024-03-26T04:10:10.905262+0000","last_peered":"2024-03-26T04:10:10.905262+0000","last_clean":"2024-03-26T04:10:10.905262+0000","last_became_active":"2024-03-26T04:08:56.865055+0000","last_became_peered":"2024-03-26T04:08:56.865055+0000","last_unstale":"2024-03-26T04:10:10.905262+0000","last_undegraded":"2024-03-26T04:10:10.905262+0000","last_fullsized":"2024-03-26T04:10:10.905262+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:45.504517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:07:45.504517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"448'33","reported_seq":1044,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906794+0000","last_change":"2024-03-26T04:08:56.915010+0000","last_active":"2024-03-26T04:10:10.906794+0000","last_peered":"2024-03-26T04:10:10.906794+0000","last_clean":"2024-03-26T04:10:10.906794+0000","last_became_active":"2024-03-26T04:08:56.864260+0000","last_became_peered":"2024-03-26T04:08:56.864260+0000","last_unstale":"2024-03-26T04:10:10.906794+0000","last_undegraded":"2024-03-26T04:10:10.906794+0000","last_fullsized":"2024-03-26T04:10:10.906794+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'25","last_scrub_stamp":"2024-03-26T04:05:06.619261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:05:06.619261+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2","version":"474'8398","reported_seq":19362,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:18.354500+0000","last_change":"2024-03-26T04:10:09.163871+0000","last_active":"2024-03-26T04:10:18.354500+0000","last_peered":"2024-03-26T04:10:18.354500+0000","last_clean":"2024-03-26T04:10:18.354500+0000","last_became_active":"2024-03-26T04:10:09.154115+0000","last_became_peered":"2024-03-26T04:10:09.154115+0000","last_unstale":"2024-03-26T04:10:18.354500+0000","last_undegraded":"2024-03-26T04:10:18.354500+0000","last_fullsized":"2024-03-26T04:10:18.354500+0000","mapping_epoch":471,"log_start":"463'5810","ondisk_log_start":"463'5810","created":427,"last_epoch_clean":472,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2588,"log_dups_size":3000,"ondisk_log_size":2588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":312999936,"num_objects":4776,"num_object_clones":0,"num_object_copies":14328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4776,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8398,"num_write_kb":421568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5422,"num_bytes_recovered":355336192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":597,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905438+0000","last_change":"2024-03-26T04:05:29.533929+0000","last_active":"2024-03-26T04:10:10.905438+0000","last_peered":"2024-03-26T04:10:10.905438+0000","last_clean":"2024-03-26T04:10:10.905438+0000","last_became_active":"2024-03-26T04:05:29.533470+0000","last_became_peered":"2024-03-26T04:05:29.533470+0000","last_unstale":"2024-03-26T04:10:10.905438+0000","last_undegraded":"2024-03-26T04:10:10.905438+0000","last_fullsized":"2024-03-26T04:10:10.905438+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:02:39.967821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:02:39.967821+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":808,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906062+0000","last_change":"2024-03-26T04:05:29.522502+0000","last_active":"2024-03-26T04:10:10.906062+0000","last_peered":"2024-03-26T04:10:10.906062+0000","last_clean":"2024-03-26T04:10:10.906062+0000","last_became_active":"2024-03-26T04:05:29.521089+0000","last_became_peered":"2024-03-26T04:05:29.521089+0000","last_unstale":"2024-03-26T04:10:10.906062+0000","last_undegraded":"2024-03-26T04:10:10.906062+0000","last_fullsized":"2024-03-26T04:10:10.906062+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:49.166761+0000","last_clean_scrub_stamp":"2024-03-26T04:02:49.166761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":216,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":873,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.910837+0000","last_change":"2024-03-26T04:04:49.184140+0000","last_active":"2024-03-26T04:10:10.910837+0000","last_peered":"2024-03-26T04:10:10.910837+0000","last_clean":"2024-03-26T04:10:10.910837+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:10:10.910837+0000","last_undegraded":"2024-03-26T04:10:10.910837+0000","last_fullsized":"2024-03-26T04:10:10.910837+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:49.184018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:04:49.184018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":847,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905469+0000","last_change":"2024-03-26T04:08:56.866747+0000","last_active":"2024-03-26T04:10:10.905469+0000","last_peered":"2024-03-26T04:10:10.905469+0000","last_clean":"2024-03-26T04:10:10.905469+0000","last_became_active":"2024-03-26T04:08:56.865172+0000","last_became_peered":"2024-03-26T04:08:56.865172+0000","last_unstale":"2024-03-26T04:10:10.905469+0000","last_undegraded":"2024-03-26T04:10:10.905469+0000","last_fullsized":"2024-03-26T04:10:10.905469+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:05:08.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:05:08.666106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"448'45","reported_seq":951,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906473+0000","last_change":"2024-03-26T04:07:25.580036+0000","last_active":"2024-03-26T04:10:10.906473+0000","last_peered":"2024-03-26T04:10:10.906473+0000","last_clean":"2024-03-26T04:10:10.906473+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:10:10.906473+0000","last_undegraded":"2024-03-26T04:10:10.906473+0000","last_fullsized":"2024-03-26T04:10:10.906473+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'31","last_scrub_stamp":"2024-03-26T04:07:25.579888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:07:25.579888+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.3","version":"474'8402","reported_seq":17111,"reported_epoch":474,"state":"active+remapped+backfilling","last_fresh":"2024-03-26T04:10:16.583487+0000","last_change":"2024-03-26T04:09:27.950550+0000","last_active":"2024-03-26T04:10:16.583487+0000","last_peered":"2024-03-26T04:10:16.583487+0000","last_clean":"2024-03-26T04:08:55.796551+0000","last_became_active":"2024-03-26T04:08:57.903542+0000","last_became_peered":"2024-03-26T04:08:57.903542+0000","last_unstale":"2024-03-26T04:10:16.583487+0000","last_undegraded":"2024-03-26T04:10:16.583487+0000","last_fullsized":"2024-03-26T04:10:16.583487+0000","mapping_epoch":450,"log_start":"464'6035","ondisk_log_start":"464'6035","created":427,"last_epoch_clean":428,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:01.605639+0000","last_clean_scrub_stamp":"2024-03-26T04:08:01.605639+0000","objects_scrubbed":0,"log_size":2367,"log_dups_size":3000,"ondisk_log_size":2367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324665344,"num_objects":4954,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1238,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8402,"num_write_kb":427392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4298,"num_bytes_recovered":281673728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,7],"avail_no_missing":["1","2","7"],"object_location_counts":[{"shards":"1,2,7","objects":4954}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":813,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.905980+0000","last_change":"2024-03-26T04:10:10.100484+0000","last_active":"2024-03-26T04:10:10.905980+0000","last_peered":"2024-03-26T04:10:10.905980+0000","last_clean":"2024-03-26T04:10:10.905980+0000","last_became_active":"2024-03-26T04:08:59.847656+0000","last_became_peered":"2024-03-26T04:08:59.847656+0000","last_unstale":"2024-03-26T04:10:10.905980+0000","last_undegraded":"2024-03-26T04:10:10.905980+0000","last_fullsized":"2024-03-26T04:10:10.905980+0000","mapping_epoch":452,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":453,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:10:10.100403+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:10:10.100403+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:11:21.830931+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":873,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906973+0000","last_change":"2024-03-26T04:07:27.607414+0000","last_active":"2024-03-26T04:10:10.906973+0000","last_peered":"2024-03-26T04:10:10.906973+0000","last_clean":"2024-03-26T04:10:10.906973+0000","last_became_active":"2024-03-26T04:05:29.520602+0000","last_became_peered":"2024-03-26T04:05:29.520602+0000","last_unstale":"2024-03-26T04:10:10.906973+0000","last_undegraded":"2024-03-26T04:10:10.906973+0000","last_fullsized":"2024-03-26T04:10:10.906973+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:07:27.607316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:07:27.607316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"450'14","reported_seq":1089,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:18.104083+0000","last_change":"2024-03-26T04:10:18.104115+0000","last_active":"2024-03-26T04:10:18.104083+0000","last_peered":"2024-03-26T04:10:18.104083+0000","last_clean":"2024-03-26T04:10:18.104083+0000","last_became_active":"2024-03-26T04:08:56.845098+0000","last_became_peered":"2024-03-26T04:08:56.845098+0000","last_unstale":"2024-03-26T04:10:18.104083+0000","last_undegraded":"2024-03-26T04:10:18.104083+0000","last_fullsized":"2024-03-26T04:10:18.104083+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'14","last_scrub_stamp":"2024-03-26T04:10:18.104014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:10:18.104014+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:11:44.815904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":461,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.912439+0000","last_change":"2024-03-26T04:04:06.376407+0000","last_active":"2024-03-26T04:10:10.912439+0000","last_peered":"2024-03-26T04:10:10.912439+0000","last_clean":"2024-03-26T04:10:10.912439+0000","last_became_active":"2024-03-26T04:02:52.491417+0000","last_became_peered":"2024-03-26T04:02:52.491417+0000","last_unstale":"2024-03-26T04:10:10.912439+0000","last_undegraded":"2024-03-26T04:10:10.912439+0000","last_fullsized":"2024-03-26T04:10:10.912439+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:04:06.376284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:04:06.376284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"448'30","reported_seq":1007,"reported_epoch":474,"state":"active+clean","last_fresh":"2024-03-26T04:10:10.906112+0000","last_change":"2024-03-26T04:08:56.892368+0000","last_active":"2024-03-26T04:10:10.906112+0000","last_peered":"2024-03-26T04:10:10.906112+0000","last_clean":"2024-03-26T04:10:10.906112+0000","last_became_active":"2024-03-26T04:08:56.863470+0000","last_became_peered":"2024-03-26T04:08:56.863470+0000","last_unstale":"2024-03-26T04:10:10.906112+0000","last_undegraded":"2024-03-26T04:10:10.906112+0000","last_fullsized":"2024-03-26T04:10:10.906112+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'20","last_scrub_stamp":"2024-03-26T04:04:18.408372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:04:18.408372+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5143199768,"num_objects":78480,"num_object_clones":0,"num_object_copies":235440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3733,"num_objects_unfound":0,"num_objects_dirty":78480,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":132870,"num_write_kb":6763137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39495,"num_bytes_recovered":2588344320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":17935249408,"data_stored":17935237192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40438,"ondisk_log_size":40438,"up":48,"acting":48,"num_store_stats":5},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":80,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":446,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":26,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":488689,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3891200,"data_stored":3869888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":5664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":7471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1824,"num_read_kb":1549,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":574,"num_bytes_recovered":10320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":675840,"data_stored":17960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1249,"ondisk_log_size":1249,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":253403070726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":214748365068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5563192,"kb_used_data":4471344,"kb_used_omap":1,"kb_used_meta":1091838,"kb_avail":88181960,"statfs":{"total":95995035648,"available":90298327040,"internally_reserved":0,"allocated":4578656256,"data_stored":4577957325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1118042565},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":118,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5101924,"kb_used_data":3892636,"kb_used_omap":1,"kb_used_meta":1209278,"kb_avail":88643228,"statfs":{"total":95995035648,"available":90770665472,"internally_reserved":0,"allocated":3986059264,"data_stored":3985380490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1238301125},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637947,"num_pgs":120,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4603484,"kb_used_data":3998932,"kb_used_omap":6,"kb_used_meta":604537,"kb_avail":89141668,"statfs":{"total":95995035648,"available":91281068032,"internally_reserved":0,"allocated":4094906368,"data_stored":4094235545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":619046181},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953749,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3730976,"kb_used_data":2604440,"kb_used_omap":1,"kb_used_meta":1126526,"kb_avail":90014176,"statfs":{"total":95995035648,"available":92174516224,"internally_reserved":0,"allocated":2666946560,"data_stored":2666270928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1153563077},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":773094113453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3965190144,"data_stored":3965190144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4524937216,"data_stored":4524933144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4078833664,"data_stored":4078829592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2649034752,"data_stored":2649030680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2717253632,"data_stored":2717253632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:10:21.718 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-03-26T04:10:21.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:21.861 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:21.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:21.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:21.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:22.088 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:22.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:22.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:22.257 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:21 smithi006 ceph-mon[29446]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 815 op/s; 2911/234951 objects misplaced (1.239%); 34 MiB/s, 552 objects/s recovering 2024-03-26T04:10:22.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:21 smithi006 ceph-mon[37501]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 815 op/s; 2911/234951 objects misplaced (1.239%); 34 MiB/s, 552 objects/s recovering 2024-03-26T04:10:22.281 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:22.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:21 smithi086 ceph-mon[32965]: pgmap v1034: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.8 GiB data, 18 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 815 op/s; 2911/234951 objects misplaced (1.239%); 34 MiB/s, 552 objects/s recovering 2024-03-26T04:10:22.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:22.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:22.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:22.534 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:22.636 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:22.683 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:22.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:22.784 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:22.823 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:22.912 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:10:22.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:22.950 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:22.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:22.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:23.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:23.093 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:23.111 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:23.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:23.159 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:23.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:23.264 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:23.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:23.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:23.391 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:23.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:23.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:23.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:23.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:23.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.643 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:23.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:23.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:23.769 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:23.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:23.894 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:23.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:23.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:23.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:24.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:24.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:24.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[29446]: pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.5 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 2491/221472 objects misplaced (1.125%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[37501]: pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.5 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 2491/221472 objects misplaced (1.125%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T04:10:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:23 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T04:10:24.160 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:24.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:24.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:24.287 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:23 smithi086 ceph-mon[32965]: pgmap v1035: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.5 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 2491/221472 objects misplaced (1.125%); 32 MiB/s, 506 objects/s recovering 2024-03-26T04:10:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:23 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T04:10:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:23 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T04:10:24.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.414 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:24.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:24.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:24.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:24.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:24.565 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:24.641 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:24.680 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:24.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:24.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:24.826 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:24.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:24.968 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:24.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:24.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:25.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:25.094 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:25.196 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:25.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:25.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:25.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:25.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:25.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:25.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:25.664 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:25.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:25.809 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:25.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:25.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:25.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:25.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:25.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:26.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:26.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:26.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[37501]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.05k op/s; 2491/218190 objects misplaced (1.142%); 13 MiB/s, 209 objects/s recovering 2024-03-26T04:10:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T04:10:26.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[29446]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.05k op/s; 2491/218190 objects misplaced (1.142%); 13 MiB/s, 209 objects/s recovering 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:26.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:26.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:26.209 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:26.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:26.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:26.335 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:26.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-mon[32965]: pgmap v1036: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 340 GiB / 358 GiB avail; 0 B/s wr, 1.05k op/s; 2491/218190 objects misplaced (1.142%); 13 MiB/s, 209 objects/s recovering 2024-03-26T04:10:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T04:10:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T04:10:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:26.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:26.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:26.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:26.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:26.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:26.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:26.732 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:26.834 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:26.877 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:26.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:26.971 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:10:26.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:27.003 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:27.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:27.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:27.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:27.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:27.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:26 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:10:27.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:27.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:26 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:10:27.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.276 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:27.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:27.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:27.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:26 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:10:27.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:27.402 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:27.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:27.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.528 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:27.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:27.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:27.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:27.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:27.755 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:27.793 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:27.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:27.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:27.919 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:27.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:28.020 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:28.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:28.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:28.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:28.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:28.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:28.301 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:28.329 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 1728/217032 objects misplaced (0.796%); 16 MiB/s, 262 objects/s recovering 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: osdmap e475: 8 total, 8 up, 4 in 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:27 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 1728/217032 objects misplaced (0.796%); 16 MiB/s, 262 objects/s recovering 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: osdmap e475: 8 total, 8 up, 4 in 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: pgmap v1037: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.4 GiB data, 17 GiB used, 341 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.03k op/s; 1728/217032 objects misplaced (0.796%); 16 MiB/s, 262 objects/s recovering 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:10:28.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4220972106' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:10:28.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: osdmap e475: 8 total, 8 up, 4 in 2024-03-26T04:10:28.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:28.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:27 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:28.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:28.469 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:28.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:28.595 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:28.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:28.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:28.721 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:28.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:28.823 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:28.867 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:28.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:28.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:28.994 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:28.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:29.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:29.125 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:29.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:29.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:28 smithi086 ceph-mon[32965]: osdmap e476: 8 total, 8 up, 4 in 2024-03-26T04:10:29.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:28 smithi086 ceph-mon[32965]: 1.8 scrub starts 2024-03-26T04:10:29.238 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:28 smithi086 ceph-mon[32965]: 1.8 scrub ok 2024-03-26T04:10:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:29.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:29.369 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:29.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[37501]: osdmap e476: 8 total, 8 up, 4 in 2024-03-26T04:10:29.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[37501]: 1.8 scrub starts 2024-03-26T04:10:29.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[37501]: 1.8 scrub ok 2024-03-26T04:10:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[29446]: osdmap e476: 8 total, 8 up, 4 in 2024-03-26T04:10:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[29446]: 1.8 scrub starts 2024-03-26T04:10:29.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:28 smithi006 ceph-mon[29446]: 1.8 scrub ok 2024-03-26T04:10:29.409 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:29.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:29.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:29.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:29.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:29.690 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:29.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:29.820 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:29.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:29.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:29.966 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:29.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:30.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.119 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:30.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:30.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:30.243 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:30.299 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:29 smithi006 ceph-mon[37501]: pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 1318/200214 objects misplaced (0.658%); 10 MiB/s, 167 objects/s recovering 2024-03-26T04:10:30.299 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:29 smithi006 ceph-mon[29446]: pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 1318/200214 objects misplaced (0.658%); 10 MiB/s, 167 objects/s recovering 2024-03-26T04:10:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:29 smithi086 ceph-mon[32965]: pgmap v1040: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.1 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s; 1318/200214 objects misplaced (0.658%); 10 MiB/s, 167 objects/s recovering 2024-03-26T04:10:30.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:30.369 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:30.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:30.495 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:30.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:30.621 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:30.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:30.746 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:30.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:30.872 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:30.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:30.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:30.937 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:10:30.937 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:10:30.937 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:10:30.937 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:10:30.937 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:10:30.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:30.999 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:31.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:31.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:30 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T04:10:31.131 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:30 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T04:10:31.131 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:30 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T04:10:31.131 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:31 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T04:10:31.139 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:31.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:31.241 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:31.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:30 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T04:10:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:30 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T04:10:31.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:31.408 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:31.412 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:31.510 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:31.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:31.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:31.656 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:31.695 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:31.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:31.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:31.822 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:31.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:31.963 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:31.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:32.022 INFO:teuthology.orchestra.run.smithi006.stderr:set pool 11 pgp_num to 16 2024-03-26T04:10:32.032 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:10:32.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:32.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:32.190 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:32.228 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:32.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s; 699/199176 objects misplaced (0.351%); 12 MiB/s, 194 objects/s recovering 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:32 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T04:10:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.354 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:32.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:32.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s; 699/199176 objects misplaced (0.351%); 12 MiB/s, 194 objects/s recovering 2024-03-26T04:10:32.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: pgmap v1041: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 4.0 GiB data, 16 GiB used, 342 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 929 op/s; 699/199176 objects misplaced (0.351%); 12 MiB/s, 194 objects/s recovering 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:32 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-26T04:10:32.455 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:32.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-26T04:10:32.480 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:32.582 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:32.606 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:32.708 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:32.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.733 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:32.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:32.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:32.859 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:32.960 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:32.984 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:32.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:32.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:33.085 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:33.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:33.259 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: osdmap e477: 8 total, 8 up, 4 in 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: 3.1f deep-scrub starts 2024-03-26T04:10:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:33 smithi086 ceph-mon[32965]: 3.1f deep-scrub ok 2024-03-26T04:10:33.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.386 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:33.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T04:10:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: osdmap e477: 8 total, 8 up, 4 in 2024-03-26T04:10:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:33.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: 3.1f deep-scrub starts 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[37501]: 3.1f deep-scrub ok 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1843874823' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: osdmap e477: 8 total, 8 up, 4 in 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: 3.1f deep-scrub starts 2024-03-26T04:10:33.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:33 smithi006 ceph-mon[29446]: 3.1f deep-scrub ok 2024-03-26T04:10:33.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:33.612 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:33.636 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:33.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:33.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:33.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:33.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:33.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:33.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:33.988 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:34.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:34.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:34.062 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.8 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 328/185817 objects misplaced (0.177%); 13 MiB/s, 212 objects/s recovering 2024-03-26T04:10:34.062 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.8 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 328/185817 objects misplaced (0.177%); 13 MiB/s, 212 objects/s recovering 2024-03-26T04:10:34.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:34.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:34.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:34.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:34.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.264 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:34.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: osdmap e478: 8 total, 8 up, 4 in 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: 1.0 deep-scrub starts 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[37501]: 1.0 deep-scrub ok 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: osdmap e478: 8 total, 8 up, 4 in 2024-03-26T04:10:34.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: 1.0 deep-scrub starts 2024-03-26T04:10:34.321 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-mon[29446]: 1.0 deep-scrub ok 2024-03-26T04:10:34.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: pgmap v1043: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.8 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.74k op/s; 328/185817 objects misplaced (0.177%); 13 MiB/s, 212 objects/s recovering 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: osdmap e478: 8 total, 8 up, 4 in 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: 1.0 deep-scrub starts 2024-03-26T04:10:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:34 smithi086 ceph-mon[32965]: 1.0 deep-scrub ok 2024-03-26T04:10:34.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:34.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:34.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:34.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:34.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:34.639 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:34.677 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:34.779 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:34.817 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:34.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:34.943 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:34.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:34.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:35.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:35.089 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:35.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:35.191 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:35.248 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-mon[32965]: osdmap e479: 8 total, 8 up, 4 in 2024-03-26T04:10:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T04:10:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T04:10:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T04:10:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T04:10:35.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:35.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:35.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[37501]: osdmap e479: 8 total, 8 up, 4 in 2024-03-26T04:10:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T04:10:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T04:10:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T04:10:35.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T04:10:35.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[29446]: osdmap e479: 8 total, 8 up, 4 in 2024-03-26T04:10:35.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T04:10:35.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T04:10:35.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T04:10:35.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:35 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T04:10:35.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:35.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:35.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:35.640 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:35.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:35.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:35.766 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:35.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:35.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:35.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:35.891 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:35.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:35.992 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:36.030 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:36.132 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:36.169 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:36.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:36.310 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:36 smithi086 ceph-mon[32965]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 328/183006 objects misplaced (0.179%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T04:10:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:36 smithi086 ceph-mon[32965]: 5.14 scrub starts 2024-03-26T04:10:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:36 smithi086 ceph-mon[32965]: 5.14 scrub ok 2024-03-26T04:10:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:36 smithi086 ceph-mon[32965]: 3.1c deep-scrub starts 2024-03-26T04:10:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:36 smithi086 ceph-mon[32965]: 3.1c deep-scrub ok 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[29446]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 328/183006 objects misplaced (0.179%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[29446]: 5.14 scrub starts 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[29446]: 5.14 scrub ok 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[29446]: 3.1c deep-scrub starts 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[29446]: 3.1c deep-scrub ok 2024-03-26T04:10:36.388 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[37501]: pgmap v1046: 158 pgs: 2 active+remapped+backfilling, 156 active+clean; 3.7 GiB data, 15 GiB used, 342 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 956 op/s; 328/183006 objects misplaced (0.179%); 9.6 MiB/s, 154 objects/s recovering 2024-03-26T04:10:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[37501]: 5.14 scrub starts 2024-03-26T04:10:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[37501]: 5.14 scrub ok 2024-03-26T04:10:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[37501]: 3.1c deep-scrub starts 2024-03-26T04:10:36.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:36 smithi006 ceph-mon[37501]: 3.1c deep-scrub ok 2024-03-26T04:10:36.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:36.436 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:36.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:36.452 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:36.538 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:36.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:36.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:36.722 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:36.824 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:36.862 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:36.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:36.987 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:36.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:37.089 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.113 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:37.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:37.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:37.238 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:37.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:37.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:37.364 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:37.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:37.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:37.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[37501]: 3.8 scrub starts 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[37501]: 3.8 scrub ok 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[37501]: 5.1 scrub starts 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[37501]: 5.1 scrub ok 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[29446]: 3.8 scrub starts 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[29446]: 3.8 scrub ok 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[29446]: 5.1 scrub starts 2024-03-26T04:10:37.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:37 smithi006 ceph-mon[29446]: 5.1 scrub ok 2024-03-26T04:10:37.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:37.489 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:37.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:37 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:37 smithi086 ceph-mon[32965]: 3.8 scrub starts 2024-03-26T04:10:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:37 smithi086 ceph-mon[32965]: 3.8 scrub ok 2024-03-26T04:10:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:37 smithi086 ceph-mon[32965]: 5.1 scrub starts 2024-03-26T04:10:37.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:37 smithi086 ceph-mon[32965]: 5.1 scrub ok 2024-03-26T04:10:37.591 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:37.629 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:37.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:37.770 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:37.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:37.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:37.897 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:37.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:37.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:38.022 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:38.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:38.165 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:38.267 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.307 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:38.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:38.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:38 smithi006 ceph-mon[37501]: pgmap v1047: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 943 op/s; 37/182208 objects misplaced (0.020%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T04:10:38.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:38 smithi006 ceph-mon[29446]: pgmap v1047: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 943 op/s; 37/182208 objects misplaced (0.020%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T04:10:38.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:38.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:38.549 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:38.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.588 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:38.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:38.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:38 smithi086 ceph-mon[32965]: pgmap v1047: 158 pgs: 1 active+remapped+backfilling, 157 active+clean; 3.7 GiB data, 15 GiB used, 343 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 943 op/s; 37/182208 objects misplaced (0.020%); 6.7 MiB/s, 106 objects/s recovering 2024-03-26T04:10:38.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:38.713 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:38.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:38.838 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:38.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:38.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:38.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:39.064 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:39.108 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:39.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:39.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:39.374 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:39 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T04:10:39.374 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:39 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T04:10:39.374 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:39 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T04:10:39.374 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:39 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T04:10:39.387 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:39.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:39.513 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:39.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:39 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T04:10:39.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:39 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T04:10:39.615 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:39.654 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:39.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:39.756 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.796 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:39.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:39.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:39.921 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:39.922 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:40.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:40.026 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:10:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:40.071 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:10:40.116 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:10:40.149 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:40.154 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:10:40.160 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4] out_osds: [6, 5, 3, 7] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2024-03-26T04:10:40.160 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-26T04:10:40.160 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-26T04:10:40.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight-by-utilization 110 1.0 2024-03-26T04:10:40.192 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:10:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:40.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:40.234 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:10:40.274 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:10:40.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:40.314 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:10:40.315 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[37501]: pgmap v1048: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 847 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-03-26T04:10:40.315 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T04:10:40.315 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T04:10:40.315 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1474607782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:40.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[29446]: pgmap v1048: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 847 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-03-26T04:10:40.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T04:10:40.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T04:10:40.316 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1474607782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:40.319 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:40.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:10:40.324 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:10:40.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:40.446 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:40.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:40.548 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:40.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:40 smithi086 ceph-mon[32965]: pgmap v1048: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 847 op/s; 3.1 MiB/s, 49 objects/s recovering 2024-03-26T04:10:40.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:40 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T04:10:40.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:40 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T04:10:40.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1474607782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:40.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:40.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:40.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:40.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:40.838 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:40.883 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:40.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:40.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:41.009 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:41.097 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:41.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:41.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:41.252 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:41.294 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:41.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:41.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:41.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:41.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:41.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:41.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T04:10:41.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T04:10:41.481 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[37501]: pgmap v1049: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 738 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-26T04:10:41.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:41.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:41.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T04:10:41.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T04:10:41.482 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:41 smithi006 ceph-mon[29446]: pgmap v1049: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 738 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-26T04:10:41.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:41.560 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:41 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:41 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:41 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T04:10:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:41 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T04:10:41.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:41 smithi086 ceph-mon[32965]: pgmap v1049: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 343 GiB / 358 GiB avail; 0 B/s wr, 738 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-26T04:10:41.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:41.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:41.813 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:41.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:41.939 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:41.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:41.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:42.041 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:42.078 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:42.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:42.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:42.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:42.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.347 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:42.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:42.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:42 smithi006 ceph-mon[37501]: 5.15 scrub starts 2024-03-26T04:10:42.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:42 smithi006 ceph-mon[37501]: 5.15 scrub ok 2024-03-26T04:10:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:42 smithi006 ceph-mon[29446]: 5.15 scrub starts 2024-03-26T04:10:42.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:42 smithi006 ceph-mon[29446]: 5.15 scrub ok 2024-03-26T04:10:42.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:42.472 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:42.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:42.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:42 smithi086 ceph-mon[32965]: 5.15 scrub starts 2024-03-26T04:10:42.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:42 smithi086 ceph-mon[32965]: 5.15 scrub ok 2024-03-26T04:10:42.617 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:42.719 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:42.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.758 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:42.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:42.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:42.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:42.884 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:42.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:42.985 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:43.012 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:43.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:43.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:10:43.114 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:43.180 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:43.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:43.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:43.290 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:10:43.307 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:43.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:43.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:43.459 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:43.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:43.584 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:43.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:10:43.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: pgmap v1050: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.12k op/s; 2.1 MiB/s, 34 objects/s recovering 2024-03-26T04:10:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T04:10:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:43.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:43 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:43.639 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: pgmap v1050: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.12k op/s; 2.1 MiB/s, 34 objects/s recovering 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:43.696 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:43.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:43.697 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: pgmap v1050: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.12k op/s; 2.1 MiB/s, 34 objects/s recovering 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:43.698 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:43 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:43.721 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:43.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:43.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:43.846 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:43.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:43.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:43.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:43.973 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:43.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:44.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:44.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:44.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:44.223 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:44.304 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:44.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:44.349 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:44.380 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:44.401 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:10:44.401 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:10:44.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:44.475 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:44.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:44.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:44 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:10:44.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:10:44.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:44 smithi086 ceph-mon[32965]: osdmap e480: 8 total, 8 up, 4 in 2024-03-26T04:10:44.597 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:44.597 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:10:44.597 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:10:44.598 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[37501]: osdmap e480: 8 total, 8 up, 4 in 2024-03-26T04:10:44.598 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:10:44.598 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1581892292' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:10:44.598 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:44 smithi006 ceph-mon[29446]: osdmap e480: 8 total, 8 up, 4 in 2024-03-26T04:10:44.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:44.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:44.728 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:44.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:44.830 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:44.875 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:44.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:44.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:44.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:45.002 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:45.104 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:45.144 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:45.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:45.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:45.270 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:45.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:45.395 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:45.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:45.538 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:45.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:45.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:45.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:45.791 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:45 smithi086 ceph-mon[32965]: pgmap v1052: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.08k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:10:45.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:45.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:45 smithi006 ceph-mon[37501]: pgmap v1052: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.08k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:10:45.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:45 smithi006 ceph-mon[29446]: pgmap v1052: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 344 GiB / 358 GiB avail; 0 B/s wr, 1.08k op/s; 2.0 MiB/s, 32 objects/s recovering 2024-03-26T04:10:45.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:45.917 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:45.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:46.019 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:46.040 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-26T04:10:46.060 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:46.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:46.162 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:46.185 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:46.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:46.312 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:46.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:46.446 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:46.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:46.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:46.584 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:46.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:46.711 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:46.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:46.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:46.836 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:46.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:46.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:46.938 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:46.977 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:46.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:46.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:47.078 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:47.124 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:47.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:47.226 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:47.264 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:47.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:47.390 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:47.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:47.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:47.516 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:47.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:47.617 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:47.628 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:47 smithi006 ceph-mon[29446]: pgmap v1053: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 215 KiB/s, 3 objects/s recovering 2024-03-26T04:10:47.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:47.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:47.795 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:47.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:47.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:47 smithi006 ceph-mon[37501]: pgmap v1053: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 215 KiB/s, 3 objects/s recovering 2024-03-26T04:10:47.897 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:47.933 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:47.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:48.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:48.059 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:48.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:47 smithi086 ceph-mon[32965]: pgmap v1053: 158 pgs: 158 active+clean; 3.0 GiB data, 13 GiB used, 345 GiB / 358 GiB avail; 0 B/s wr, 1.09k op/s; 215 KiB/s, 3 objects/s recovering 2024-03-26T04:10:48.161 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:48.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:48.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.370 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:48.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:48.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:48.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:48.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:48.596 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:48.637 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:48.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:48.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:48.762 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:48.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:48.888 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:48.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:48.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.031 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:49.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:49.156 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:49.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:49.258 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:49.297 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:49.399 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:49.436 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:49.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:49.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:49.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:49.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:49.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.706 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:49.807 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:49.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:49.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:49 smithi006 ceph-mon[37501]: pgmap v1054: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T04:10:49.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:49 smithi006 ceph-mon[29446]: pgmap v1054: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T04:10:49.951 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:49.992 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:49.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:49 smithi086 ceph-mon[32965]: pgmap v1054: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s 2024-03-26T04:10:50.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:50.130 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:50.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:50.232 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:50.270 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:50.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:50.414 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:50.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:50.516 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:50.553 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:50.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:50.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:50.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.680 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:50.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:50.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:50.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:50.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:50.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:50.946 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:50.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:50.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:51.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:51.073 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:50 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:50 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:51.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:50 smithi086 ceph-mon[32965]: osdmap e481: 8 total, 8 up, 4 in 2024-03-26T04:10:51.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[37501]: osdmap e481: 8 total, 8 up, 4 in 2024-03-26T04:10:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:50 smithi006 ceph-mon[29446]: osdmap e481: 8 total, 8 up, 4 in 2024-03-26T04:10:51.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:51.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:51.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:51.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:51.303 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-26T04:10:51.325 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:51.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:51.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:51.451 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:51.553 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:51.603 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:51.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:51.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:51.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:51.780 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:51 smithi006 ceph-mon[29446]: pgmap v1056: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:10:51.780 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:51 smithi006 ceph-mon[37501]: pgmap v1056: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:10:51.868 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:51.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:51.916 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:51.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:52.003 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:52.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:51 smithi086 ceph-mon[32965]: pgmap v1056: 158 pgs: 158 active+clean; 2.6 GiB data, 12 GiB used, 346 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 960 op/s 2024-03-26T04:10:52.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.091 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:52.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:52.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.241 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:52.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:52.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:52.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:52.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:52.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:52.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:52.631 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:52.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:52.757 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:52.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:52.883 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:52.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:52.954 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:10:52.955 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:10:52.955 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[37501]: osdmap e482: 8 total, 8 up, 4 in 2024-03-26T04:10:52.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:10:52.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:10:52.955 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:52 smithi006 ceph-mon[29446]: osdmap e482: 8 total, 8 up, 4 in 2024-03-26T04:10:52.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:53.010 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:52 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:10:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:52 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:10:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:52 smithi086 ceph-mon[32965]: osdmap e482: 8 total, 8 up, 4 in 2024-03-26T04:10:53.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:53.135 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:53.237 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:53.278 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:53.380 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:53.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:53.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:53.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:53.546 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:53.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:53.648 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.697 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:53.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:53.798 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:53.837 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:53.939 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:53.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:53 smithi006 ceph-mon[29446]: pgmap v1058: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-26T04:10:53.946 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:53 smithi006 ceph-mon[37501]: pgmap v1058: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-26T04:10:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:53.984 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:53.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:54.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:53 smithi086 ceph-mon[32965]: pgmap v1058: 158 pgs: 158 active+clean; 2.4 GiB data, 11 GiB used, 346 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.41k op/s 2024-03-26T04:10:54.109 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:54.188 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:10:54.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:54.235 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:54.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:54.361 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:54.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:54.463 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.505 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:54.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:54.582 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:10:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:10:54.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:54.629 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:54.704 INFO:teuthology.orchestra.run.smithi006.stdout:moved 14 / 474 (2.95359%) 2024-03-26T04:10:54.704 INFO:teuthology.orchestra.run.smithi006.stdout:avg 118.5 2024-03-26T04:10:54.704 INFO:teuthology.orchestra.run.smithi006.stdout:stddev 6.42262 -> 4.60977 (expected baseline 9.42735) 2024-03-26T04:10:54.704 INFO:teuthology.orchestra.run.smithi006.stdout:min osd.4 with 109 -> 115 pgs (0.919831 -> 0.970464 * mean) 2024-03-26T04:10:54.704 INFO:teuthology.orchestra.run.smithi006.stdout:max osd.1 with 127 -> 113 pgs (1.07173 -> 0.953587 * mean) 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:oload 110 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:max_change 1 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:max_change_osds 4 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:average_utilization 0.0315 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:overload_utilization 0.0347 2024-03-26T04:10:54.705 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 weight 1.0000 -> 0.8980 2024-03-26T04:10:54.730 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:54.775 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:54.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:54.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[29446]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:54.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:54.836 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:54.836 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[37501]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:54.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:54.837 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:54 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:54.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:54.918 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:54.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:54.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:54.954 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:10:55.002 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:10:55.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:55.043 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:55.045 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:10:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:54 smithi086 ceph-mon[32965]: from='client.15885 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:10:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:54 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:54 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]: dispatch 2024-03-26T04:10:55.088 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:10:55.131 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:10:55.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:55.169 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:55.180 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:10:55.218 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:10:55.261 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:10:55.269 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:10:55.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.297 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:55.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:55.422 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:55.524 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:55.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:55.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:55.702 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:55.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:55.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:55.829 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:55.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:10:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: pgmap v1059: 158 pgs: 158 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T04:10:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]': finished 2024-03-26T04:10:55.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: osdmap e483: 8 total, 8 up, 4 in 2024-03-26T04:10:55.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3955563455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:55.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:55.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:55.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:55.954 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:55.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:55.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:56.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:56.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:56.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:56.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: pgmap v1059: 158 pgs: 158 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]': finished 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: osdmap e483: 8 total, 8 up, 4 in 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3955563455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: pgmap v1059: 158 pgs: 158 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.51k op/s 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'58854'}"}]': finished 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: osdmap e483: 8 total, 8 up, 4 in 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3955563455' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:10:56.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:10:56.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:56.204 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:56.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:56.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:56.345 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:56.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:56.447 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:56.488 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:56.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:56.614 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:56.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:56.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.741 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:56.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:56.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:56.882 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:56.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:56.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:56.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:57.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.023 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:57.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:57.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:57.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[37501]: osdmap e484: 8 total, 8 up, 4 in 2024-03-26T04:10:57.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:10:57.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[29446]: osdmap e484: 8 total, 8 up, 4 in 2024-03-26T04:10:57.065 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:56 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:10:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:10:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:56 smithi086 ceph-mon[32965]: osdmap e484: 8 total, 8 up, 4 in 2024-03-26T04:10:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:56 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:10:57.124 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:57.162 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:57.264 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:57.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:57.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:57.405 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:57.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:57.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:57.575 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:57.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:57.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:57.700 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:57.802 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:57.835 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:57.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:57.962 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:57.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:58.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:58.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:57 smithi086 ceph-mon[32965]: pgmap v1062: 158 pgs: 5 peering, 153 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T04:10:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.088 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:58.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:58.142 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:57 smithi006 ceph-mon[37501]: pgmap v1062: 158 pgs: 5 peering, 153 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T04:10:58.143 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:57 smithi006 ceph-mon[29446]: pgmap v1062: 158 pgs: 5 peering, 153 active+clean; 2.3 GiB data, 11 GiB used, 347 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-26T04:10:58.189 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:58.233 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:58.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:58.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:58.358 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:58.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:58.460 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:58.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:58.623 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:58.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:58.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:58.748 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:58.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:10:58.889 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:58.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:58.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:10:58.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:10:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.017 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:59.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:10:59.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:10:59.142 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:10:59.244 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:10:59.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:59.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:10:59.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:10:59.409 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:10:59.507 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2024-03-26T04:10:59.507 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight-by-utilization 110 3.0 2024-03-26T04:10:59.511 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:10:59.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.599 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:59.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:10:59.602 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:10:59 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a[63504]: 2024-03-26T04:10:59.343+0000 7f80a6940700 -1 rgw watcher librados: RGWWatcher::handle_error cookie 94232629236096 err (110) Connection timed out 2024-03-26T04:10:59.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:10:59.724 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:10:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:10:59.826 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:10:59.885 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:10:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:10:59.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:10:59.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:10:59 smithi006 ceph-mon[37501]: pgmap v1063: 158 pgs: 5 peering, 153 active+clean; 1.9 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 149 B/s rd, 0 B/s wr, 1.29k op/s; 32 B/s, 1 objects/s recovering 2024-03-26T04:10:59.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:10:59 smithi006 ceph-mon[29446]: pgmap v1063: 158 pgs: 5 peering, 153 active+clean; 1.9 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 149 B/s rd, 0 B/s wr, 1.29k op/s; 32 B/s, 1 objects/s recovering 2024-03-26T04:10:59.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:00.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:00.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:10:59 smithi086 ceph-mon[32965]: pgmap v1063: 158 pgs: 5 peering, 153 active+clean; 1.9 GiB data, 9.9 GiB used, 348 GiB / 358 GiB avail; 149 B/s rd, 0 B/s wr, 1.29k op/s; 32 B/s, 1 objects/s recovering 2024-03-26T04:11:00.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:00.138 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:00.239 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.282 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:00.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:00.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:00.407 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:00.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:00.551 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:00.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:00.653 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:00.698 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:00.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:00.766 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is set 2024-03-26T04:11:00.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:00.826 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4150580554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:00.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:00 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4150580554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:00.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:00 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:00.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:00.952 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:00.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:01.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:01.079 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:01.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4150580554' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:00 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:01.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:01.205 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:01.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:01.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:01.347 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:01.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:01.448 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:01.485 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:01.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.611 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:01.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:01.712 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:01.778 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:01.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:01.798 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:01.881 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:01.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:01.930 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:01.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:02.031 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:02.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[37501]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 348 GiB / 358 GiB avail; 127 B/s rd, 0 B/s wr, 908 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:11:02.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[37501]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:02.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T04:11:02.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:02.071 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[37501]: osdmap e485: 8 total, 8 up, 4 in 2024-03-26T04:11:02.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[29446]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 348 GiB / 358 GiB avail; 127 B/s rd, 0 B/s wr, 908 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:11:02.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[29446]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:02.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T04:11:02.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:02.072 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:01 smithi006 ceph-mon[29446]: osdmap e485: 8 total, 8 up, 4 in 2024-03-26T04:11:02.075 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:02.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:01 smithi086 ceph-mon[32965]: pgmap v1064: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 348 GiB / 358 GiB avail; 127 B/s rd, 0 B/s wr, 908 op/s; 41 B/s, 1 objects/s recovering 2024-03-26T04:11:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:01 smithi086 ceph-mon[32965]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:01 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2024-03-26T04:11:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:01 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:02.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:01 smithi086 ceph-mon[32965]: osdmap e485: 8 total, 8 up, 4 in 2024-03-26T04:11:02.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:02.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:02.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:02.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:02.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:02.428 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:02.467 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:02.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:02.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:02.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.593 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:02.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:02.651 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:11:02.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:02.718 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:02.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:02.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:02.909 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:02.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:03.010 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:03.046 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:02 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:03.046 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:02 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:03.052 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:03.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:02 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:11:03.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:03.178 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:03.280 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.326 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:03.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:03.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:03.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:03.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:03.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:03.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:03.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:03.679 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:03.723 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:03.780 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:11:03.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:03.848 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:03.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:03.974 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:03.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [ 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.25727.0:186570 11.6 e485/427 11:6087acb8:::benchmark_data_smithi006_185099_object80893:head v 485'11568, mlcod=485'11563)", 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.634571+0000", 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.050032906000000002, 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.050084469999999999, 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.014 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634571+0000", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634571+0000", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.015 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634571+0000", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.3099999999999999e-07 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634578+0000", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4569999999999999e-06 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634584+0000", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.2780000000000001e-06 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.016 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.634588+0000", 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7149999999999997e-06 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636361+0000", 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001773083 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636381+0000", 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9074999999999999e-05 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.017 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.25727.0:186578 11.e e485/477 11:7c402524:::benchmark_data_smithi006_185099_object80901:head v 485'11662, mlcod=485'11658)", 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.636214+0000", 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.048389430999999997, 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.048494830000000003, 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636214+0000", 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.018 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636214+0000", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636215+0000", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.8100000000000001e-07 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636219+0000", 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.4299999999999999e-06 2024-03-26T04:11:04.019 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636228+0000", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3839999999999997e-06 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.636234+0000", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.5270000000000004e-06 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.638906+0000", 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0026716019999999999 2024-03-26T04:11:04.020 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.638926+0000", 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0264000000000001e-05 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.25727.0:186575 11.6 e485/427 11:63eef4ae:::benchmark_data_smithi006_185099_object80898:head v 485'11569, mlcod=485'11565)", 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.640007+0000", 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.044596992000000002, 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.044740066000000002, 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.021 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640007+0000", 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640007+0000", 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.022 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640007+0000", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.1e-07 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640011+0000", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 3.766e-06 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640016+0000", 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6609999999999997e-06 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.023 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.640022+0000", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.293e-06 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.641080+0000", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001058348 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.641098+0000", 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.7929000000000001e-05 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.024 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.25727.0:186582 11.6 e485/427 11:62154017:::benchmark_data_smithi006_185099_object80905:head v 485'11570, mlcod=485'11565)", 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.643614+0000", 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.040989484999999999, 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.041167492999999999, 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643614+0000", 2024-03-26T04:11:04.025 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643614+0000", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643615+0000", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1799999999999995e-07 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643619+0000", 2024-03-26T04:11:04.026 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.6979999999999997e-06 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643627+0000", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.3710000000000002e-06 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.643632+0000", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.7409999999999998e-06 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645742+0000", 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0021096090000000001 2024-03-26T04:11:04.027 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645761+0000", 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.9460999999999999e-05 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186585 11.0 11:081cdacb:::benchmark_data_smithi006_185099_object80908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.644138+0000", 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.040465452999999998, 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.040683221999999998, 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.028 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186585 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644138+0000", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644138+0000", 2024-03-26T04:11:04.029 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644140+0000", 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.4619999999999999e-06 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644146+0000", 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1510000000000001e-06 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.030 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644157+0000", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.1561e-05 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644165+0000", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.5340000000000002e-06 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644226+0000", 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.1161000000000006e-05 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.031 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644341+0000", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011470499999999999 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644451+0000", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110448 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "op_commit", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.646225+0000", 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.001774295 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.032 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186586 11.7 11:ec901319:::benchmark_data_smithi006_185099_object80909:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.644912+0000", 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.039691847000000002, 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039959815000000003, 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186586 2024-03-26T04:11:04.033 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644912+0000", 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644912+0000", 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.034 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644913+0000", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.1299999999999999e-07 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644917+0000", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.7589999999999997e-06 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644928+0000", 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0451e-05 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.035 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644935+0000", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.6140000000000003e-06 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.646298+0000", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0013630179999999999 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.646405+0000", 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106782 2024-03-26T04:11:04.036 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,4", 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.646510+0000", 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010454999999999999 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.648926+0000", 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0024166859999999999 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186587 11.c 11:324066f4:::benchmark_data_smithi006_185099_object80910:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.037 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.644954+0000", 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.039649816999999997, 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039967079000000003, 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186587 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.038 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644954+0000", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644954+0000", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644954+0000", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.32e-07 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.039 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644957+0000", 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.8270000000000002e-06 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644963+0000", 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9120000000000003e-06 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.644968+0000", 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.3909999999999997e-06 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.040 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645020+0000", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1243999999999999e-05 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645128+0000", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00010814 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645235+0000", 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000106891 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.041 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.648140+0000", 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0029051429999999998 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186588 11.a 11:5ca5692f:::benchmark_data_smithi006_185099_object80911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.645785+0000", 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.038818416000000001, 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.039184179999999999, 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.042 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186588 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645785+0000", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645785+0000", 2024-03-26T04:11:04.043 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645786+0000", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7899999999999997e-07 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645791+0000", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.5140000000000003e-06 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645800+0000", 2024-03-26T04:11:04.044 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.1200000000000008e-06 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645808+0000", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.3159999999999997e-06 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645860+0000", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1894999999999997e-05 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.645971+0000", 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000110775 2024-03-26T04:11:04.045 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 2,4", 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.646090+0000", 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118973 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "sub_op_commit_rec", 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649532+0000", 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.003442578 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.046 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_repop(client.25727.0:186569 11.e e485/477 11:7b3c15fa:::benchmark_data_smithi006_185099_object80892:head v 485'11661, mlcod=485'11654)", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.628218+0000", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.056385165000000001, 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.056807478000000002, 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628218+0000", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628218+0000", 2024-03-26T04:11:04.047 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628219+0000", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.6599999999999996e-07 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628224+0000", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0799999999999996e-06 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628233+0000", 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 9.285e-06 2024-03-26T04:11:04.048 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.628240+0000", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.6050000000000003e-06 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.637354+0000", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.0091139370000000008 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.637374+0000", 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.0466e-05 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.049 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186590 11.0 11:037cbcfb:::benchmark_data_smithi006_185099_object80913:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.647772+0000", 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.036831402999999999, 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.037290038999999997, 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "waiting for sub ops", 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186590 2024-03-26T04:11:04.050 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647772+0000", 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647772+0000", 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.051 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647774+0000", 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.367e-06 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647779+0000", 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.93e-06 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647790+0000", 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.0575e-05 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.052 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647798+0000", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.605e-06 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647857+0000", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.9304e-05 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.647983+0000", 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000126229 2024-03-26T04:11:04.053 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "waiting for subops from 0,2", 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.648110+0000", 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000126718 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186591 11.b 11:df673874:::benchmark_data_smithi006_185099_object80914:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.649646+0000", 2024-03-26T04:11:04.054 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.034957958999999997, 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035459282000000002, 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186591 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649646+0000", 2024-03-26T04:11:04.055 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649646+0000", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649647+0000", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 1.001e-06 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649652+0000", 2024-03-26T04:11:04.056 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.0379999999999999e-06 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649660+0000", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 8.7909999999999993e-06 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649667+0000", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 6.4910000000000002e-06 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649785+0000", 2024-03-26T04:11:04.057 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000117747 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649895+0000", 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.00011051 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "description": "osd_op(client.25727.0:186592 11.f 11:fe7e301c:::benchmark_data_smithi006_185099_object80915:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e485)", 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "initiated_at": "2024-03-26T04:11:02.649687+0000", 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "age": 0.034916326999999997, 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.035456851999999997, 2024-03-26T04:11:04.058 INFO:teuthology.orchestra.run.smithi006.stdout: "continuous": false, 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "type_data": { 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "flag_point": "started", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "client_info": { 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "client": "client.25727", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "client_addr": "172.21.15.6:0/646203996", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "tid": 186592 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "events": [ 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "initiated", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649687+0000", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "header_read", 2024-03-26T04:11:04.059 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649687+0000", 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "throttled", 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649688+0000", 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 4.2e-07 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "all_read", 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649690+0000", 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 2.6860000000000002e-06 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.060 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "dispatched", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649696+0000", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.8880000000000002e-06 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "queued_for_pg", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649701+0000", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 5.1250000000000001e-06 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "reached_pg", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649775+0000", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 7.4010000000000005e-05 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: }, 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: { 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "event": "started", 2024-03-26T04:11:04.061 INFO:teuthology.orchestra.run.smithi006.stdout: "time": "2024-03-26T04:11:02.649894+0000", 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: "duration": 0.000118738 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: ] 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: } 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: ], 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 12 2024-03-26T04:11:04.062 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:11:04.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:03 smithi086 ceph-mon[32965]: pgmap v1066: 158 pgs: 158 active+clean; 1.6 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 785 B/s rd, 0 B/s wr, 1.38k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:04.100 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:03 smithi006 ceph-mon[29446]: pgmap v1066: 158 pgs: 158 active+clean; 1.6 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 785 B/s rd, 0 B/s wr, 1.38k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:03 smithi006 ceph-mon[37501]: pgmap v1066: 158 pgs: 158 active+clean; 1.6 GiB data, 9.1 GiB used, 349 GiB / 358 GiB avail; 785 B/s rd, 0 B/s wr, 1.38k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:04.202 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.247 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:04.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:04.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:04.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:04.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:04.498 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:04.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:04.625 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:04.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:04.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:04.727 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:04.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:04.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:04.893 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:04.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:04.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:04.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:05.031 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:05.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:04 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:04 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:05.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:04 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:11:05.133 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:05.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:05.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:11:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:11:05.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:04 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:11:05.180 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:05.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:05.306 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:05.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:05.407 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:05.446 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:05.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-26T04:11:05.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:05.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.573 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:05.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:05.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:05.738 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:05.788 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:05.840 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:05.903 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:05.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:06.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:06.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.029 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:06.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:05 smithi006 ceph-mon[37501]: pgmap v1067: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s; 38 B/s, 1 objects/s recovering 2024-03-26T04:11:06.075 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:05 smithi006 ceph-mon[37501]: pgmap v1068: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 934 B/s rd, 0 B/s wr, 1.50k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:06.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:05 smithi006 ceph-mon[29446]: pgmap v1067: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s; 38 B/s, 1 objects/s recovering 2024-03-26T04:11:06.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:05 smithi006 ceph-mon[29446]: pgmap v1068: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 934 B/s rd, 0 B/s wr, 1.50k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:05 smithi086 ceph-mon[32965]: pgmap v1067: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 844 B/s rd, 0 B/s wr, 1.36k op/s; 38 B/s, 1 objects/s recovering 2024-03-26T04:11:06.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:05 smithi086 ceph-mon[32965]: pgmap v1068: 158 pgs: 158 active+clean; 1.6 GiB data, 8.8 GiB used, 349 GiB / 358 GiB avail; 934 B/s rd, 0 B/s wr, 1.50k op/s; 42 B/s, 2 objects/s recovering 2024-03-26T04:11:06.087 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:06.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:06.154 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:06.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:06.304 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:06.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:06.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:06.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.431 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:06.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.557 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:06.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:06.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:06.805 INFO:teuthology.orchestra.run.smithi006.stdout:moved 36 / 474 (7.59494%) 2024-03-26T04:11:06.805 INFO:teuthology.orchestra.run.smithi006.stdout:avg 118.5 2024-03-26T04:11:06.805 INFO:teuthology.orchestra.run.smithi006.stdout:stddev 4.60977 -> 14.2039 (expected baseline 9.42735) 2024-03-26T04:11:06.805 INFO:teuthology.orchestra.run.smithi006.stdout:min osd.1 with 113 -> 97 pgs (0.953587 -> 0.818565 * mean) 2024-03-26T04:11:06.805 INFO:teuthology.orchestra.run.smithi006.stdout:max osd.2 with 124 -> 131 pgs (1.04641 -> 1.10549 * mean) 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:oload 110 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:max_change 3 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:max_change_osds 4 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:average_utilization 0.0241 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:overload_utilization 0.0266 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0 weight 1.0000 -> 0.8544 2024-03-26T04:11:06.806 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1 weight 0.8980 -> 0.7682 2024-03-26T04:11:06.810 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:06.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:06.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:06.936 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:06.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:07.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:07.061 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:06 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:06 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:06 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]: dispatch 2024-03-26T04:11:07.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:07.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:07.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:07.314 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:07.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:07.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:07.440 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:07.542 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.568 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:07.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:07.637 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:07.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:07.694 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:07.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:07.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:07.827 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:07.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:07.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:07.951 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:07.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 8.6 GiB used, 349 GiB / 358 GiB avail; 667 B/s rd, 0 B/s wr, 851 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: from='client.15900 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]': finished 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: osdmap e486: 8 total, 8 up, 4 in 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:11:08.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 8.6 GiB used, 349 GiB / 358 GiB avail; 667 B/s rd, 0 B/s wr, 851 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: from='client.15900 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]': finished 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: osdmap e486: 8 total, 8 up, 4 in 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:11:08.029 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:07 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:11:08.052 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:08.054 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-26T04:11:08.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 1.0 2024-03-26T04:11:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: pgmap v1069: 158 pgs: 158 active+clean; 1.5 GiB data, 8.6 GiB used, 349 GiB / 358 GiB avail; 667 B/s rd, 0 B/s wr, 851 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:11:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: from='client.15900 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'0':'55992','1':'50347'}"}]': finished 2024-03-26T04:11:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: osdmap e486: 8 total, 8 up, 4 in 2024-03-26T04:11:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:11:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:07 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:11:08.117 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:08.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:08.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:08.244 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:08.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:08.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:08.373 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:08.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:08.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:08.501 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:08.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:08.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:08.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:08.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:08.729 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.754 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:08.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:08.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:08.882 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:08.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:08.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:09.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:09.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:08 smithi086 ceph-mon[32965]: osdmap e487: 8 total, 8 up, 4 in 2024-03-26T04:11:09.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:09.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:08 smithi006 ceph-mon[37501]: osdmap e487: 8 total, 8 up, 4 in 2024-03-26T04:11:09.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:08 smithi006 ceph-mon[29446]: osdmap e487: 8 total, 8 up, 4 in 2024-03-26T04:11:09.147 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:09.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:09.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:09.288 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:09.390 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:09.430 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:09.532 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:09.558 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:09.660 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:09.696 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:09.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:09.823 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:09.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:09.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:09.967 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:09.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:10.068 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:09 smithi086 ceph-mon[32965]: pgmap v1072: 158 pgs: 5 peering, 153 active+clean; 1.3 GiB data, 8.1 GiB used, 350 GiB / 358 GiB avail; 902 B/s rd, 0 B/s wr, 947 op/s 2024-03-26T04:11:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:09 smithi086 ceph-mon[32965]: osdmap e488: 8 total, 8 up, 4 in 2024-03-26T04:11:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:09 smithi086 ceph-mon[32965]: 11.8 starting backfill to osd.1 from (0'0,0'0] MAX to 485'11824 2024-03-26T04:11:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:09 smithi086 ceph-mon[32965]: 11.a starting backfill to osd.0 from (0'0,0'0] MAX to 485'11855 2024-03-26T04:11:10.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:09 smithi086 ceph-mon[32965]: osdmap e489: 8 total, 8 up, 4 in 2024-03-26T04:11:10.110 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:10.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[37501]: pgmap v1072: 158 pgs: 5 peering, 153 active+clean; 1.3 GiB data, 8.1 GiB used, 350 GiB / 358 GiB avail; 902 B/s rd, 0 B/s wr, 947 op/s 2024-03-26T04:11:10.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[37501]: osdmap e488: 8 total, 8 up, 4 in 2024-03-26T04:11:10.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[37501]: 11.8 starting backfill to osd.1 from (0'0,0'0] MAX to 485'11824 2024-03-26T04:11:10.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[37501]: 11.a starting backfill to osd.0 from (0'0,0'0] MAX to 485'11855 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[37501]: osdmap e489: 8 total, 8 up, 4 in 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[29446]: pgmap v1072: 158 pgs: 5 peering, 153 active+clean; 1.3 GiB data, 8.1 GiB used, 350 GiB / 358 GiB avail; 902 B/s rd, 0 B/s wr, 947 op/s 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[29446]: osdmap e488: 8 total, 8 up, 4 in 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[29446]: 11.8 starting backfill to osd.1 from (0'0,0'0] MAX to 485'11824 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[29446]: 11.a starting backfill to osd.0 from (0'0,0'0] MAX to 485'11855 2024-03-26T04:11:10.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:09 smithi006 ceph-mon[29446]: osdmap e489: 8 total, 8 up, 4 in 2024-03-26T04:11:10.213 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:10.255 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:10.357 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:10.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.415 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:10.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:10.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:10.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:10.692 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:10.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:10.834 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:10.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:10.960 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:10.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:11.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:10 smithi086 ceph-mon[32965]: osdmap e490: 8 total, 8 up, 4 in 2024-03-26T04:11:11.096 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:10 smithi006 ceph-mon[37501]: osdmap e490: 8 total, 8 up, 4 in 2024-03-26T04:11:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:10 smithi006 ceph-mon[29446]: osdmap e490: 8 total, 8 up, 4 in 2024-03-26T04:11:11.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.241 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:11.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:11.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:11.367 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:11.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:11.469 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.523 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:11.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:11.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:11.648 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:11.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:11.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:11.774 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:11.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:11.900 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:11.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:12.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:12.026 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:11 smithi086 ceph-mon[32965]: pgmap v1075: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 5 peering, 145 active+clean; 1.2 GiB data, 7.8 GiB used, 350 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 81 B/s, 3 objects/s recovering 2024-03-26T04:11:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:11 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:11:12.128 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:12.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:11 smithi006 ceph-mon[37501]: pgmap v1075: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 5 peering, 145 active+clean; 1.2 GiB data, 7.8 GiB used, 350 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 81 B/s, 3 objects/s recovering 2024-03-26T04:11:12.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:11 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:11:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:11 smithi006 ceph-mon[29446]: pgmap v1075: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 5 peering, 145 active+clean; 1.2 GiB data, 7.8 GiB used, 350 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s; 81 B/s, 3 objects/s recovering 2024-03-26T04:11:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:11 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-26T04:11:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:12.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.295 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:12.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:12.419 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:12.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:12.545 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:12.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:12.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.687 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:12.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:12.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:12.812 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:12.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:12.937 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:12.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:12.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:13.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:13.063 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:13.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:13.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:13.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:13.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:13.315 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:13.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:13.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:13.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:13.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:13.582 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:13.684 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:13.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.727 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:13.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:13.828 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:13.884 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:13.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:13.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:14.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:13 smithi086 ceph-mon[32965]: pgmap v1077: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 150 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 933 B/s rd, 0 B/s wr, 785 op/s; 406 B/s, 6 objects/s recovering 2024-03-26T04:11:14.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:13 smithi006 ceph-mon[37501]: pgmap v1077: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 150 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 933 B/s rd, 0 B/s wr, 785 op/s; 406 B/s, 6 objects/s recovering 2024-03-26T04:11:14.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:13 smithi006 ceph-mon[29446]: pgmap v1077: 158 pgs: 3 active+clean+remapped, 5 remapped+peering, 150 active+clean; 1.1 GiB data, 7.4 GiB used, 350 GiB / 358 GiB avail; 933 B/s rd, 0 B/s wr, 785 op/s; 406 B/s, 6 objects/s recovering 2024-03-26T04:11:14.151 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:14.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:14.277 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:14.379 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:14.407 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:14.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:14.532 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:14.574 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:14.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:14.658 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:14.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:14.783 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:14.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:14.910 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:14.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:15.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:15.035 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:15.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:15.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:15.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:14 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T04:11:15.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:14 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T04:11:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.177 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:15.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:15.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:15.302 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:14 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-26T04:11:15.404 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:15.448 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:15.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:15.550 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.594 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:15.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:15.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:15.718 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:15.820 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:15.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:15.859 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:15.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:15.961 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:16.004 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:16.105 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:15 smithi006 ceph-mon[37501]: pgmap v1078: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1.0 GiB data, 7.2 GiB used, 350 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 825 op/s; 1124/49875 objects misplaced (2.254%); 8.9 MiB/s, 147 objects/s recovering 2024-03-26T04:11:16.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:15 smithi006 ceph-mon[29446]: pgmap v1078: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1.0 GiB data, 7.2 GiB used, 350 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 825 op/s; 1124/49875 objects misplaced (2.254%); 8.9 MiB/s, 147 objects/s recovering 2024-03-26T04:11:16.146 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:16.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:16.272 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:15 smithi086 ceph-mon[32965]: pgmap v1078: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1.0 GiB data, 7.2 GiB used, 350 GiB / 358 GiB avail; 682 B/s rd, 0 B/s wr, 825 op/s; 1124/49875 objects misplaced (2.254%); 8.9 MiB/s, 147 objects/s recovering 2024-03-26T04:11:16.374 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:16.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:16.515 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:16.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:16.693 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:16.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:16.819 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:16.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:16.921 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:16.958 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:16.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:17.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.099 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:17.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:17.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:17.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:17.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:17.224 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:17.326 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:17.363 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:17.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:17.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.491 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:17.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:17.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:17.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.618 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:17.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:17.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:17.743 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:17.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:17.895 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:17.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:17.996 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.043 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:18.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:18.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:18.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:18.168 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:17 smithi006 ceph-mon[37501]: pgmap v1079: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1005 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 554 B/s rd, 0 B/s wr, 563 op/s; 1124/48897 objects misplaced (2.299%); 7.2 MiB/s, 118 objects/s recovering 2024-03-26T04:11:18.169 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:17 smithi006 ceph-mon[29446]: pgmap v1079: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1005 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 554 B/s rd, 0 B/s wr, 563 op/s; 1124/48897 objects misplaced (2.299%); 7.2 MiB/s, 118 objects/s recovering 2024-03-26T04:11:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:18.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:18.302 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:18.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:17 smithi086 ceph-mon[32965]: pgmap v1079: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 1005 MiB data, 7.1 GiB used, 351 GiB / 358 GiB avail; 554 B/s rd, 0 B/s wr, 563 op/s; 1124/48897 objects misplaced (2.299%); 7.2 MiB/s, 118 objects/s recovering 2024-03-26T04:11:18.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:18.428 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:18.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.554 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:18.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:18.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:18.680 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:18.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:18.822 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:18.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:18.924 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:18.970 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:18.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:18.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:19.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:19.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.097 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:19.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:19.198 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:19.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.223 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:19.224 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:19.324 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:19.349 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:19.451 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:19.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.475 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:19.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:19.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:19.601 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:19.613 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:19.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:19.727 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:19.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:19.829 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:19.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:19.954 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:19.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:19.980 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:19.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:20.081 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:20.104 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:20.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[37501]: pgmap v1080: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 809 MiB data, 6.6 GiB used, 351 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 1124/39516 objects misplaced (2.844%); 6.2 MiB/s, 101 objects/s recovering 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[37501]: osdmap e491: 8 total, 8 up, 4 in 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/32907521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[37501]: osdmap e492: 8 total, 8 up, 4 in 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[29446]: pgmap v1080: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 809 MiB data, 6.6 GiB used, 351 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 1124/39516 objects misplaced (2.844%); 6.2 MiB/s, 101 objects/s recovering 2024-03-26T04:11:20.124 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[29446]: osdmap e491: 8 total, 8 up, 4 in 2024-03-26T04:11:20.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/32907521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[29446]: osdmap e492: 8 total, 8 up, 4 in 2024-03-26T04:11:20.125 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:19 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.206 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:20.230 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:20.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:20.332 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:19 smithi086 ceph-mon[32965]: pgmap v1080: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 809 MiB data, 6.6 GiB used, 351 GiB / 358 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 1124/39516 objects misplaced (2.844%); 6.2 MiB/s, 101 objects/s recovering 2024-03-26T04:11:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:19 smithi086 ceph-mon[32965]: osdmap e491: 8 total, 8 up, 4 in 2024-03-26T04:11:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/32907521' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:19 smithi086 ceph-mon[32965]: osdmap e492: 8 total, 8 up, 4 in 2024-03-26T04:11:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:19 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.356 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:20.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:20.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:20.482 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:20.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:20.607 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:20.628 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:11:20.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.734 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:20.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:20.835 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:20.874 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:20.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:20.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:21.000 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:21.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:21.126 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:21.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:21.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:21.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.252 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:21.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2080122219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:20 smithi086 ceph-mon[32965]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:20 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:11:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:20 smithi086 ceph-mon[32965]: osdmap e493: 8 total, 8 up, 4 in 2024-03-26T04:11:21.354 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:21.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2080122219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:21.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[37501]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:21.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:11:21.380 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[37501]: osdmap e493: 8 total, 8 up, 4 in 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:11:21.381 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:11:21.382 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:11:21.382 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:11:21.382 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:11:21.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2080122219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:21.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[29446]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:21.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:11:21.382 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:20 smithi006 ceph-mon[29446]: osdmap e493: 8 total, 8 up, 4 in 2024-03-26T04:11:21.395 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:21.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:21.497 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:21.539 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:21.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:21.664 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:21.766 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:21.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.808 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:21.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:21.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:21.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:21.950 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:21.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:22.031 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:21 smithi006 ceph-mon[37501]: pgmap v1083: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 687 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 865 op/s; 111/33630 objects misplaced (0.330%); 13 MiB/s, 204 objects/s recovering 2024-03-26T04:11:22.031 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:21 smithi006 ceph-mon[29446]: pgmap v1083: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 687 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 865 op/s; 111/33630 objects misplaced (0.330%); 13 MiB/s, 204 objects/s recovering 2024-03-26T04:11:22.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:22.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:22.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:22.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:22.199 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:22.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:22.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:22.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.335 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:22.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:21 smithi086 ceph-mon[32965]: pgmap v1083: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 687 MiB data, 6.2 GiB used, 351 GiB / 358 GiB avail; 639 B/s rd, 0 B/s wr, 865 op/s; 111/33630 objects misplaced (0.330%); 13 MiB/s, 204 objects/s recovering 2024-03-26T04:11:22.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:22.461 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:22.563 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:22.603 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:22.705 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:22.741 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:22.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:22.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:22.867 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:22.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:22.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:22.985 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:11:22.985 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:11:22.993 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:22.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:23.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:23.130 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:23.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:23.234 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:23.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:23.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:23.399 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:23.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:23.527 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:23.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:23.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.654 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:23.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:23.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.780 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:23.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:23.880 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:23.920 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:23.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:24.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:24.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:23 smithi006 ceph-mon[29446]: pgmap v1085: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 517 MiB data, 5.7 GiB used, 352 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s; 111/25476 objects misplaced (0.436%); 8.2 MiB/s, 130 objects/s recovering 2024-03-26T04:11:24.045 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:23 smithi006 ceph-mon[37501]: pgmap v1085: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 517 MiB data, 5.7 GiB used, 352 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s; 111/25476 objects misplaced (0.436%); 8.2 MiB/s, 130 objects/s recovering 2024-03-26T04:11:24.061 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:24.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:24.082 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-26T04:11:24.101 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-26T04:11:24.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:24.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.251 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:24.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:24.321 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:23 smithi086 ceph-mon[32965]: pgmap v1085: 158 pgs: 2 active+remapped+backfilling, 5 active+clean+remapped, 151 active+clean; 517 MiB data, 5.7 GiB used, 352 GiB / 358 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.30k op/s; 111/25476 objects misplaced (0.436%); 8.2 MiB/s, 130 objects/s recovering 2024-03-26T04:11:24.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:24.376 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:24.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:24.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:24.532 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:24.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:24.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:24.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:24.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:24.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:24.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:24.949 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:24.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:25.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:25.074 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:25.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:25.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:25.200 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:25.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:25.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.327 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:25.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:25.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:25.452 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:25.554 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:25.590 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:25.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:25.692 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:25.741 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:25.760 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:25.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:25.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-mon[32965]: pgmap v1086: 158 pgs: 5 active+clean+remapped, 153 active+clean; 430 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 9.3 MiB/s, 148 objects/s recovering 2024-03-26T04:11:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-mon[32965]: osdmap e494: 8 total, 8 up, 4 in 2024-03-26T04:11:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T04:11:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T04:11:25.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:25.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:25.886 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:25.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:25.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:25.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[29446]: pgmap v1086: 158 pgs: 5 active+clean+remapped, 153 active+clean; 430 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 9.3 MiB/s, 148 objects/s recovering 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[29446]: osdmap e494: 8 total, 8 up, 4 in 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[37501]: pgmap v1086: 158 pgs: 5 active+clean+remapped, 153 active+clean; 430 MiB data, 5.5 GiB used, 352 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 1.01k op/s; 9.3 MiB/s, 148 objects/s recovering 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[37501]: osdmap e494: 8 total, 8 up, 4 in 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T04:11:25.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:25.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:26.011 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:26.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:26.137 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:26.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:26.263 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:26.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:26.389 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:26.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:26.515 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:26.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:26.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:26.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:26.781 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:26.801 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:26.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:26.907 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:26.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:26.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:26.913 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: 1.a scrub starts 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[37501]: 1.a scrub ok 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:26.914 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:26.915 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:26.915 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: 1.a scrub starts 2024-03-26T04:11:26.915 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:26 smithi006 ceph-mon[29446]: 1.a scrub ok 2024-03-26T04:11:27.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:27.032 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T04:11:27.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T04:11:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]: dispatch 2024-03-26T04:11:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: 1.a scrub starts 2024-03-26T04:11:27.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:26 smithi086 ceph-mon[32965]: 1.a scrub ok 2024-03-26T04:11:27.134 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:27.260 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:27.284 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:27.386 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:27.410 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:27.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:11:27.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:27.538 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:27.640 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:27.682 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:27.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:27.808 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:27.906 INFO:teuthology.orchestra.run.smithi006.stdout:moved 0 / 474 (0%) 2024-03-26T04:11:27.906 INFO:teuthology.orchestra.run.smithi006.stdout:avg 118.5 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:stddev 14.5516 -> 14.5516 (expected baseline 9.42735) 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:min osd.1 with 97 -> 97 pgs (0.818565 -> 0.818565 * mean) 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:max osd.2 with 131 -> 131 pgs (1.10549 -> 1.10549 * mean) 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:oload 1000 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:max_change 1 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:max_change_osds 4 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:average_utilization 0.0132 2024-03-26T04:11:27.907 INFO:teuthology.orchestra.run.smithi006.stdout:overload_utilization 0.1320 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[37501]: pgmap v1088: 158 pgs: 5 active+clean+remapped, 153 active+clean; 410 MiB data, 5.3 GiB used, 352 GiB / 358 GiB avail; 778 B/s rd, 0 B/s wr, 718 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]': finished 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[37501]: osdmap e495: 8 total, 8 up, 4 in 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[37501]: 1.3 scrub starts 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[37501]: 1.3 scrub ok 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[29446]: pgmap v1088: 158 pgs: 5 active+clean+remapped, 153 active+clean; 410 MiB data, 5.3 GiB used, 352 GiB / 358 GiB avail; 778 B/s rd, 0 B/s wr, 718 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-26T04:11:27.908 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]': finished 2024-03-26T04:11:27.909 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[29446]: osdmap e495: 8 total, 8 up, 4 in 2024-03-26T04:11:27.909 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[29446]: 1.3 scrub starts 2024-03-26T04:11:27.909 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:27 smithi006 ceph-mon[29446]: 1.3 scrub ok 2024-03-26T04:11:27.909 INFO:teuthology.orchestra.run.smithi006.stderr:no change 2024-03-26T04:11:27.910 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:27.936 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:27.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:28.038 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:28.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.066 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:28.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:27 smithi086 ceph-mon[32965]: pgmap v1088: 158 pgs: 5 active+clean+remapped, 153 active+clean; 410 MiB data, 5.3 GiB used, 352 GiB / 358 GiB avail; 778 B/s rd, 0 B/s wr, 718 op/s; 1.0 MiB/s, 16 objects/s recovering 2024-03-26T04:11:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:27 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "11.2", "id": [0, 4]}]': finished 2024-03-26T04:11:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:27 smithi086 ceph-mon[32965]: osdmap e495: 8 total, 8 up, 4 in 2024-03-26T04:11:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:27 smithi086 ceph-mon[32965]: 1.3 scrub starts 2024-03-26T04:11:28.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:27 smithi086 ceph-mon[32965]: 1.3 scrub ok 2024-03-26T04:11:28.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:28.194 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:28.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:28.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:28.323 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:28.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:28.425 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:28.449 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:28.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:28.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:28.708 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:28.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:28.810 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:28.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[37501]: 6.6 scrub starts 2024-03-26T04:11:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[37501]: 6.6 scrub ok 2024-03-26T04:11:28.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[37501]: osdmap e496: 8 total, 8 up, 4 in 2024-03-26T04:11:28.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[37501]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[29446]: 6.6 scrub starts 2024-03-26T04:11:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[29446]: 6.6 scrub ok 2024-03-26T04:11:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[29446]: osdmap e496: 8 total, 8 up, 4 in 2024-03-26T04:11:28.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:28 smithi006 ceph-mon[29446]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:28.952 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:28.995 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:28.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:28.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:28 smithi086 ceph-mon[32965]: 6.6 scrub starts 2024-03-26T04:11:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:28 smithi086 ceph-mon[32965]: 6.6 scrub ok 2024-03-26T04:11:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:28 smithi086 ceph-mon[32965]: osdmap e496: 8 total, 8 up, 4 in 2024-03-26T04:11:29.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:28 smithi086 ceph-mon[32965]: from='client.25841 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:29.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:29.122 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:29.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:29.248 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:29.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:29.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.375 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:29.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:29.476 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:29.520 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:29.622 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:29.665 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:29.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:29.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:29.790 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: pgmap v1091: 158 pgs: 5 active+clean+remapped, 153 active+clean; 174 MiB data, 4.7 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 914 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: 11.2 starting backfill to osd.4 from (0'0,0'0] MAX to 494'13083 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: osdmap e497: 8 total, 8 up, 4 in 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[37501]: osdmap e498: 8 total, 8 up, 4 in 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: pgmap v1091: 158 pgs: 5 active+clean+remapped, 153 active+clean; 174 MiB data, 4.7 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 914 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:11:29.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: 11.2 starting backfill to osd.4 from (0'0,0'0] MAX to 494'13083 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: osdmap e497: 8 total, 8 up, 4 in 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T04:11:29.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:29 smithi006 ceph-mon[29446]: osdmap e498: 8 total, 8 up, 4 in 2024-03-26T04:11:29.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:29.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:29.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:29.988 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: pgmap v1091: 158 pgs: 5 active+clean+remapped, 153 active+clean; 174 MiB data, 4.7 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 914 op/s; 1.1 MiB/s, 17 objects/s recovering 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: 11.2 starting backfill to osd.4 from (0'0,0'0] MAX to 494'13083 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: osdmap e497: 8 total, 8 up, 4 in 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T04:11:29.989 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:29 smithi086 ceph-mon[32965]: osdmap e498: 8 total, 8 up, 4 in 2024-03-26T04:11:30.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:30.041 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:30.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:30.168 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:30.270 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:30.396 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:30.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:30.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:30.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:30.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:30.675 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:30.680 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Removed 105674 objects 2024-03-26T04:11:30.680 INFO:tasks.radosbench.radosbench.0.smithi006.stdout:Clean up completed and total clean up time :107.551 2024-03-26T04:11:30.777 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:30.801 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:30.903 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:30.926 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:30.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:30.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:31.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:31.056 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:30 smithi086 ceph-mon[32965]: 1.2 scrub starts 2024-03-26T04:11:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:30 smithi086 ceph-mon[32965]: 1.2 scrub ok 2024-03-26T04:11:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:30 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T04:11:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:30 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T04:11:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:30 smithi086 ceph-mon[32965]: osdmap e499: 8 total, 8 up, 4 in 2024-03-26T04:11:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[37501]: 1.2 scrub starts 2024-03-26T04:11:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[37501]: 1.2 scrub ok 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[37501]: osdmap e499: 8 total, 8 up, 4 in 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[29446]: 1.2 scrub starts 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[29446]: 1.2 scrub ok 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T04:11:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:30 smithi006 ceph-mon[29446]: osdmap e499: 8 total, 8 up, 4 in 2024-03-26T04:11:31.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:31.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.182 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:31.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:31.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.308 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:31.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:31.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:31.433 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:31.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:31.562 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:31.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:31.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:31.688 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:31.790 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:31.815 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:31.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:31.945 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:31.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:32.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:32.075 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:32.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:31 smithi086 ceph-mon[32965]: pgmap v1094: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 80 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T04:11:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:31 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T04:11:32.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:31 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T04:11:32.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[29446]: pgmap v1094: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 80 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T04:11:32.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T04:11:32.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T04:11:32.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[37501]: pgmap v1094: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 80 MiB data, 4.4 GiB used, 353 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-26T04:11:32.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T04:11:32.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:31 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T04:11:32.177 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.203 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:32.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:32.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:32.329 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:32.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:32.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:32.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:32.588 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:32.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:32.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:32.717 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:32.819 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:32.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.847 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:32.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:32.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:32.975 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:32.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:33.076 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:32 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T04:11:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:32 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T04:11:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:32 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T04:11:33.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:32 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T04:11:33.102 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T04:11:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T04:11:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:32 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T04:11:33.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:33.232 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:33.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:33.362 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:33.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:33.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:33.492 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:33.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:33.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:33.622 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:33.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:33.632 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:11:33.632 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:11:33.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:33.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:33.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:33.878 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:33.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:33.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[37501]: pgmap v1096: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 29 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 511 op/s 2024-03-26T04:11:33.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[37501]: 1.7 scrub starts 2024-03-26T04:11:33.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[37501]: 1.7 scrub ok 2024-03-26T04:11:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[29446]: pgmap v1096: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 29 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 511 op/s 2024-03-26T04:11:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[29446]: 1.7 scrub starts 2024-03-26T04:11:33.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:33 smithi006 ceph-mon[29446]: 1.7 scrub ok 2024-03-26T04:11:33.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:34.008 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:33 smithi086 ceph-mon[32965]: pgmap v1096: 158 pgs: 1 remapped+peering, 5 active+clean+remapped, 152 active+clean; 29 MiB data, 4.2 GiB used, 353 GiB / 358 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 511 op/s 2024-03-26T04:11:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:33 smithi086 ceph-mon[32965]: 1.7 scrub starts 2024-03-26T04:11:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:33 smithi086 ceph-mon[32965]: 1.7 scrub ok 2024-03-26T04:11:34.110 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.136 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:34.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:34.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:34.260 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:34.346 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-26T04:11:34.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-26T04:11:34.362 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:34.413 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:34.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:34.484 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-26T04:11:34.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:34.543 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:34.582 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:34.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:34.672 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:34.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:34.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:34.802 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:34.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:34.931 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:34.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:35.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.062 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:35.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:35.163 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:35.192 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:35.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:35.209 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-26T04:11:35.209 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 0.05 2024-03-26T04:11:35.295 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:35.318 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:35.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:35.420 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.445 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:35.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:35.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:35.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:35.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:35.676 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:35.705 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:35.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:35.810 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:35 smithi006 ceph-mon[29446]: pgmap v1097: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 463 op/s; 597 KiB/s, 9 objects/s recovering 2024-03-26T04:11:35.810 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:35 smithi006 ceph-mon[37501]: pgmap v1097: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 463 op/s; 597 KiB/s, 9 objects/s recovering 2024-03-26T04:11:35.835 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:35.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:35 smithi086 ceph-mon[32965]: pgmap v1097: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 853 B/s rd, 0 B/s wr, 463 op/s; 597 KiB/s, 9 objects/s recovering 2024-03-26T04:11:35.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:35.838 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:11:35.838 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:11:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:35.854 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:11:35.883 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:11:35.901 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:35.909 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:11:35.931 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:11:35.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:35.952 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:11:35.963 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:35.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:35.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:35.973 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:11:35.974 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:11:35.974 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:11:35.974 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:11:35.974 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:11:35.975 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:11:35.975 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:11:35.975 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:11:35.975 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:11:36.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:36.091 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:36.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:36.219 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:36.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:36.346 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:36.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:36.474 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:36.575 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:36.601 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:36.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:36.703 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:36.726 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:36.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:36.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:36.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T04:11:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T04:11:36.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3332674409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T04:11:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T04:11:36.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:36 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3332674409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:36.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:36.985 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:36.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:36.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:37.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:36 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T04:11:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:36 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T04:11:37.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:36 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3332674409' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:37.115 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:37.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:37.244 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:37.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:37.374 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:37.477 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:37.502 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:37.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:37.627 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:37.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:37.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:37.756 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:37.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:37.858 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:37.883 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:37.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:37.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:38.007 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:38.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:37 smithi086 ceph-mon[32965]: pgmap v1098: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 679 B/s rd, 0 B/s wr, 198 op/s; 475 KiB/s, 7 objects/s recovering 2024-03-26T04:11:38.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:37 smithi086 ceph-mon[32965]: 5.12 scrub starts 2024-03-26T04:11:38.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:37 smithi086 ceph-mon[32965]: 5.12 scrub ok 2024-03-26T04:11:38.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:38.132 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:38.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[37501]: pgmap v1098: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 679 B/s rd, 0 B/s wr, 198 op/s; 475 KiB/s, 7 objects/s recovering 2024-03-26T04:11:38.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[37501]: 5.12 scrub starts 2024-03-26T04:11:38.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[37501]: 5.12 scrub ok 2024-03-26T04:11:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[29446]: pgmap v1098: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 679 B/s rd, 0 B/s wr, 198 op/s; 475 KiB/s, 7 objects/s recovering 2024-03-26T04:11:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[29446]: 5.12 scrub starts 2024-03-26T04:11:38.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:37 smithi006 ceph-mon[29446]: 5.12 scrub ok 2024-03-26T04:11:38.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.265 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:38.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:38.366 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:38.389 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:38.491 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:38.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:38.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:38.645 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:38.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:38.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:38.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.771 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:38.872 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:38 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T04:11:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:38 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T04:11:38.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:38 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T04:11:38.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:38 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T04:11:38.896 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:38.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:38.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:39.021 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:38 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T04:11:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:38 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T04:11:39.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:39.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.147 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:39.248 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:39.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:39.403 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:39.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:39.505 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.529 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:39.630 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:39.656 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:39.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:39.786 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:39.888 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:39.911 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:39.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:39.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:40.003 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T04:11:40.003 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T04:11:40.003 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[37501]: pgmap v1099: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 174 op/s; 418 KiB/s, 6 objects/s recovering 2024-03-26T04:11:40.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T04:11:40.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T04:11:40.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:39 smithi006 ceph-mon[29446]: pgmap v1099: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 174 op/s; 418 KiB/s, 6 objects/s recovering 2024-03-26T04:11:40.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:40.041 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:39 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T04:11:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:39 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T04:11:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:39 smithi086 ceph-mon[32965]: pgmap v1099: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 174 op/s; 418 KiB/s, 6 objects/s recovering 2024-03-26T04:11:40.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:40.166 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:40.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:40.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:40.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:40.310 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:40.325 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:11:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:11:40.325 INFO:teuthology.orchestra.run.smithi006.stdout: "num_ops": 0 2024-03-26T04:11:40.325 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:11:40.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:40.421 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:40.523 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:40.544 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-26T04:11:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.548 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:40.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:40.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:40.677 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:40.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:40.805 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:40.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:40.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:40.906 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:40.932 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:40.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:41.034 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:41.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.059 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:41.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:41.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:41.187 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:41.289 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:41.314 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:41.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:41.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:41.445 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:41.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:41.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:41.574 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:41.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.575 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:41.675 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:41.701 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:41.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:41.714 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:11:41.717 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-26T04:11:41.803 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:41.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:41.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:41.929 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:41.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[37501]: pgmap v1100: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 127 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:41.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T04:11:41.944 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T04:11:41.945 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:41.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[29446]: pgmap v1100: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 127 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:41.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T04:11:41.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T04:11:41.945 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:41 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:41.954 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:41.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:41.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:42.056 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:42.081 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:41 smithi086 ceph-mon[32965]: pgmap v1100: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 127 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:41 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T04:11:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:41 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T04:11:42.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:41 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:11:42.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:42.212 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:42.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:42.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:42.338 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:42.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:42.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:42.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.469 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:42.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:42.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:42.598 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:42.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:42.699 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:42.726 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:42.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:42.856 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:42.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:42.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[37501]: 3.12 scrub starts 2024-03-26T04:11:42.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[37501]: 3.12 scrub ok 2024-03-26T04:11:42.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[37501]: osdmap e500: 8 total, 8 up, 4 in 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[29446]: 3.12 scrub starts 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[29446]: 3.12 scrub ok 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:42.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:42 smithi006 ceph-mon[29446]: osdmap e500: 8 total, 8 up, 4 in 2024-03-26T04:11:42.958 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:42.982 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:42.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:43.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:42 smithi086 ceph-mon[32965]: 3.12 scrub starts 2024-03-26T04:11:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:42 smithi086 ceph-mon[32965]: 3.12 scrub ok 2024-03-26T04:11:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:42 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-26T04:11:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:42 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2921070745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:11:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:42 smithi086 ceph-mon[32965]: osdmap e500: 8 total, 8 up, 4 in 2024-03-26T04:11:43.109 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:moved 0 / 474 (0%) 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:avg 118.5 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:stddev 14.5516 -> 14.5516 (expected baseline 9.42735) 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:min osd.1 with 97 -> 97 pgs (0.818565 -> 0.818565 * mean) 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:max osd.2 with 131 -> 131 pgs (1.10549 -> 1.10549 * mean) 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:11:43.180 INFO:teuthology.orchestra.run.smithi006.stdout:oload 1000 2024-03-26T04:11:43.181 INFO:teuthology.orchestra.run.smithi006.stdout:max_change 0.05 2024-03-26T04:11:43.181 INFO:teuthology.orchestra.run.smithi006.stdout:max_change_osds 4 2024-03-26T04:11:43.181 INFO:teuthology.orchestra.run.smithi006.stdout:average_utilization 0.0115 2024-03-26T04:11:43.181 INFO:teuthology.orchestra.run.smithi006.stdout:overload_utilization 0.1145 2024-03-26T04:11:43.182 INFO:teuthology.orchestra.run.smithi006.stderr:no change 2024-03-26T04:11:43.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.239 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:43.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:43.340 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:43.363 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:43.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:43.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:43.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.494 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:43.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:43.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:43.618 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:43.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:43.720 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:43.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[37501]: pgmap v1102: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 46 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[29446]: pgmap v1102: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 46 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T04:11:43.823 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:43 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T04:11:43.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:43.848 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-03-26T04:11:43.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd test-reweight-by-utilization 1000 3.0 2024-03-26T04:11:43.873 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:43.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:43.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:43.975 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:43.998 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:43.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:43 smithi086 ceph-mon[32965]: pgmap v1102: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 B/s wr, 46 op/s; 358 KiB/s, 5 objects/s recovering 2024-03-26T04:11:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:43 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T04:11:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:43 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T04:11:44.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:44.128 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:44.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:44.258 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:44.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:44.360 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:44.485 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:44.509 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:44.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:44.511 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:44.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:44.633 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:44.642 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:44.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:44.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd set noscrub 2024-03-26T04:11:44.744 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:44.768 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:44.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:44.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:44.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T04:11:44.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: from='client.25859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 4.9 scrub starts 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 4.9 scrub ok 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[37501]: osdmap e501: 8 total, 8 up, 4 in 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T04:11:44.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: from='client.25859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 4.9 scrub starts 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 4.9 scrub ok 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T04:11:44.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:44 smithi006 ceph-mon[29446]: osdmap e501: 8 total, 8 up, 4 in 2024-03-26T04:11:44.893 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:44.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:44.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:44.995 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.020 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:45.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: from='client.25859 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 4.9 scrub starts 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 4.9 scrub ok 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T04:11:45.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:44 smithi086 ceph-mon[32965]: osdmap e501: 8 total, 8 up, 4 in 2024-03-26T04:11:45.122 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:45.146 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:45.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:45.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:45.276 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:45.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:45.406 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:45.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:45.535 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:45.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:45.637 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2 2024-03-26T04:11:45.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.661 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:45.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.2' 2024-03-26T04:11:45.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:45.790 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:45.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[37501]: pgmap v1103: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[29446]: pgmap v1103: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T04:11:45.829 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:45 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T04:11:45.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:45.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:45 smithi086 ceph-mon[32965]: pgmap v1103: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:11:45.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:45 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T04:11:45.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:45 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T04:11:45.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1 2024-03-26T04:11:45.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:45.917 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:45.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.1' 2024-03-26T04:11:46.018 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:46.041 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:46.143 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:46.167 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:46.266 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:46.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4 2024-03-26T04:11:46.296 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:46.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.4' 2024-03-26T04:11:46.398 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:46.423 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:46.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:46.533 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:46.534 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:46.554 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:46.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0 2024-03-26T04:11:46.679 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.0' 2024-03-26T04:11:46.752 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'unique_pool_4' removed 2024-03-26T04:11:46.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3 2024-03-26T04:11:46.798 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[37501]: 4.3 deep-scrub starts 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[37501]: 4.3 deep-scrub ok 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3433796293' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[29446]: 4.3 deep-scrub starts 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[29446]: 4.3 deep-scrub ok 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3433796293' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:46.799 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:46 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:46.818 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: docker: command not found 2024-03-26T04:11:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi006 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.3' 2024-03-26T04:11:46.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:46.948 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:46.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:47.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:47.077 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:47.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:47.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:46 smithi086 ceph-mon[32965]: 4.3 deep-scrub starts 2024-03-26T04:11:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:46 smithi086 ceph-mon[32965]: 4.3 deep-scrub ok 2024-03-26T04:11:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:46 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3433796293' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:46 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:11:47.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5 2024-03-26T04:11:47.211 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:47.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.5' 2024-03-26T04:11:47.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6 2024-03-26T04:11:47.338 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:47.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:47.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.6' 2024-03-26T04:11:47.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7 2024-03-26T04:11:47.445 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-26T04:11:47.455 INFO:tasks.thrashosds:joining thrashosds 2024-03-26T04:11:47.463 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: docker: command not found 2024-03-26T04:11:47.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:11:47.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi086 with status 1: 'sudo docker kill -s 1 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd.7' 2024-03-26T04:11:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:47 smithi086 ceph-mon[32965]: pgmap v1105: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:11:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:11:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:47 smithi086 ceph-mon[32965]: osdmap e502: 8 total, 8 up, 4 in 2024-03-26T04:11:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:47 smithi086 ceph-mon[32965]: 6.3 scrub starts 2024-03-26T04:11:48.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:47 smithi086 ceph-mon[32965]: 6.3 scrub ok 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[37501]: pgmap v1105: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[37501]: osdmap e502: 8 total, 8 up, 4 in 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[37501]: 6.3 scrub starts 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[37501]: 6.3 scrub ok 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[29446]: pgmap v1105: 158 pgs: 5 active+clean+remapped, 153 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:11:48.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[29446]: osdmap e502: 8 total, 8 up, 4 in 2024-03-26T04:11:48.124 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[29446]: 6.3 scrub starts 2024-03-26T04:11:48.124 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:47 smithi006 ceph-mon[29446]: 6.3 scrub ok 2024-03-26T04:11:48.249 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2024-03-26T04:11:48.249 INFO:teuthology.orchestra.run.smithi006.stdout: "ops": [], 2024-03-26T04:11:48.249 INFO:teuthology.orchestra.run.smithi006.stdout: "complaint_time": 30, 2024-03-26T04:11:48.249 INFO:teuthology.orchestra.run.smithi006.stdout: "num_blocked_ops": 0 2024-03-26T04:11:48.250 INFO:teuthology.orchestra.run.smithi006.stdout:} 2024-03-26T04:11:49.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:48 smithi006 ceph-mon[37501]: 6.18 deep-scrub starts 2024-03-26T04:11:49.038 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:48 smithi006 ceph-mon[37501]: 6.18 deep-scrub ok 2024-03-26T04:11:49.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:48 smithi006 ceph-mon[29446]: 6.18 deep-scrub starts 2024-03-26T04:11:49.038 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:48 smithi006 ceph-mon[29446]: 6.18 deep-scrub ok 2024-03-26T04:11:49.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:48 smithi086 ceph-mon[32965]: 6.18 deep-scrub starts 2024-03-26T04:11:49.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:48 smithi086 ceph-mon[32965]: 6.18 deep-scrub ok 2024-03-26T04:11:49.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: pgmap v1107: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:11:49.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T04:11:49.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T04:11:49.877 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 4.14 scrub starts 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: 4.14 scrub ok 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[37501]: osdmap e503: 8 total, 8 up, 4 in 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: pgmap v1107: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 4.14 scrub starts 2024-03-26T04:11:49.878 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: 4.14 scrub ok 2024-03-26T04:11:49.879 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:49 smithi006 ceph-mon[29446]: osdmap e503: 8 total, 8 up, 4 in 2024-03-26T04:11:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: pgmap v1107: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 354 GiB / 358 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:11:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T04:11:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T04:11:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T04:11:50.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T04:11:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 4.14 scrub starts 2024-03-26T04:11:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: 4.14 scrub ok 2024-03-26T04:11:50.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:49 smithi086 ceph-mon[32965]: osdmap e503: 8 total, 8 up, 4 in 2024-03-26T04:11:51.344 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[37501]: pgmap v1109: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[29446]: pgmap v1109: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:11:51.830 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:51 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:11:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:51 smithi086 ceph-mon[32965]: pgmap v1109: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 682 B/s rd, 0 op/s 2024-03-26T04:11:52.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:51 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:11:52.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:51 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:11:52.391 INFO:teuthology.orchestra.run.smithi006.stdout:osd.0: {} 2024-03-26T04:11:52.411 INFO:teuthology.orchestra.run.smithi006.stdout:osd.1: {} 2024-03-26T04:11:52.431 INFO:teuthology.orchestra.run.smithi006.stdout:osd.2: {} 2024-03-26T04:11:52.450 INFO:teuthology.orchestra.run.smithi006.stdout:osd.3: {} 2024-03-26T04:11:52.471 INFO:teuthology.orchestra.run.smithi006.stdout:osd.4: {} 2024-03-26T04:11:52.491 INFO:teuthology.orchestra.run.smithi006.stdout:osd.5: {} 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:moved 0 / 426 (0%) 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:avg 106.5 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:stddev 15.7877 -> 15.7877 (expected baseline 8.93728) 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:min osd.1 with 83 -> 83 pgs (0.779343 -> 0.779343 * mean) 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:max osd.4 with 122 -> 122 pgs (1.14554 -> 1.14554 * mean) 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:oload 1000 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:max_change 3 2024-03-26T04:11:52.506 INFO:teuthology.orchestra.run.smithi006.stdout:max_change_osds 4 2024-03-26T04:11:52.507 INFO:teuthology.orchestra.run.smithi006.stdout:average_utilization 0.0116 2024-03-26T04:11:52.507 INFO:teuthology.orchestra.run.smithi006.stdout:overload_utilization 0.1155 2024-03-26T04:11:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:no change 2024-03-26T04:11:52.515 INFO:teuthology.orchestra.run.smithi006.stdout:osd.6: {} 2024-03-26T04:11:52.538 INFO:teuthology.orchestra.run.smithi006.stdout:osd.7: {} 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-26T04:11:52.540 INFO:teuthology.orchestra.run.smithi006.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-26T04:11:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:11:53.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: osdmap e504: 8 total, 8 up, 4 in 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T04:11:53.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:52 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1304338366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: osdmap e504: 8 total, 8 up, 4 in 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1304338366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: osdmap e504: 8 total, 8 up, 4 in 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T04:11:53.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:52 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1304338366' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-26T04:11:53.166 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:53.764 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is set 2024-03-26T04:11:53.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: pgmap v1111: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: from='client.25871 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2208487561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T04:11:53.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: pgmap v1111: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:11:53.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: from='client.25871 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:53.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T04:11:53.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T04:11:53.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2208487561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:53.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:53 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:54.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: pgmap v1111: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: from='client.25871 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2208487561' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:54.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:53 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-26T04:11:54.390 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:11:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:11:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:54 smithi086 ceph-mon[32965]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:54 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:11:55.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:54 smithi086 ceph-mon[32965]: osdmap e505: 8 total, 8 up, 4 in 2024-03-26T04:11:55.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:54 smithi086 ceph-mon[32965]: 6.1e scrub starts 2024-03-26T04:11:55.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:54 smithi086 ceph-mon[32965]: 6.1e scrub ok 2024-03-26T04:11:55.118 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[37501]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:55.118 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:11:55.118 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[37501]: osdmap e505: 8 total, 8 up, 4 in 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[37501]: 6.1e scrub starts 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[37501]: 6.1e scrub ok 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[29446]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[29446]: osdmap e505: 8 total, 8 up, 4 in 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[29446]: 6.1e scrub starts 2024-03-26T04:11:55.119 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:54 smithi006 ceph-mon[29446]: 6.1e scrub ok 2024-03-26T04:11:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:55 smithi086 ceph-mon[32965]: pgmap v1113: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:55 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:11:55.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:55 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:11:55.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:55.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:11:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:11:56.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[37501]: pgmap v1113: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[29446]: pgmap v1113: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:11:56.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:11:56.513 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:11:56.514 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:11:57.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:57.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:56 smithi086 ceph-mon[32965]: osdmap e506: 8 total, 8 up, 4 in 2024-03-26T04:11:57.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:56 smithi006 ceph-mon[37501]: osdmap e506: 8 total, 8 up, 4 in 2024-03-26T04:11:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:11:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:56 smithi006 ceph-mon[29446]: osdmap e506: 8 total, 8 up, 4 in 2024-03-26T04:11:57.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-26T04:11:57.551 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset noscrub 2024-03-26T04:11:57.559 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:11:57.806 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:57.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:57 smithi006 ceph-mon[37501]: pgmap v1115: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:57.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:57 smithi006 ceph-mon[29446]: pgmap v1115: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:57.964 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:57 smithi086 ceph-mon[32965]: pgmap v1115: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:11:58.666 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T04:11:58.666 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-26T04:11:58.666 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd in 5 2024-03-26T04:11:59.253 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:11:59.480 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:11:59.480 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[29446]: pgmap v1116: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 791 B/s rd, 0 op/s 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[29446]: osdmap e507: 8 total, 8 up, 4 in 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2325067535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[37501]: pgmap v1116: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 791 B/s rd, 0 op/s 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[37501]: osdmap e507: 8 total, 8 up, 4 in 2024-03-26T04:11:59.839 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2325067535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:11:59.840 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:11:59 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:12:00.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:59 smithi086 ceph-mon[32965]: pgmap v1116: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 791 B/s rd, 0 op/s 2024-03-26T04:12:00.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:59 smithi086 ceph-mon[32965]: osdmap e507: 8 total, 8 up, 4 in 2024-03-26T04:12:00.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:59 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2325067535' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:12:00.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:11:59 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-26T04:12:00.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-26T04:12:00.388 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:00.639 INFO:teuthology.orchestra.run.smithi006.stderr:noscrub is unset 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[37501]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[37501]: osdmap e508: 8 total, 8 up, 4 in 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[29446]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:00 smithi006 ceph-mon[29446]: osdmap e508: 8 total, 8 up, 4 in 2024-03-26T04:12:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:00 smithi086 ceph-mon[32965]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-26T04:12:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:00 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-26T04:12:01.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:00 smithi086 ceph-mon[32965]: osdmap e508: 8 total, 8 up, 4 in 2024-03-26T04:12:01.450 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[29446]: pgmap v1118: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[37501]: pgmap v1118: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T04:12:01.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:01 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T04:12:01.993 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:02.069 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:01 smithi086 ceph-mon[32965]: pgmap v1118: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 793 B/s rd, 0 op/s 2024-03-26T04:12:02.069 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:01 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T04:12:02.069 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:01 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T04:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:02.206 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:02.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:12:02.236+0000 7fd465f8b700 -1 mon.a@0(leader).osd e508 definitely_dead 0 2024-03-26T04:12:02.822 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.5. 2024-03-26T04:12:02.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:12:02.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:12:02.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T04:12:02.995 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:12:02.995 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:12:02.995 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:02 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T04:12:02.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:02 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:12:02.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:02 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:12:02.998 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:02 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-26T04:12:03.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-26T04:12:03.280 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: pgmap v1120: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: osdmap e509: 8 total, 8 up, 5 in 2024-03-26T04:12:04.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T04:12:04.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:03 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T04:12:04.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: pgmap v1120: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: osdmap e509: 8 total, 8 up, 5 in 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: pgmap v1120: 142 pgs: 5 active+clean+remapped, 137 active+clean; 582 KiB data, 4.1 GiB used, 353 GiB / 358 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3466634172' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-26T04:12:04.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: osdmap e509: 8 total, 8 up, 5 in 2024-03-26T04:12:04.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T04:12:04.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:03 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T04:12:04.611 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:04.653 INFO:teuthology.orchestra.run.smithi006.stderr:nodeep-scrub is unset 2024-03-26T04:12:04.700 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-26T04:12:04.701 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T04:12:04.701 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-03-26T04:12:04.701 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd in 7 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: osdmap e510: 8 total, 8 up, 5 in 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/444729470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: 3.4 scrub starts 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: 3.4 scrub ok 2024-03-26T04:12:04.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[29446]: osdmap e511: 8 total, 8 up, 5 in 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: osdmap e510: 8 total, 8 up, 5 in 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/444729470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: 3.4 scrub starts 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: 3.4 scrub ok 2024-03-26T04:12:04.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:04.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:12:04.892 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:04 smithi006 ceph-mon[37501]: osdmap e511: 8 total, 8 up, 5 in 2024-03-26T04:12:05.035 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: osdmap e510: 8 total, 8 up, 5 in 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/444729470' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: 3.4 scrub starts 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: 3.4 scrub ok 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-26T04:12:05.036 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:04 smithi086 ceph-mon[32965]: osdmap e511: 8 total, 8 up, 5 in 2024-03-26T04:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:05.068 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:05.292 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:05.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-26T04:12:05.972 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:06.004 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: pgmap v1123: 142 pgs: 2 remapped+peering, 40 peering, 5 unknown, 3 active+clean+remapped, 92 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: Health check failed: Reduced data availability: 14 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:06.005 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:05 smithi086 ceph-mon[32965]: osdmap e512: 8 total, 8 up, 5 in 2024-03-26T04:12:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: pgmap v1123: 142 pgs: 2 remapped+peering, 40 peering, 5 unknown, 3 active+clean+remapped, 92 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: Health check failed: Reduced data availability: 14 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[37501]: osdmap e512: 8 total, 8 up, 5 in 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: pgmap v1123: 142 pgs: 2 remapped+peering, 40 peering, 5 unknown, 3 active+clean+remapped, 92 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: Health check failed: Reduced data availability: 14 pgs inactive, 21 pgs peering (PG_AVAILABILITY) 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T04:12:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:12:06.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:12:06.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:06.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:05 smithi006 ceph-mon[29446]: osdmap e512: 8 total, 8 up, 5 in 2024-03-26T04:12:07.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:06 smithi086 ceph-mon[32965]: pgmap v1125: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:07.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:06 smithi086 ceph-mon[32965]: pgmap v1126: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T04:12:07.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:07.021 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:06 smithi086 ceph-mon[32965]: osdmap e513: 8 total, 8 up, 5 in 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[29446]: pgmap v1125: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[29446]: pgmap v1126: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[29446]: osdmap e513: 8 total, 8 up, 5 in 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[37501]: pgmap v1125: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 1.0 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[37501]: pgmap v1126: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:07.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:06 smithi006 ceph-mon[37501]: osdmap e513: 8 total, 8 up, 5 in 2024-03-26T04:12:07.565 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:07.565 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:07.639 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:12:07.229+0000 7fd465f8b700 -1 mon.a@0(leader).osd e513 definitely_dead 0 2024-03-26T04:12:07.669 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.7. 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: 1.a scrub starts 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: 1.a scrub ok 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3835175547' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.000 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[37501]: osdmap e514: 8 total, 8 up, 6 in 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: 1.a scrub starts 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: 1.a scrub ok 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3835175547' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T04:12:08.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:07 smithi006 ceph-mon[29446]: osdmap e514: 8 total, 8 up, 6 in 2024-03-26T04:12:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:12:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:12:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: 1.a scrub starts 2024-03-26T04:12:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: 1.a scrub ok 2024-03-26T04:12:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3835175547' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["7"]}]: dispatch 2024-03-26T04:12:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["7"]}]': finished 2024-03-26T04:12:08.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:07 smithi086 ceph-mon[32965]: osdmap e514: 8 total, 8 up, 6 in 2024-03-26T04:12:08.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-26T04:12:08.301 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-03-26T04:12:08.301 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T04:12:08.301 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-26T04:12:08.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd in 3 2024-03-26T04:12:08.485 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:08.556 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:08.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:08 smithi086 ceph-mon[32965]: pgmap v1129: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 637 B/s rd, 0 op/s; 1.1 KiB/s, 33 objects/s recovering 2024-03-26T04:12:08.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:08 smithi086 ceph-mon[32965]: 4.1d deep-scrub starts 2024-03-26T04:12:08.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:08 smithi086 ceph-mon[32965]: 4.1d deep-scrub ok 2024-03-26T04:12:08.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:08 smithi086 ceph-mon[32965]: osdmap e515: 8 total, 8 up, 6 in 2024-03-26T04:12:08.970 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[37501]: pgmap v1129: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 637 B/s rd, 0 op/s; 1.1 KiB/s, 33 objects/s recovering 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[37501]: 4.1d deep-scrub starts 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[37501]: 4.1d deep-scrub ok 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[37501]: osdmap e515: 8 total, 8 up, 6 in 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[29446]: pgmap v1129: 142 pgs: 1 active+undersized+remapped, 2 remapped+peering, 40 peering, 5 unknown, 94 active+clean; 5.9 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 637 B/s rd, 0 op/s; 1.1 KiB/s, 33 objects/s recovering 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[29446]: 4.1d deep-scrub starts 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[29446]: 4.1d deep-scrub ok 2024-03-26T04:12:08.971 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:08 smithi006 ceph-mon[29446]: osdmap e515: 8 total, 8 up, 6 in 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[37501]: 5.19 deep-scrub starts 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[37501]: 5.19 deep-scrub ok 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[37501]: osdmap e516: 8 total, 8 up, 6 in 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[29446]: 5.19 deep-scrub starts 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[29446]: 5.19 deep-scrub ok 2024-03-26T04:12:09.994 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:09 smithi006 ceph-mon[29446]: osdmap e516: 8 total, 8 up, 6 in 2024-03-26T04:12:10.131 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:09 smithi086 ceph-mon[32965]: 5.19 deep-scrub starts 2024-03-26T04:12:10.132 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:09 smithi086 ceph-mon[32965]: 5.19 deep-scrub ok 2024-03-26T04:12:10.132 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:09 smithi086 ceph-mon[32965]: osdmap e516: 8 total, 8 up, 6 in 2024-03-26T04:12:10.179 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:10.179 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:10.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-26T04:12:10.843 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:12:10.452+0000 7fd465f8b700 -1 mon.a@0(leader).osd e516 definitely_dead 0 2024-03-26T04:12:10.873 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.3. 2024-03-26T04:12:11.023 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T04:12:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T04:12:11.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: pgmap v1132: 142 pgs: 1 active+undersized+remapped, 25 peering, 116 active+clean; 315 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 198 KiB/s, 36 objects/s recovering 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 21 pgs peering) 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2949403004' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:10 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: pgmap v1132: 142 pgs: 1 active+undersized+remapped, 25 peering, 116 active+clean; 315 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 198 KiB/s, 36 objects/s recovering 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 21 pgs peering) 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2949403004' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: pgmap v1132: 142 pgs: 1 active+undersized+remapped, 25 peering, 116 active+clean; 315 KiB data, 4.6 GiB used, 442 GiB / 447 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 198 KiB/s, 36 objects/s recovering 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 16 pgs inactive, 21 pgs peering) 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2949403004' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:10 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-26T04:12:11.535 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-26T04:12:11.536 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-26T04:12:11.536 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-26T04:12:11.536 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd in 6 2024-03-26T04:12:11.784 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:12.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:11 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T04:12:12.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:11 smithi006 ceph-mon[37501]: osdmap e517: 8 total, 8 up, 7 in 2024-03-26T04:12:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:11 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T04:12:12.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:11 smithi006 ceph-mon[29446]: osdmap e517: 8 total, 8 up, 7 in 2024-03-26T04:12:12.156 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:11 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-26T04:12:12.156 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:11 smithi086 ceph-mon[32965]: osdmap e517: 8 total, 8 up, 7 in 2024-03-26T04:12:12.752 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:12.752 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:13.067 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: pgmap v1135: 142 pgs: 24 peering, 118 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s wr, 3 op/s; 197 KiB/s, 24 objects/s recovering 2024-03-26T04:12:13.067 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: osdmap e518: 8 total, 8 up, 7 in 2024-03-26T04:12:13.067 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T04:12:13.067 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: pgmap v1135: 142 pgs: 24 peering, 118 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s wr, 3 op/s; 197 KiB/s, 24 objects/s recovering 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: osdmap e518: 8 total, 8 up, 7 in 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T04:12:13.068 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T04:12:13.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:12 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: pgmap v1135: 142 pgs: 24 peering, 118 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 55 KiB/s wr, 3 op/s; 197 KiB/s, 24 objects/s recovering 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: osdmap e518: 8 total, 8 up, 7 in 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T04:12:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T04:12:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T04:12:13.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:12 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T04:12:13.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-26T04:12:13.537 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:13.881 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: osdmap e519: 8 total, 8 up, 7 in 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T04:12:13.883 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T04:12:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:12:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:12:14.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T04:12:14.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T04:12:14.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: osdmap e519: 8 total, 8 up, 7 in 2024-03-26T04:12:14.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T04:12:14.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:13 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T04:12:14.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:12:13.934+0000 7fd465f8b700 -1 mon.a@0(leader).osd e520 definitely_dead 0 2024-03-26T04:12:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:12:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T04:12:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T04:12:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: osdmap e519: 8 total, 8 up, 7 in 2024-03-26T04:12:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T04:12:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:13 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T04:12:14.638 INFO:teuthology.orchestra.run.smithi006.stderr:marked in osd.6. 2024-03-26T04:12:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: pgmap v1138: 142 pgs: 1 remapped+peering, 28 peering, 113 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 972 B/s, 25 objects/s recovering 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: osdmap e520: 8 total, 8 up, 7 in 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: 6.1 scrub starts 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: 6.1 scrub ok 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-26T04:12:14.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[37501]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: pgmap v1138: 142 pgs: 1 remapped+peering, 28 peering, 113 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 972 B/s, 25 objects/s recovering 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: osdmap e520: 8 total, 8 up, 7 in 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: 6.1 scrub starts 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: 6.1 scrub ok 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-26T04:12:14.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:14 smithi006 ceph-mon[29446]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:12:15.152 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:15.152 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: pgmap v1138: 142 pgs: 1 remapped+peering, 28 peering, 113 active+clean; 315 KiB data, 6.3 GiB used, 530 GiB / 536 GiB avail; 972 B/s, 25 objects/s recovering 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: osdmap e520: 8 total, 8 up, 7 in 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: 6.1 scrub starts 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: 6.1 scrub ok 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4056005163' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-26T04:12:15.153 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:14 smithi086 ceph-mon[32965]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:12:15.434 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-26T04:12:15.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-26T04:12:15.720 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:15.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:16.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-26T04:12:16.310 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:16.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T04:12:16.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T04:12:16.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: pgmap v1141: 142 pgs: 1 activating, 1 active+undersized+remapped, 1 remapped+peering, 24 peering, 115 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 93 KiB/s, 43 objects/s recovering 2024-03-26T04:12:16.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:16.703 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:12:16.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T04:12:16.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T04:12:16.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T04:12:16.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T04:12:16.704 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:16 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T04:12:16.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T04:12:16.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T04:12:16.894 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: pgmap v1141: 142 pgs: 1 activating, 1 active+undersized+remapped, 1 remapped+peering, 24 peering, 115 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 93 KiB/s, 43 objects/s recovering 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T04:12:16.895 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: pgmap v1141: 142 pgs: 1 activating, 1 active+undersized+remapped, 1 remapped+peering, 24 peering, 115 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 93 KiB/s, 43 objects/s recovering 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T04:12:16.896 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:16 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T04:12:17.503 INFO:teuthology.orchestra.run.smithi006.stdout:mon.a: {} 2024-03-26T04:12:17.513 INFO:teuthology.orchestra.run.smithi006.stdout:mon.b: {} 2024-03-26T04:12:17.523 INFO:teuthology.orchestra.run.smithi006.stdout:mon.c: {} 2024-03-26T04:12:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T04:12:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T04:12:17.524 INFO:teuthology.orchestra.run.smithi006.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-26T04:12:17.770 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T04:12:17.770 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3380217984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T04:12:17.771 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:17 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T04:12:17.772 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3380217984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3380217984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T04:12:17.773 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:17 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T04:12:18.178 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:18.178 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:18.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 0 1 2024-03-26T04:12:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:18.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:18 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T04:12:18.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:18 smithi086 ceph-mon[32965]: 5.1f deep-scrub starts 2024-03-26T04:12:18.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:18 smithi086 ceph-mon[32965]: 5.1f deep-scrub ok 2024-03-26T04:12:18.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:18 smithi086 ceph-mon[32965]: pgmap v1144: 142 pgs: 1 activating+undersized+remapped, 1 activating, 2 active+undersized+remapped, 20 peering, 118 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 92 KiB/s, 29 objects/s recovering 2024-03-26T04:12:18.806 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:18 smithi086 ceph-mon[32965]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:12:18.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-26T04:12:18.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T04:12:18.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[29446]: 5.1f deep-scrub starts 2024-03-26T04:12:18.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[29446]: 5.1f deep-scrub ok 2024-03-26T04:12:18.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[29446]: pgmap v1144: 142 pgs: 1 activating+undersized+remapped, 1 activating, 2 active+undersized+remapped, 20 peering, 118 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 92 KiB/s, 29 objects/s recovering 2024-03-26T04:12:18.885 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[29446]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:12:18.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T04:12:18.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[37501]: 5.1f deep-scrub starts 2024-03-26T04:12:18.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[37501]: 5.1f deep-scrub ok 2024-03-26T04:12:18.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[37501]: pgmap v1144: 142 pgs: 1 activating+undersized+remapped, 1 activating, 2 active+undersized+remapped, 20 peering, 118 active+clean; 582 KiB data, 6.8 GiB used, 619 GiB / 626 GiB avail; 92 KiB/s, 29 objects/s recovering 2024-03-26T04:12:18.886 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:18 smithi006 ceph-mon[37501]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:12:19.072 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:19.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:19 smithi086 ceph-mon[32965]: 2.0 scrub starts 2024-03-26T04:12:19.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:19 smithi086 ceph-mon[32965]: 2.0 scrub ok 2024-03-26T04:12:19.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:19 smithi086 ceph-mon[32965]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:12:19.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:19 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:12:19.804 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:19 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:12:19.841 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[37501]: 2.0 scrub starts 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[37501]: 2.0 scrub ok 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[37501]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[29446]: 2.0 scrub starts 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[29446]: 2.0 scrub ok 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[29446]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:12:19.842 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:19 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:12:20.649 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:20.649 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:20.683 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.0 to 1 (10000) 2024-03-26T04:12:20.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:20 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T04:12:20.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:20 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T04:12:20.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:20 smithi086 ceph-mon[32965]: pgmap v1147: 142 pgs: 1 activating+undersized+remapped, 1 activating, 19 peering, 121 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 111 B/s, 6 objects/s recovering 2024-03-26T04:12:20.970 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:20 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3385072520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:20.971 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:20 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[37501]: pgmap v1147: 142 pgs: 1 activating+undersized+remapped, 1 activating, 19 peering, 121 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 111 B/s, 6 objects/s recovering 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3385072520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T04:12:21.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[29446]: pgmap v1147: 142 pgs: 1 activating+undersized+remapped, 1 activating, 19 peering, 121 active+clean; 582 KiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 111 B/s, 6 objects/s recovering 2024-03-26T04:12:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3385072520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:21.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:20 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:21.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-26T04:12:21.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-26T04:12:21.511 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:21.730 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:21.851 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 1.6 deep-scrub starts 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 1.6 deep-scrub ok 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[37501]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T04:12:21.852 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 1.6 deep-scrub starts 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 1.6 deep-scrub ok 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T04:12:21.853 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:21 smithi006 ceph-mon[29446]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:12:21.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T04:12:21.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T04:12:21.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 1.6 deep-scrub starts 2024-03-26T04:12:21.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 1.6 deep-scrub ok 2024-03-26T04:12:21.940 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:21.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:12:21.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T04:12:21.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T04:12:21.941 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:21 smithi086 ceph-mon[32965]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 5.1a scrub starts 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 5.1a scrub ok 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: pgmap v1149: 142 pgs: 1 activating+undersized+remapped, 1 peering, 140 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 161 B/s, 11 objects/s recovering 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T04:12:22.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 5.4 deep-scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[37501]: 5.4 deep-scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 5.1a scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 5.1a scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: pgmap v1149: 142 pgs: 1 activating+undersized+remapped, 1 peering, 140 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 161 B/s, 11 objects/s recovering 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 5.4 deep-scrub starts 2024-03-26T04:12:22.891 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:22 smithi006 ceph-mon[29446]: 5.4 deep-scrub ok 2024-03-26T04:12:22.920 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 5.1a scrub starts 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 5.1a scrub ok 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: pgmap v1149: 142 pgs: 1 activating+undersized+remapped, 1 peering, 140 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 855 B/s rd, 0 op/s; 161 B/s, 11 objects/s recovering 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 5.4 deep-scrub starts 2024-03-26T04:12:22.921 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:22 smithi086 ceph-mon[32965]: 5.4 deep-scrub ok 2024-03-26T04:12:23.287 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:23.287 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 4.0 deep-scrub starts 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 4.0 deep-scrub ok 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T04:12:23.952 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:23 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T04:12:23.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-26T04:12:24.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 4.0 deep-scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 4.0 deep-scrub ok 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T04:12:24.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T04:12:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 4.0 deep-scrub starts 2024-03-26T04:12:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 4.0 deep-scrub ok 2024-03-26T04:12:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T04:12:24.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:23 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T04:12:24.223 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:24.640 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:24.711 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 4.1f deep-scrub starts 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 4.1f deep-scrub ok 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 708 B/s rd, 0 op/s; 45 B/s, 7 objects/s recovering 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 4.17 scrub starts 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 4.17 scrub ok 2024-03-26T04:12:25.024 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/704841910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 4.1f deep-scrub starts 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 4.1f deep-scrub ok 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 708 B/s rd, 0 op/s; 45 B/s, 7 objects/s recovering 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 4.17 scrub starts 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 4.17 scrub ok 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/704841910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.025 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:24 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 4.1f deep-scrub starts 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 4.1f deep-scrub ok 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: pgmap v1151: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 708 B/s rd, 0 op/s; 45 B/s, 7 objects/s recovering 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 4.17 scrub starts 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 4.17 scrub ok 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/704841910' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.026 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-26T04:12:25.027 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:24 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T04:12:25.493 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 1 1 2024-03-26T04:12:25.686 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:25.714 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T04:12:25.714 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T04:12:25.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T04:12:25.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T04:12:25.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:25.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:12:25.715 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T04:12:25.749 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:25.814 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:25.814 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 4.10 deep-scrub starts 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 4.10 deep-scrub ok 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 2.1 deep-scrub starts 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: 2.1 deep-scrub ok 2024-03-26T04:12:26.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 4.10 deep-scrub starts 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 4.10 deep-scrub ok 2024-03-26T04:12:26.003 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 2.1 deep-scrub starts 2024-03-26T04:12:26.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: 2.1 deep-scrub ok 2024-03-26T04:12:26.004 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T04:12:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T04:12:26.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 4.10 deep-scrub starts 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 4.10 deep-scrub ok 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 2.1 deep-scrub starts 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: 2.1 deep-scrub ok 2024-03-26T04:12:26.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:26.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-26T04:12:26.643 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 91 B/s, 7 objects/s recovering 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:26.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 91 B/s, 7 objects/s recovering 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:26.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:26 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:26.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 91 B/s, 7 objects/s recovering 2024-03-26T04:12:26.968 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:26.969 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:26 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]: dispatch 2024-03-26T04:12:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:27 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-03-26T04:12:27.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:27 smithi086 ceph-mon[32965]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:12:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:27 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-03-26T04:12:28.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:27 smithi006 ceph-mon[29446]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:12:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:27 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [3, 5]}]': finished 2024-03-26T04:12:28.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:27 smithi006 ceph-mon[37501]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:12:28.162 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:28.162 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:28.744 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.1 to 1 (10000) 2024-03-26T04:12:28.874 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T04:12:28.874 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 6.15 scrub starts 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 6.15 scrub ok 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 889 B/s rd, 0 op/s; 38 B/s, 2 objects/s recovering 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 1.1 scrub starts 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 1.1 scrub ok 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4042662257' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 6.13 scrub starts 2024-03-26T04:12:28.875 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:28 smithi086 ceph-mon[32965]: 6.13 scrub ok 2024-03-26T04:12:28.902 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-26T04:12:29.049 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 6.15 scrub starts 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 6.15 scrub ok 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 889 B/s rd, 0 op/s; 38 B/s, 2 objects/s recovering 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 1.1 scrub starts 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 1.1 scrub ok 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4042662257' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 6.13 scrub starts 2024-03-26T04:12:29.050 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[37501]: 6.13 scrub ok 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 6.15 scrub starts 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 6.15 scrub ok 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 889 B/s rd, 0 op/s; 38 B/s, 2 objects/s recovering 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 1.1 scrub starts 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 1.1 scrub ok 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4042662257' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 6.13 scrub starts 2024-03-26T04:12:29.051 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:28 smithi006 ceph-mon[29446]: 6.13 scrub ok 2024-03-26T04:12:29.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-26T04:12:29.599 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:30.053 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[37501]: 5.9 scrub starts 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[37501]: 5.9 scrub ok 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[37501]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[37501]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[29446]: 5.9 scrub starts 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[29446]: 5.9 scrub ok 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[29446]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:12:30.054 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:29 smithi006 ceph-mon[29446]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:12:30.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:29 smithi086 ceph-mon[32965]: 5.9 scrub starts 2024-03-26T04:12:30.089 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:29 smithi086 ceph-mon[32965]: 5.9 scrub ok 2024-03-26T04:12:30.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:29 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:30.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:29 smithi086 ceph-mon[32965]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:12:30.090 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:29 smithi086 ceph-mon[32965]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:12:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:31.028 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:30 smithi006 ceph-mon[37501]: pgmap v1159: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:12:31.028 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:30 smithi006 ceph-mon[29446]: pgmap v1159: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:12:31.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:30 smithi086 ceph-mon[32965]: pgmap v1159: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.7 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:12:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:12:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:12:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T04:12:31.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T04:12:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T04:12:31.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:31 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:12:32.001 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:12:32.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T04:12:32.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T04:12:32.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T04:12:32.002 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:31 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T04:12:32.774 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 4.d scrub starts 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 4.d scrub ok 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: pgmap v1161: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 46 B/s, 1 objects/s recovering 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 6.2 scrub starts 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 6.2 scrub ok 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 5.b scrub starts 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: 5.b scrub ok 2024-03-26T04:12:32.849 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2188380172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 4.d scrub starts 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 4.d scrub ok 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: pgmap v1161: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 46 B/s, 1 objects/s recovering 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 6.2 scrub starts 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 6.2 scrub ok 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 5.b scrub starts 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: 5.b scrub ok 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2188380172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:32.850 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:32 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:32.976 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:32.977 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 4.d scrub starts 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 4.d scrub ok 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: pgmap v1161: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 46 B/s, 1 objects/s recovering 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 6.2 scrub starts 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 6.2 scrub ok 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 5.b scrub starts 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: 5.b scrub ok 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2188380172' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:33.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:32 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-26T04:12:33.920 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-26T04:12:34.065 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[37501]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[37501]: 3.1a deep-scrub starts 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[37501]: 3.1a deep-scrub ok 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[29446]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[29446]: 3.1a deep-scrub starts 2024-03-26T04:12:34.066 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:33 smithi006 ceph-mon[29446]: 3.1a deep-scrub ok 2024-03-26T04:12:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:33 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-26T04:12:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:33 smithi086 ceph-mon[32965]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:12:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:33 smithi086 ceph-mon[32965]: 3.1a deep-scrub starts 2024-03-26T04:12:34.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:33 smithi086 ceph-mon[32965]: 3.1a deep-scrub ok 2024-03-26T04:12:34.577 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:34.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 3 1 2024-03-26T04:12:34.624 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:34.834 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 3.13 deep-scrub starts 2024-03-26T04:12:34.834 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 3.13 deep-scrub starts 2024-03-26T04:12:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 3.13 deep-scrub starts 2024-03-26T04:12:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 3.13 deep-scrub ok 2024-03-26T04:12:35.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: pgmap v1163: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 3.1 scrub starts 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 3.1 scrub ok 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 4.1c scrub starts 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: 4.1c scrub ok 2024-03-26T04:12:35.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:34 smithi086 ceph-mon[32965]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 3.13 deep-scrub ok 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: pgmap v1163: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 3.1 scrub starts 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 3.1 scrub ok 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 4.1c scrub starts 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: 4.1c scrub ok 2024-03-26T04:12:35.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[37501]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 3.13 deep-scrub ok 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: pgmap v1163: 142 pgs: 3 peering, 139 active+clean; 582 KiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 897 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 3.1 scrub starts 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 3.1 scrub ok 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 4.1c scrub starts 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: 4.1c scrub ok 2024-03-26T04:12:35.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:34 smithi006 ceph-mon[29446]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:12:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:35 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:12:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:35 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:12:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:35 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:12:36.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:35 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:12:36.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:12:36.109 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:35 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:12:36.147 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:37.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[37501]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:37.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:37.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T04:12:37.074 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T04:12:37.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[29446]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:37.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:37.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T04:12:37.075 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:36 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T04:12:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:36 smithi086 ceph-mon[32965]: pgmap v1165: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:36 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:36 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T04:12:37.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:36 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T04:12:37.814 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[37501]: 6.f deep-scrub starts 2024-03-26T04:12:37.814 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T04:12:37.814 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T04:12:37.814 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[29446]: 6.f deep-scrub starts 2024-03-26T04:12:37.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T04:12:37.815 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:37 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T04:12:37.841 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:37.841 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:37 smithi086 ceph-mon[32965]: 6.f deep-scrub starts 2024-03-26T04:12:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:37 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T04:12:38.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:37 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T04:12:38.663 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-26T04:12:38.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:38 smithi006 ceph-mon[37501]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:38.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:38 smithi006 ceph-mon[37501]: 6.f deep-scrub ok 2024-03-26T04:12:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:38 smithi006 ceph-mon[29446]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:38.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:38 smithi006 ceph-mon[29446]: 6.f deep-scrub ok 2024-03-26T04:12:39.054 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:38 smithi086 ceph-mon[32965]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 0 B/s, 1 objects/s recovering 2024-03-26T04:12:39.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:38 smithi086 ceph-mon[32965]: 6.f deep-scrub ok 2024-03-26T04:12:39.816 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.3 to 1 (10000) 2024-03-26T04:12:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:39 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:12:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:39 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:39 smithi086 ceph-mon[32965]: 1.3 deep-scrub starts 2024-03-26T04:12:40.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:39 smithi086 ceph-mon[32965]: 1.3 deep-scrub ok 2024-03-26T04:12:40.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[37501]: 1.3 deep-scrub starts 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[37501]: 1.3 deep-scrub ok 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[29446]: 1.3 deep-scrub starts 2024-03-26T04:12:40.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:39 smithi006 ceph-mon[29446]: 1.3 deep-scrub ok 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: 6.a deep-scrub starts 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: 6.a deep-scrub ok 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:40.901 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: 6.a deep-scrub starts 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: 6.a deep-scrub ok 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:12:40.902 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:40 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:40.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-26T04:12:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:12:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: 6.a deep-scrub starts 2024-03-26T04:12:41.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: 6.a deep-scrub ok 2024-03-26T04:12:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/867986871' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:12:41.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:40 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:41.460 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:41.477 INFO:teuthology.orchestra.run.smithi006.stdout:ERROR: (22) Invalid argument 2024-03-26T04:12:41.477 INFO:teuthology.orchestra.run.smithi006.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-26T04:12:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 5.10 scrub starts 2024-03-26T04:12:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 5.10 scrub ok 2024-03-26T04:12:43.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T04:12:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T04:12:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T04:12:43.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:42 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 5.10 scrub starts 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 5.10 scrub ok 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T04:12:43.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 5.10 scrub starts 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 5.10 scrub ok 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: pgmap v1169: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T04:12:43.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:42 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T04:12:43.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:43 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T04:12:43.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:43 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T04:12:43.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:43 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T04:12:43.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:43 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T04:12:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:43 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T04:12:44.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:43 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T04:12:44.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:44.649 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-26T04:12:44.973 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[37501]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:44.973 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:44.973 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:44.974 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[37501]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:12:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[29446]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:44.974 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:44 smithi006 ceph-mon[29446]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:12:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:44 smithi086 ceph-mon[32965]: pgmap v1170: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:12:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-26T04:12:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:44 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/310430959' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-26T04:12:45.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:44 smithi086 ceph-mon[32965]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:12:45.334 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 4 1 2024-03-26T04:12:45.588 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:46.061 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:45 smithi006 ceph-mon[37501]: 6.10 scrub starts 2024-03-26T04:12:46.061 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:45 smithi006 ceph-mon[29446]: 6.10 scrub starts 2024-03-26T04:12:46.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:45 smithi086 ceph-mon[32965]: 6.10 scrub starts 2024-03-26T04:12:46.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: 6.10 scrub ok 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: 6.1b scrub starts 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: 6.1b scrub ok 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: 3.1b scrub starts 2024-03-26T04:12:46.976 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[37501]: 3.1b scrub ok 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: 6.10 scrub ok 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: 6.1b scrub starts 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: 6.1b scrub ok 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: 3.1b scrub starts 2024-03-26T04:12:46.977 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:46 smithi006 ceph-mon[29446]: 3.1b scrub ok 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: pgmap v1172: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: 6.10 scrub ok 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: 6.1b scrub starts 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: 6.1b scrub ok 2024-03-26T04:12:47.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: 3.1b scrub starts 2024-03-26T04:12:47.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:46 smithi086 ceph-mon[32965]: 3.1b scrub ok 2024-03-26T04:12:47.843 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.4 to 1 (10000) 2024-03-26T04:12:48.120 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 6.8 scrub starts 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 6.8 scrub ok 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 5.14 deep-scrub starts 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 5.14 deep-scrub ok 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3664882648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 6.8 scrub starts 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 6.8 scrub ok 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 5.14 deep-scrub starts 2024-03-26T04:12:48.121 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 5.14 deep-scrub ok 2024-03-26T04:12:48.122 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T04:12:48.122 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T04:12:48.122 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3664882648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.122 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:47 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 6.8 scrub starts 2024-03-26T04:12:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 6.8 scrub ok 2024-03-26T04:12:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 5.14 deep-scrub starts 2024-03-26T04:12:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 5.14 deep-scrub ok 2024-03-26T04:12:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T04:12:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T04:12:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3664882648' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:47 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:48.579 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-26T04:12:48.828 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:49.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: 1.8 scrub starts 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: 1.8 scrub ok 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: 3.e scrub starts 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: 3.e scrub ok 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: 1.8 scrub starts 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: 1.8 scrub ok 2024-03-26T04:12:49.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:49.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:12:49.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: 3.e scrub starts 2024-03-26T04:12:49.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: 3.e scrub ok 2024-03-26T04:12:49.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:48 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T04:12:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: pgmap v1173: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: 1.8 scrub starts 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: 1.8 scrub ok 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: 3.e scrub starts 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: 3.e scrub ok 2024-03-26T04:12:49.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:48 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T04:12:50.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T04:12:50.043 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: 4.a scrub starts 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: 4.a scrub ok 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[37501]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: 4.a scrub starts 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: 4.a scrub ok 2024-03-26T04:12:50.044 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:49 smithi006 ceph-mon[29446]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:12:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T04:12:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T04:12:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T04:12:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: 4.a scrub starts 2024-03-26T04:12:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: 4.a scrub ok 2024-03-26T04:12:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:49 smithi086 ceph-mon[32965]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:12:50.861 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-26T04:12:51.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[37501]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[37501]: 3.8 scrub starts 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[37501]: 3.8 scrub ok 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3601792980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[29446]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[29446]: 3.8 scrub starts 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[29446]: 3.8 scrub ok 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3601792980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:50 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:50 smithi086 ceph-mon[32965]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:12:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:50 smithi086 ceph-mon[32965]: 3.8 scrub starts 2024-03-26T04:12:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:50 smithi086 ceph-mon[32965]: 3.8 scrub ok 2024-03-26T04:12:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:50 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3601792980' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:50 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-26T04:12:51.646 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 6 1 2024-03-26T04:12:51.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:51 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:51.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:51 smithi006 ceph-mon[37501]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:12:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:51 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:51.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:51 smithi006 ceph-mon[29446]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:12:51.902 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:51 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-26T04:12:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:51 smithi086 ceph-mon[32965]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:12:53.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 6.1c scrub starts 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 6.1c scrub ok 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 1.2 scrub starts 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 1.2 scrub ok 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 1.9 scrub starts 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[37501]: 1.9 scrub ok 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 6.1c scrub starts 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 6.1c scrub ok 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 1.2 scrub starts 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 1.2 scrub ok 2024-03-26T04:12:53.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 1.9 scrub starts 2024-03-26T04:12:53.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:52 smithi006 ceph-mon[29446]: 1.9 scrub ok 2024-03-26T04:12:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 6.1c scrub starts 2024-03-26T04:12:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 6.1c scrub ok 2024-03-26T04:12:53.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 1.2 scrub starts 2024-03-26T04:12:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 1.2 scrub ok 2024-03-26T04:12:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 1.9 scrub starts 2024-03-26T04:12:53.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:52 smithi086 ceph-mon[32965]: 1.9 scrub ok 2024-03-26T04:12:54.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:12:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:12:54.655 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.6 to 1 (10000) 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[29446]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[29446]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[37501]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:54.984 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:54 smithi006 ceph-mon[37501]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:12:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:54 smithi086 ceph-mon[32965]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:12:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:54 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4253106636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:55.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:54 smithi086 ceph-mon[32965]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:12:55.403 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-26T04:12:55.660 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:55 smithi086 ceph-mon[32965]: 1.5 scrub starts 2024-03-26T04:12:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:55 smithi086 ceph-mon[32965]: 1.5 scrub ok 2024-03-26T04:12:56.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:55 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:56.088 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:12:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:12:56.112 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[37501]: 1.5 scrub starts 2024-03-26T04:12:56.112 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[37501]: 1.5 scrub ok 2024-03-26T04:12:56.112 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:56.113 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[29446]: 1.5 scrub starts 2024-03-26T04:12:56.113 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[29446]: 1.5 scrub ok 2024-03-26T04:12:56.113 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:55 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:12:57.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 5.d scrub starts 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 5.d scrub ok 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 5.12 deep-scrub starts 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 5.12 deep-scrub ok 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 3.12 scrub starts 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[37501]: 3.12 scrub ok 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 5.d scrub starts 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 5.d scrub ok 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:57.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 5.12 deep-scrub starts 2024-03-26T04:12:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 5.12 deep-scrub ok 2024-03-26T04:12:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 3.12 scrub starts 2024-03-26T04:12:57.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:56 smithi006 ceph-mon[29446]: 3.12 scrub ok 2024-03-26T04:12:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:12:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 5.d scrub starts 2024-03-26T04:12:57.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 5.d scrub ok 2024-03-26T04:12:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:12:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 5.12 deep-scrub starts 2024-03-26T04:12:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 5.12 deep-scrub ok 2024-03-26T04:12:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 3.12 scrub starts 2024-03-26T04:12:57.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:56 smithi086 ceph-mon[32965]: 3.12 scrub ok 2024-03-26T04:12:57.888 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-26T04:12:58.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 3.5 scrub starts 2024-03-26T04:12:58.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 3.5 scrub ok 2024-03-26T04:12:58.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 1.7 scrub starts 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: 1.7 scrub ok 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 3.5 scrub starts 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 3.5 scrub ok 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 1.7 scrub starts 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: 1.7 scrub ok 2024-03-26T04:12:58.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:57 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:58.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 3.5 scrub starts 2024-03-26T04:12:58.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 3.5 scrub ok 2024-03-26T04:12:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T04:12:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T04:12:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 1.7 scrub starts 2024-03-26T04:12:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: 1.7 scrub ok 2024-03-26T04:12:58.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:57 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-26T04:12:58.670 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 7 1 2024-03-26T04:12:58.925 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:59.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 3.1c scrub starts 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 3.1c scrub ok 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 5.e scrub starts 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 5.e scrub ok 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 4.1e scrub starts 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[37501]: 4.1e scrub ok 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 3.1c scrub starts 2024-03-26T04:12:59.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 3.1c scrub ok 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 5.e scrub starts 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 5.e scrub ok 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 4.1e scrub starts 2024-03-26T04:12:59.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:58 smithi006 ceph-mon[29446]: 4.1e scrub ok 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 3.1c scrub starts 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 3.1c scrub ok 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1999473592' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 5.e scrub starts 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 5.e scrub ok 2024-03-26T04:12:59.239 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 4.1e scrub starts 2024-03-26T04:12:59.240 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:58 smithi086 ceph-mon[32965]: 4.1e scrub ok 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[29446]: 3.7 scrub starts 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[29446]: 3.7 scrub ok 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[29446]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[37501]: 3.7 scrub starts 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[37501]: 3.7 scrub ok 2024-03-26T04:13:00.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T04:13:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T04:13:00.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:12:59 smithi006 ceph-mon[37501]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:13:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:59 smithi086 ceph-mon[32965]: 3.7 scrub starts 2024-03-26T04:13:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:59 smithi086 ceph-mon[32965]: 3.7 scrub ok 2024-03-26T04:13:00.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:59 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T04:13:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:59 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T04:13:00.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:12:59 smithi086 ceph-mon[32965]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:13:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:00 smithi086 ceph-mon[32965]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:00 smithi086 ceph-mon[32965]: 5.11 scrub starts 2024-03-26T04:13:01.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:00 smithi086 ceph-mon[32965]: 5.11 scrub ok 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[37501]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[37501]: 5.11 scrub starts 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[37501]: 5.11 scrub ok 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[29446]: pgmap v1184: 142 pgs: 142 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[29446]: 5.11 scrub starts 2024-03-26T04:13:01.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:00 smithi006 ceph-mon[29446]: 5.11 scrub ok 2024-03-26T04:13:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.7 to 1 (10000) 2024-03-26T04:13:02.208 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3207896221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.208 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:01 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.209 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3207896221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.209 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:01 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3207896221' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:01 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:02.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-26T04:13:02.841 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[37501]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[37501]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[29446]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:03.179 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:02 smithi006 ceph-mon[29446]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:13:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:02 smithi086 ceph-mon[32965]: pgmap v1186: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:03.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:02 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:03.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:02 smithi086 ceph-mon[32965]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:13:04.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[37501]: 5.c scrub starts 2024-03-26T04:13:04.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[37501]: 5.c scrub ok 2024-03-26T04:13:04.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:04.196 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:04.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[29446]: 5.c scrub starts 2024-03-26T04:13:04.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[29446]: 5.c scrub ok 2024-03-26T04:13:04.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:04.197 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:03 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:03 smithi086 ceph-mon[32965]: 5.c scrub starts 2024-03-26T04:13:04.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:03 smithi086 ceph-mon[32965]: 5.c scrub ok 2024-03-26T04:13:04.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:03 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:04.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:03 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:04.496 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:05.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: 6.3 deep-scrub starts 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: 6.17 scrub starts 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: 6.3 deep-scrub ok 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: 4.5 scrub starts 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: 4.5 scrub ok 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/698675478' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:04 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: 6.3 deep-scrub starts 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: 6.17 scrub starts 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: 6.3 deep-scrub ok 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: 4.5 scrub starts 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: 4.5 scrub ok 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/698675478' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: pgmap v1188: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: 6.3 deep-scrub starts 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: 6.17 scrub starts 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: 6.3 deep-scrub ok 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: 4.5 scrub starts 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: 4.5 scrub ok 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/698675478' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:04 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-26T04:13:05.666 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-26T04:13:05.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: 6.17 scrub ok 2024-03-26T04:13:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: 6.7 scrub starts 2024-03-26T04:13:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: 6.7 scrub ok 2024-03-26T04:13:06.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[37501]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: 6.17 scrub ok 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: 6.7 scrub starts 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: 6.7 scrub ok 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:06.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:05 smithi006 ceph-mon[29446]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:13:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: 6.17 scrub ok 2024-03-26T04:13:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: 6.7 scrub starts 2024-03-26T04:13:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: 6.7 scrub ok 2024-03-26T04:13:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:06.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-26T04:13:06.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:05 smithi086 ceph-mon[32965]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:13:06.462 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 5 1 2024-03-26T04:13:06.734 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: 4.18 deep-scrub starts 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: 4.18 deep-scrub ok 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:07.122 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: 4.18 deep-scrub starts 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: 4.18 deep-scrub ok 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:07.123 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:06 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:13:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: pgmap v1190: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: 4.18 deep-scrub starts 2024-03-26T04:13:07.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: 4.18 deep-scrub ok 2024-03-26T04:13:07.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:13:07.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:07.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:07.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:06 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: 5.0 scrub starts 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: 5.0 scrub ok 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: 5.2 scrub starts 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: 5.2 scrub ok 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:08.182 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: 5.0 scrub starts 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: 5.0 scrub ok 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: 5.2 scrub starts 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: 5.2 scrub ok 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:08.183 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:07 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: pgmap v1193: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: pgmap v1194: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:13:08.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: 5.0 scrub starts 2024-03-26T04:13:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: 5.0 scrub ok 2024-03-26T04:13:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: 5.2 scrub starts 2024-03-26T04:13:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: 5.2 scrub ok 2024-03-26T04:13:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:08.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:07 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:08.937 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.5 to 1 (10000) 2024-03-26T04:13:09.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[37501]: 3.b scrub starts 2024-03-26T04:13:09.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[37501]: 3.b scrub ok 2024-03-26T04:13:09.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[37501]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:13:09.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3194322950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.251 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[29446]: 3.b scrub starts 2024-03-26T04:13:09.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[29446]: 3.b scrub ok 2024-03-26T04:13:09.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[29446]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:13:09.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3194322950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.252 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:08 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:08 smithi086 ceph-mon[32965]: 3.b scrub starts 2024-03-26T04:13:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:08 smithi086 ceph-mon[32965]: 3.b scrub ok 2024-03-26T04:13:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:08 smithi086 ceph-mon[32965]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:13:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:08 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3194322950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:08 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:09.725 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-26T04:13:09.977 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:10.098 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: 5.15 deep-scrub starts 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: 5.15 deep-scrub ok 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: 3.d scrub starts 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[37501]: 3.d scrub ok 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: 5.15 deep-scrub starts 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: 5.15 deep-scrub ok 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: 3.d scrub starts 2024-03-26T04:13:10.099 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:09 smithi006 ceph-mon[29446]: 3.d scrub ok 2024-03-26T04:13:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: 5.15 deep-scrub starts 2024-03-26T04:13:10.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: 5.15 deep-scrub ok 2024-03-26T04:13:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: pgmap v1196: 142 pgs: 142 active+clean; 582 KiB data, 6.2 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:13:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: 3.d scrub starts 2024-03-26T04:13:10.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:09 smithi086 ceph-mon[32965]: 3.d scrub ok 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[37501]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:11.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:11.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:10 smithi006 ceph-mon[29446]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:13:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:10 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:10 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:11.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:11.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:10 smithi086 ceph-mon[32965]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:13:12.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:11 smithi086 ceph-mon[32965]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:12.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:11 smithi006 ceph-mon[29446]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:12.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:11 smithi006 ceph-mon[37501]: pgmap v1199: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:12.951 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-26T04:13:13.334 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:13.334 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:13.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-26T04:13:13.759 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd reweight 2 1 2024-03-26T04:13:14.013 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[37501]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[37501]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[29446]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:14.128 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:13 smithi006 ceph-mon[29446]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:13:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:13 smithi086 ceph-mon[32965]: pgmap v1200: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:13 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/4030068583' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-26T04:13:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:13 smithi086 ceph-mon[32965]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:13:14.389 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 5.1b scrub starts 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 5.1b scrub ok 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 6.6 deep-scrub starts 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 6.6 deep-scrub ok 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 5.16 scrub starts 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: 5.16 scrub ok 2024-03-26T04:13:15.286 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[37501]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 5.1b scrub starts 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 5.1b scrub ok 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 6.6 deep-scrub starts 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 6.6 deep-scrub ok 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 5.16 scrub starts 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: 5.16 scrub ok 2024-03-26T04:13:15.287 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:14 smithi006 ceph-mon[29446]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:13:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 5.1b scrub starts 2024-03-26T04:13:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 5.1b scrub ok 2024-03-26T04:13:15.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 6.6 deep-scrub starts 2024-03-26T04:13:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 6.6 deep-scrub ok 2024-03-26T04:13:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:13:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 5.16 scrub starts 2024-03-26T04:13:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: 5.16 scrub ok 2024-03-26T04:13:15.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:14 smithi086 ceph-mon[32965]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:13:15.958 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:16.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 3.1e scrub starts 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 3.1e scrub ok 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 4.1b scrub starts 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 4.1b scrub ok 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 3.16 scrub starts 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: 3.16 scrub ok 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3150219079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:15 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 3.1e scrub starts 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 3.1e scrub ok 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 4.1b scrub starts 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 4.1b scrub ok 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 3.16 scrub starts 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: 3.16 scrub ok 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3150219079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: pgmap v1204: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 3.1e scrub starts 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 3.1e scrub ok 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 4.1b scrub starts 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 4.1b scrub ok 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 3.16 scrub starts 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: 3.16 scrub ok 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3150219079' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:15 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:16.672 INFO:teuthology.orchestra.run.smithi006.stderr:reweighted osd.2 to 1 (10000) 2024-03-26T04:13:17.224 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:17.224 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[37501]: 2.2 scrub starts 2024-03-26T04:13:17.224 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[37501]: 2.2 scrub ok 2024-03-26T04:13:17.224 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[37501]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[29446]: 2.2 scrub starts 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[29446]: 2.2 scrub ok 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:17.225 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:16 smithi006 ceph-mon[29446]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:13:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:16 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:16 smithi086 ceph-mon[32965]: 2.2 scrub starts 2024-03-26T04:13:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:16 smithi086 ceph-mon[32965]: 2.2 scrub ok 2024-03-26T04:13:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:16 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:16 smithi086 ceph-mon[32965]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:13:17.424 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-26T04:13:17.675 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:18.063 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[37501]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:18.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[37501]: 1.a scrub starts 2024-03-26T04:13:18.064 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[37501]: 1.a scrub ok 2024-03-26T04:13:18.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[29446]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:18.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[29446]: 1.a scrub starts 2024-03-26T04:13:18.064 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:17 smithi006 ceph-mon[29446]: 1.a scrub ok 2024-03-26T04:13:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:17 smithi086 ceph-mon[32965]: pgmap v1206: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:18.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:17 smithi086 ceph-mon[32965]: 1.a scrub starts 2024-03-26T04:13:18.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:17 smithi086 ceph-mon[32965]: 1.a scrub ok 2024-03-26T04:13:19.263 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:18 smithi006 ceph-mon[29446]: 5.1d scrub starts 2024-03-26T04:13:19.264 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:18 smithi006 ceph-mon[29446]: 5.1d scrub ok 2024-03-26T04:13:19.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:18 smithi006 ceph-mon[37501]: 5.1d scrub starts 2024-03-26T04:13:19.264 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:18 smithi006 ceph-mon[37501]: 5.1d scrub ok 2024-03-26T04:13:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:18 smithi086 ceph-mon[32965]: 5.1d scrub starts 2024-03-26T04:13:19.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:18 smithi086 ceph-mon[32965]: 5.1d scrub ok 2024-03-26T04:13:19.661 INFO:teuthology.orchestra.run.smithi006.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 3.14 scrub starts 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 3.14 scrub ok 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 5.17 scrub starts 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 5.17 scrub ok 2024-03-26T04:13:20.259 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 6.12 scrub starts 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: 6.12 scrub ok 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2302300584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[37501]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 3.14 scrub starts 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 3.14 scrub ok 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-26T04:13:20.260 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 5.17 scrub starts 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 5.17 scrub ok 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 6.12 scrub starts 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: 6.12 scrub ok 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2302300584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:20.261 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:19 smithi006 ceph-mon[29446]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:13:20.287 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-26T04:13:20.287 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-26T04:13:20.287 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-26T04:13:20.288 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-26T04:13:20.288 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:13:20.288 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:13:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 3.14 scrub starts 2024-03-26T04:13:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 3.14 scrub ok 2024-03-26T04:13:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: pgmap v1207: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 893 B/s rd, 0 op/s 2024-03-26T04:13:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 5.17 scrub starts 2024-03-26T04:13:20.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 5.17 scrub ok 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 6.12 scrub starts 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: 6.12 scrub ok 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2302300584' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-26T04:13:20.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:19 smithi086 ceph-mon[32965]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:13:20.536 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:20 smithi086 ceph-mon[32965]: 4.19 scrub starts 2024-03-26T04:13:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:20 smithi086 ceph-mon[32965]: 4.19 scrub ok 2024-03-26T04:13:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:20 smithi086 ceph-mon[32965]: 4.3 scrub starts 2024-03-26T04:13:21.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:20 smithi086 ceph-mon[32965]: 4.3 scrub ok 2024-03-26T04:13:21.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[37501]: 4.19 scrub starts 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[37501]: 4.19 scrub ok 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[37501]: 4.3 scrub starts 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[37501]: 4.3 scrub ok 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[29446]: 4.19 scrub starts 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[29446]: 4.19 scrub ok 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[29446]: 4.3 scrub starts 2024-03-26T04:13:21.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:20 smithi006 ceph-mon[29446]: 4.3 scrub ok 2024-03-26T04:13:22.229 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[37501]: 4.13 scrub starts 2024-03-26T04:13:22.229 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[37501]: 4.13 scrub ok 2024-03-26T04:13:22.230 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[37501]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:22.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[29446]: 4.13 scrub starts 2024-03-26T04:13:22.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[29446]: 4.13 scrub ok 2024-03-26T04:13:22.230 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:21 smithi006 ceph-mon[29446]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:21 smithi086 ceph-mon[32965]: 4.13 scrub starts 2024-03-26T04:13:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:21 smithi086 ceph-mon[32965]: 4.13 scrub ok 2024-03-26T04:13:22.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:21 smithi086 ceph-mon[32965]: pgmap v1209: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:13:22.474 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:13:22.474 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":556,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","created":"2024-03-26T03:43:14.534401+0000","modified":"2024-03-26T04:13:21.972145+0000","last_up_change":"2024-03-26T04:05:28.913716+0000","last_in_change":"2024-03-26T04:12:14.632752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T03:46:30.183658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"289","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"271","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T03:48:41.622497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T03:48:42.191818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T03:48:44.079106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T03:48:46.035261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T03:48:48.086810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d1c84cf-5fa7-437e-8004-3b7aca831d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":526,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":1246039834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1246039834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1246039834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":1246039834}]},"public_addr":"172.21.15.6:6801/1246039834","cluster_addr":"172.21.15.6:6802/1246039834","heartbeat_back_addr":"172.21.15.6:6804/1246039834","heartbeat_front_addr":"172.21.15.6:6803/1246039834","state":["exists","up"]},{"osd":1,"uuid":"753f7459-62ec-447b-a0da-64d06e813f16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":260311411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":260311411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":260311411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":260311411}]},"public_addr":"172.21.15.6:6805/260311411","cluster_addr":"172.21.15.6:6806/260311411","heartbeat_back_addr":"172.21.15.6:6808/260311411","heartbeat_front_addr":"172.21.15.6:6807/260311411","state":["exists","up"]},{"osd":2,"uuid":"b7d7409f-341a-4711-8570-92fe9082a528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":335,"up_from":378,"up_thru":523,"down_at":336,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2571159281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2571159281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2571159281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2571159281}]},"public_addr":"172.21.15.6:6809/2571159281","cluster_addr":"172.21.15.6:6810/2571159281","heartbeat_back_addr":"172.21.15.6:6812/2571159281","heartbeat_front_addr":"172.21.15.6:6811/2571159281","state":["exists","up"]},{"osd":3,"uuid":"a91459a0-dee4-420c-ae2f-940756c985a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2795883342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2795883342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2795883342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2795883342}]},"public_addr":"172.21.15.6:6813/2795883342","cluster_addr":"172.21.15.6:6814/2795883342","heartbeat_back_addr":"172.21.15.6:6816/2795883342","heartbeat_front_addr":"172.21.15.6:6815/2795883342","state":["exists","up"]},{"osd":4,"uuid":"087da212-7bf7-4982-b7cc-11f3f66c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":931538411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":931538411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6803","nonce":931538411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":931538411}]},"public_addr":"172.21.15.86:6800/931538411","cluster_addr":"172.21.15.86:6801/931538411","heartbeat_back_addr":"172.21.15.86:6803/931538411","heartbeat_front_addr":"172.21.15.86:6802/931538411","state":["exists","up"]},{"osd":5,"uuid":"7a688745-b213-4bbc-a735-42ffd2a5bfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":167,"up_from":180,"up_thru":532,"down_at":168,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":909197223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":909197223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6807","nonce":909197223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":909197223}]},"public_addr":"172.21.15.86:6804/909197223","cluster_addr":"172.21.15.86:6805/909197223","heartbeat_back_addr":"172.21.15.86:6807/909197223","heartbeat_front_addr":"172.21.15.86:6806/909197223","state":["exists","up"]},{"osd":6,"uuid":"58b58555-59a3-4620-a15f-024f7041713f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1580133777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":1580133777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6811","nonce":1580133777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1580133777}]},"public_addr":"172.21.15.86:6808/1580133777","cluster_addr":"172.21.15.86:6809/1580133777","heartbeat_back_addr":"172.21.15.86:6811/1580133777","heartbeat_front_addr":"172.21.15.86:6810/1580133777","state":["exists","up"]},{"osd":7,"uuid":"49b1833d-0a24-4168-a195-47b8ac1d9bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":546,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3905426306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":3905426306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6815","nonce":3905426306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3905426306}]},"public_addr":"172.21.15.86:6812/3905426306","cluster_addr":"172.21.15.86:6813/3905426306","heartbeat_back_addr":"172.21.15.86:6815/3905426306","heartbeat_front_addr":"172.21.15.86:6814/3905426306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:11:55.308972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:07.061100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:13.481504+0000","dead_epoch":335},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:12:12.354011+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:21.475916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:09.897542+0000","dead_epoch":167},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:12:25.474293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:03.316688+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.2","osds":[7,4,6]}],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.10","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1978459317":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/1771670410":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/3964053112":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/14180211":"2024-03-27T03:49:23.135700+0000","172.21.15.6:6800/2547319348":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/601470125":"2024-03-27T03:44:16.095969+0000","172.21.15.6:6800/3870917014":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1892511221":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/435737617":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/93567168":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/2150730999":"2024-03-27T03:43:49.288943+0000","172.21.15.6:6800/1166209788":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/1248481299":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1832156537":"2024-03-27T03:44:16.095969+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:13:23.033 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:23.033 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:23.033 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: 6.14 scrub starts 2024-03-26T04:13:23.033 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: 6.14 scrub ok 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/662303079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: 5.7 scrub starts 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[37501]: 5.7 scrub ok 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: 6.14 scrub starts 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: 6.14 scrub ok 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:13:23.034 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/662303079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:23.035 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: 5.7 scrub starts 2024-03-26T04:13:23.035 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:22 smithi006 ceph-mon[29446]: 5.7 scrub ok 2024-03-26T04:13:23.059 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:13:23.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:13:23.305 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:23.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:23.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: 6.14 scrub starts 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: 6.14 scrub ok 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/662303079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: 5.7 scrub starts 2024-03-26T04:13:23.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:22 smithi086 ceph-mon[32965]: 5.7 scrub ok 2024-03-26T04:13:24.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[37501]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T04:13:24.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[37501]: 5.4 scrub starts 2024-03-26T04:13:24.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[37501]: 5.4 scrub ok 2024-03-26T04:13:24.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[37501]: 3.1d scrub starts 2024-03-26T04:13:24.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[37501]: 3.1d scrub ok 2024-03-26T04:13:24.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[29446]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T04:13:24.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[29446]: 5.4 scrub starts 2024-03-26T04:13:24.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[29446]: 5.4 scrub ok 2024-03-26T04:13:24.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[29446]: 3.1d scrub starts 2024-03-26T04:13:24.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:23 smithi006 ceph-mon[29446]: 3.1d scrub ok 2024-03-26T04:13:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:23 smithi086 ceph-mon[32965]: pgmap v1211: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 851 B/s rd, 0 op/s 2024-03-26T04:13:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:23 smithi086 ceph-mon[32965]: 5.4 scrub starts 2024-03-26T04:13:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:23 smithi086 ceph-mon[32965]: 5.4 scrub ok 2024-03-26T04:13:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:23 smithi086 ceph-mon[32965]: 3.1d scrub starts 2024-03-26T04:13:24.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:23 smithi086 ceph-mon[32965]: 3.1d scrub ok 2024-03-26T04:13:24.625 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:24] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:25.239 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:13:25.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[29446]: 5.18 scrub starts 2024-03-26T04:13:25.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[29446]: 5.18 scrub ok 2024-03-26T04:13:25.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[29446]: 6.19 scrub starts 2024-03-26T04:13:25.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[29446]: 6.19 scrub ok 2024-03-26T04:13:25.240 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[29446]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:13:25.241 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[37501]: 5.18 scrub starts 2024-03-26T04:13:25.241 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[37501]: 5.18 scrub ok 2024-03-26T04:13:25.241 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[37501]: 6.19 scrub starts 2024-03-26T04:13:25.241 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[37501]: 6.19 scrub ok 2024-03-26T04:13:25.241 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:24 smithi006 ceph-mon[37501]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:13:25.241 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":557,"fsid":"de980954-eb22-11ee-95d0-87774f69a715","created":"2024-03-26T03:43:14.534401+0000","modified":"2024-03-26T04:13:24.654035+0000","last_up_change":"2024-03-26T04:05:28.913716+0000","last_in_change":"2024-03-26T04:12:14.632752+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T03:46:30.183658+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"289","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"271","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T03:48:41.622497+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":69,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T03:48:42.191818+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T03:48:44.079106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T03:48:46.035261+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T03:48:48.086810+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1d1c84cf-5fa7-437e-8004-3b7aca831d6e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":526,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":1246039834}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":1246039834}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":1246039834}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":1246039834}]},"public_addr":"172.21.15.6:6801/1246039834","cluster_addr":"172.21.15.6:6802/1246039834","heartbeat_back_addr":"172.21.15.6:6804/1246039834","heartbeat_front_addr":"172.21.15.6:6803/1246039834","state":["exists","up"]},{"osd":1,"uuid":"753f7459-62ec-447b-a0da-64d06e813f16","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":260311411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":260311411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":260311411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":260311411}]},"public_addr":"172.21.15.6:6805/260311411","cluster_addr":"172.21.15.6:6806/260311411","heartbeat_back_addr":"172.21.15.6:6808/260311411","heartbeat_front_addr":"172.21.15.6:6807/260311411","state":["exists","up"]},{"osd":2,"uuid":"b7d7409f-341a-4711-8570-92fe9082a528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":18,"last_clean_end":335,"up_from":378,"up_thru":523,"down_at":336,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":2571159281}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":2571159281}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6812","nonce":2571159281}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":2571159281}]},"public_addr":"172.21.15.6:6809/2571159281","cluster_addr":"172.21.15.6:6810/2571159281","heartbeat_back_addr":"172.21.15.6:6812/2571159281","heartbeat_front_addr":"172.21.15.6:6811/2571159281","state":["exists","up"]},{"osd":3,"uuid":"a91459a0-dee4-420c-ae2f-940756c985a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6813","nonce":2795883342}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6814","nonce":2795883342}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6816","nonce":2795883342}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6815","nonce":2795883342}]},"public_addr":"172.21.15.6:6813/2795883342","cluster_addr":"172.21.15.6:6814/2795883342","heartbeat_back_addr":"172.21.15.6:6816/2795883342","heartbeat_front_addr":"172.21.15.6:6815/2795883342","state":["exists","up"]},{"osd":4,"uuid":"087da212-7bf7-4982-b7cc-11f3f66c7ff8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":931538411}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6801","nonce":931538411}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6803","nonce":931538411}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":931538411}]},"public_addr":"172.21.15.86:6800/931538411","cluster_addr":"172.21.15.86:6801/931538411","heartbeat_back_addr":"172.21.15.86:6803/931538411","heartbeat_front_addr":"172.21.15.86:6802/931538411","state":["exists","up"]},{"osd":5,"uuid":"7a688745-b213-4bbc-a735-42ffd2a5bfdc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":41,"last_clean_end":167,"up_from":180,"up_thru":532,"down_at":168,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":909197223}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6805","nonce":909197223}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6807","nonce":909197223}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":909197223}]},"public_addr":"172.21.15.86:6804/909197223","cluster_addr":"172.21.15.86:6805/909197223","heartbeat_back_addr":"172.21.15.86:6807/909197223","heartbeat_front_addr":"172.21.15.86:6806/909197223","state":["exists","up"]},{"osd":6,"uuid":"58b58555-59a3-4620-a15f-024f7041713f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1580133777}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6809","nonce":1580133777}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6811","nonce":1580133777}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1580133777}]},"public_addr":"172.21.15.86:6808/1580133777","cluster_addr":"172.21.15.86:6809/1580133777","heartbeat_back_addr":"172.21.15.86:6811/1580133777","heartbeat_front_addr":"172.21.15.86:6810/1580133777","state":["exists","up"]},{"osd":7,"uuid":"49b1833d-0a24-4168-a195-47b8ac1d9bc7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":546,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":3905426306}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6813","nonce":3905426306}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6815","nonce":3905426306}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":3905426306}]},"public_addr":"172.21.15.86:6812/3905426306","cluster_addr":"172.21.15.86:6813/3905426306","heartbeat_back_addr":"172.21.15.86:6815/3905426306","heartbeat_front_addr":"172.21.15.86:6814/3905426306","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:11:55.308972+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:07.061100+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:13.481504+0000","dead_epoch":335},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:12:12.354011+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:21.475916+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:09.897542+0000","dead_epoch":167},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:12:25.474293+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:13:03.316688+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.2","osds":[7,4,6]}],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":4,"to":2}]},{"pgid":"4.10","mappings":[{"from":3,"to":5}]},{"pgid":"4.17","mappings":[{"from":1,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.6:0/1978459317":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/1771670410":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/3964053112":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/14180211":"2024-03-27T03:49:23.135700+0000","172.21.15.6:6800/2547319348":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/601470125":"2024-03-27T03:44:16.095969+0000","172.21.15.6:6800/3870917014":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1892511221":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/435737617":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/93567168":"2024-03-27T03:49:23.135700+0000","172.21.15.6:0/2150730999":"2024-03-27T03:43:49.288943+0000","172.21.15.6:6800/1166209788":"2024-03-27T03:43:49.288943+0000","172.21.15.6:0/1248481299":"2024-03-27T03:44:16.095969+0000","172.21.15.6:0/1832156537":"2024-03-27T03:44:16.095969+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:13:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:24 smithi086 ceph-mon[32965]: 5.18 scrub starts 2024-03-26T04:13:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:24 smithi086 ceph-mon[32965]: 5.18 scrub ok 2024-03-26T04:13:25.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:24 smithi086 ceph-mon[32965]: 6.19 scrub starts 2024-03-26T04:13:25.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:24 smithi086 ceph-mon[32965]: 6.19 scrub ok 2024-03-26T04:13:25.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:24 smithi086 ceph-mon[32965]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:13:25.988 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:25.997 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:13:25.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:13:25.998 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:13:25.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T04:13:25.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T04:13:25.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T04:13:26.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T04:13:26.000 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T04:13:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 5.a scrub starts 2024-03-26T04:13:26.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 5.a scrub ok 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 6.18 scrub starts 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 5.1 scrub starts 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 5.1 scrub ok 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 6.18 scrub ok 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3986704341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 6.1 deep-scrub starts 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: 6.1 deep-scrub ok 2024-03-26T04:13:26.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:25 smithi086 ceph-mon[32965]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:13:26.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 5.a scrub starts 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 5.a scrub ok 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 6.18 scrub starts 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 5.1 scrub starts 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 5.1 scrub ok 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 6.18 scrub ok 2024-03-26T04:13:26.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3986704341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 6.1 deep-scrub starts 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: 6.1 deep-scrub ok 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[37501]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: pgmap v1213: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 5.a scrub starts 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 5.a scrub ok 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:26.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 6.18 scrub starts 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 5.1 scrub starts 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 5.1 scrub ok 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 6.18 scrub ok 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3986704341' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 6.1 deep-scrub starts 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: 6.1 deep-scrub ok 2024-03-26T04:13:26.392 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:25 smithi006 ceph-mon[29446]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:13:26.446 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.528 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.533 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.533 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.537 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:26.964 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:27.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 4.1f scrub starts 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 4.1f scrub ok 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 4.b scrub starts 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 4.b scrub ok 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 3.c scrub starts 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: 3.c scrub ok 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:27.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:26 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:27.354 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:27.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 4.1f scrub starts 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 4.1f scrub ok 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 4.b scrub starts 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 4.b scrub ok 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 3.c scrub starts 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: 3.c scrub ok 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:27.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 4.1f scrub starts 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 4.1f scrub ok 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 4.b scrub starts 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 4.b scrub ok 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 3.c scrub starts 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: 3.c scrub ok 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:27.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:26 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-26T04:13:28.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:28.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T04:13:28.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: 5.f scrub starts 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: 5.f scrub ok 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: 5.19 scrub starts 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[37501]: 5.19 scrub ok 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: 5.f scrub starts 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: 5.f scrub ok 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: 5.19 scrub starts 2024-03-26T04:13:28.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:28 smithi006 ceph-mon[29446]: 5.19 scrub ok 2024-03-26T04:13:28.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: pgmap v1215: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: 5.f scrub starts 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: 5.f scrub ok 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: 5.19 scrub starts 2024-03-26T04:13:28.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:28 smithi086 ceph-mon[32965]: 5.19 scrub ok 2024-03-26T04:13:29.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[29446]: 5.5 scrub starts 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[29446]: 5.5 scrub ok 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[29446]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[29446]: 1.0 scrub starts 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[29446]: 1.0 scrub ok 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[37501]: 5.5 scrub starts 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[37501]: 5.5 scrub ok 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[37501]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:13:29.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[37501]: 1.0 scrub starts 2024-03-26T04:13:29.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:29 smithi006 ceph-mon[37501]: 1.0 scrub ok 2024-03-26T04:13:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:29 smithi086 ceph-mon[32965]: 5.5 scrub starts 2024-03-26T04:13:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:29 smithi086 ceph-mon[32965]: 5.5 scrub ok 2024-03-26T04:13:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:29 smithi086 ceph-mon[32965]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:13:29.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:29 smithi086 ceph-mon[32965]: 1.0 scrub starts 2024-03-26T04:13:29.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:29 smithi086 ceph-mon[32965]: 1.0 scrub ok 2024-03-26T04:13:30.095 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: pgmap v1218: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: 3.6 scrub starts 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: 3.6 scrub ok 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: 4.1 scrub starts 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: 4.1 scrub ok 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[37501]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: pgmap v1218: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:30.096 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: 3.6 scrub starts 2024-03-26T04:13:30.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: 3.6 scrub ok 2024-03-26T04:13:30.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: 4.1 scrub starts 2024-03-26T04:13:30.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: 4.1 scrub ok 2024-03-26T04:13:30.097 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:30 smithi006 ceph-mon[29446]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:13:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: pgmap v1218: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: 3.6 scrub starts 2024-03-26T04:13:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: 3.6 scrub ok 2024-03-26T04:13:30.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: 4.1 scrub starts 2024-03-26T04:13:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: 4.1 scrub ok 2024-03-26T04:13:30.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:30 smithi086 ceph-mon[32965]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:13:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 4.14 scrub starts 2024-03-26T04:13:31.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 4.14 scrub ok 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 1.1 scrub starts 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 1.1 scrub ok 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 3.4 deep-scrub starts 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[37501]: 3.4 deep-scrub ok 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 4.14 scrub starts 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 4.14 scrub ok 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 1.1 scrub starts 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 1.1 scrub ok 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 3.4 deep-scrub starts 2024-03-26T04:13:31.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:31 smithi006 ceph-mon[29446]: 3.4 deep-scrub ok 2024-03-26T04:13:31.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 4.14 scrub starts 2024-03-26T04:13:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 4.14 scrub ok 2024-03-26T04:13:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 1.1 scrub starts 2024-03-26T04:13:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 1.1 scrub ok 2024-03-26T04:13:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 3.4 deep-scrub starts 2024-03-26T04:13:31.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:31 smithi086 ceph-mon[32965]: 3.4 deep-scrub ok 2024-03-26T04:13:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:32 smithi086 ceph-mon[32965]: pgmap v1220: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:13:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:32 smithi086 ceph-mon[32965]: 3.10 scrub starts 2024-03-26T04:13:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:32 smithi086 ceph-mon[32965]: 3.10 scrub ok 2024-03-26T04:13:32.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:32 smithi086 ceph-mon[32965]: 6.16 scrub starts 2024-03-26T04:13:32.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:32 smithi086 ceph-mon[32965]: 6.16 scrub ok 2024-03-26T04:13:32.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[37501]: pgmap v1220: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:13:32.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[37501]: 3.10 scrub starts 2024-03-26T04:13:32.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[37501]: 3.10 scrub ok 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[37501]: 6.16 scrub starts 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[37501]: 6.16 scrub ok 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[29446]: pgmap v1220: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[29446]: 3.10 scrub starts 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[29446]: 3.10 scrub ok 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[29446]: 6.16 scrub starts 2024-03-26T04:13:32.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:32 smithi006 ceph-mon[29446]: 6.16 scrub ok 2024-03-26T04:13:33.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:33 smithi006 ceph-mon[37501]: 4.9 deep-scrub starts 2024-03-26T04:13:33.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:33 smithi006 ceph-mon[37501]: 4.9 deep-scrub ok 2024-03-26T04:13:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:33 smithi006 ceph-mon[29446]: 4.9 deep-scrub starts 2024-03-26T04:13:33.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:33 smithi006 ceph-mon[29446]: 4.9 deep-scrub ok 2024-03-26T04:13:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:33 smithi086 ceph-mon[32965]: 4.9 deep-scrub starts 2024-03-26T04:13:33.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:33 smithi086 ceph-mon[32965]: 4.9 deep-scrub ok 2024-03-26T04:13:34.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[37501]: 3.18 scrub starts 2024-03-26T04:13:34.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[37501]: 3.18 scrub ok 2024-03-26T04:13:34.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[37501]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T04:13:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[29446]: 3.18 scrub starts 2024-03-26T04:13:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[29446]: 3.18 scrub ok 2024-03-26T04:13:34.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-mon[29446]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T04:13:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:34 smithi086 ceph-mon[32965]: 3.18 scrub starts 2024-03-26T04:13:34.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:34 smithi086 ceph-mon[32965]: 3.18 scrub ok 2024-03-26T04:13:34.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:34 smithi086 ceph-mon[32965]: pgmap v1221: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 18 B/s, 0 objects/s recovering 2024-03-26T04:13:34.445 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:34 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:34] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:35.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[37501]: 3.17 scrub starts 2024-03-26T04:13:35.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[37501]: 3.17 scrub ok 2024-03-26T04:13:35.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[37501]: 6.1d scrub starts 2024-03-26T04:13:35.319 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[29446]: 3.17 scrub starts 2024-03-26T04:13:35.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[29446]: 3.17 scrub ok 2024-03-26T04:13:35.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:35 smithi006 ceph-mon[29446]: 6.1d scrub starts 2024-03-26T04:13:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:35 smithi086 ceph-mon[32965]: 3.17 scrub starts 2024-03-26T04:13:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:35 smithi086 ceph-mon[32965]: 3.17 scrub ok 2024-03-26T04:13:35.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:35 smithi086 ceph-mon[32965]: 6.1d scrub starts 2024-03-26T04:13:36.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:35 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: 6.1d scrub ok 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: 4.e scrub starts 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: 4.e scrub ok 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: 6.5 scrub starts 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[37501]: 6.5 scrub ok 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: 6.1d scrub ok 2024-03-26T04:13:36.269 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: 4.e scrub starts 2024-03-26T04:13:36.270 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: 4.e scrub ok 2024-03-26T04:13:36.270 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: 6.5 scrub starts 2024-03-26T04:13:36.270 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:36 smithi006 ceph-mon[29446]: 6.5 scrub ok 2024-03-26T04:13:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: pgmap v1222: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 666 B/s rd, 0 op/s; 14 B/s, 0 objects/s recovering 2024-03-26T04:13:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: 6.1d scrub ok 2024-03-26T04:13:36.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: 4.e scrub starts 2024-03-26T04:13:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: 4.e scrub ok 2024-03-26T04:13:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: 6.5 scrub starts 2024-03-26T04:13:36.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:36 smithi086 ceph-mon[32965]: 6.5 scrub ok 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[37501]: 5.8 scrub starts 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[37501]: 5.8 scrub ok 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[37501]: 6.0 scrub starts 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[37501]: 6.0 scrub ok 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[29446]: 5.8 scrub starts 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[29446]: 5.8 scrub ok 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[29446]: 6.0 scrub starts 2024-03-26T04:13:37.129 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:37 smithi006 ceph-mon[29446]: 6.0 scrub ok 2024-03-26T04:13:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:37 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:37 smithi086 ceph-mon[32965]: 5.8 scrub starts 2024-03-26T04:13:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:37 smithi086 ceph-mon[32965]: 5.8 scrub ok 2024-03-26T04:13:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:37 smithi086 ceph-mon[32965]: 6.0 scrub starts 2024-03-26T04:13:37.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:37 smithi086 ceph-mon[32965]: 6.0 scrub ok 2024-03-26T04:13:38.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:38 smithi086 ceph-mon[32965]: pgmap v1223: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-26T04:13:38.340 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:38 smithi006 ceph-mon[37501]: pgmap v1223: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-26T04:13:38.340 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:38 smithi006 ceph-mon[29446]: pgmap v1223: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 12 B/s, 0 objects/s recovering 2024-03-26T04:13:39.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[29446]: 4.0 scrub starts 2024-03-26T04:13:39.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[29446]: 4.0 scrub ok 2024-03-26T04:13:39.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[29446]: 5.1e scrub starts 2024-03-26T04:13:39.040 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[29446]: 5.1e scrub ok 2024-03-26T04:13:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:39 smithi086 ceph-mon[32965]: 4.0 scrub starts 2024-03-26T04:13:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:39 smithi086 ceph-mon[32965]: 4.0 scrub ok 2024-03-26T04:13:39.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:39 smithi086 ceph-mon[32965]: 5.1e scrub starts 2024-03-26T04:13:39.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:39 smithi086 ceph-mon[32965]: 5.1e scrub ok 2024-03-26T04:13:39.346 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[37501]: 4.0 scrub starts 2024-03-26T04:13:39.346 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[37501]: 4.0 scrub ok 2024-03-26T04:13:39.346 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[37501]: 5.1e scrub starts 2024-03-26T04:13:39.346 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:39 smithi006 ceph-mon[37501]: 5.1e scrub ok 2024-03-26T04:13:40.153 INFO:teuthology.orchestra.run.smithi006.stdout:137438953790 2024-03-26T04:13:40.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T04:13:40.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 5.13 scrub starts 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 5.13 scrub ok 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: pgmap v1224: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 6.1e deep-scrub starts 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 6.1e deep-scrub ok 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 4.4 scrub starts 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: 4.4 scrub ok 2024-03-26T04:13:40.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:40 smithi086 ceph-mon[32965]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:13:40.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:40.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 5.13 scrub starts 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 5.13 scrub ok 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: pgmap v1224: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 6.1e deep-scrub starts 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 6.1e deep-scrub ok 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 4.4 scrub starts 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: 4.4 scrub ok 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[37501]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:13:40.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 5.13 scrub starts 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 5.13 scrub ok 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: pgmap v1224: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 10 B/s, 0 objects/s recovering 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 6.1e deep-scrub starts 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 6.1e deep-scrub ok 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 4.4 scrub starts 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: 4.4 scrub ok 2024-03-26T04:13:40.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:40 smithi006 ceph-mon[29446]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: 5.9 deep-scrub starts 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: 5.9 deep-scrub ok 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: 4.10 scrub starts 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: 4.10 scrub ok 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: purged_snaps scrub starts 2024-03-26T04:13:41.302 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[37501]: purged_snaps scrub ok 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: 5.9 deep-scrub starts 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: 5.9 deep-scrub ok 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: 4.10 scrub starts 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: 4.10 scrub ok 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: purged_snaps scrub starts 2024-03-26T04:13:41.303 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:41 smithi006 ceph-mon[29446]: purged_snaps scrub ok 2024-03-26T04:13:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: 5.9 deep-scrub starts 2024-03-26T04:13:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: 5.9 deep-scrub ok 2024-03-26T04:13:41.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: 4.10 scrub starts 2024-03-26T04:13:41.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: 4.10 scrub ok 2024-03-26T04:13:41.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:41.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: purged_snaps scrub starts 2024-03-26T04:13:41.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:41 smithi086 ceph-mon[32965]: purged_snaps scrub ok 2024-03-26T04:13:42.025 INFO:teuthology.orchestra.run.smithi006.stdout:773094113494 2024-03-26T04:13:42.026 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T04:13:42.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: pgmap v1227: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:42.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: 3.11 scrub starts 2024-03-26T04:13:42.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: 3.11 scrub ok 2024-03-26T04:13:42.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:13:42.319 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: 3.1 deep-scrub starts 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[37501]: 3.1 deep-scrub ok 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: pgmap v1227: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: 3.11 scrub starts 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: 3.11 scrub ok 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: 3.1 deep-scrub starts 2024-03-26T04:13:42.320 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:42 smithi006 ceph-mon[29446]: 3.1 deep-scrub ok 2024-03-26T04:13:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: pgmap v1227: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: 3.11 scrub starts 2024-03-26T04:13:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: 3.11 scrub ok 2024-03-26T04:13:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:13:42.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: 3.1 deep-scrub starts 2024-03-26T04:13:42.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:42 smithi086 ceph-mon[32965]: 3.1 deep-scrub ok 2024-03-26T04:13:42.621 INFO:teuthology.orchestra.run.smithi006.stdout:214748365110 2024-03-26T04:13:42.621 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T04:13:43.116 INFO:teuthology.orchestra.run.smithi006.stdout:34359738706 2024-03-26T04:13:43.116 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:13:43.321 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[37501]: 1.6 scrub starts 2024-03-26T04:13:43.321 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[37501]: 1.6 scrub ok 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[37501]: 4.c scrub starts 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[37501]: 4.c scrub ok 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[29446]: 1.6 scrub starts 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[29446]: 1.6 scrub ok 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[29446]: 4.c scrub starts 2024-03-26T04:13:43.322 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:43 smithi006 ceph-mon[29446]: 4.c scrub ok 2024-03-26T04:13:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:43 smithi086 ceph-mon[32965]: 1.6 scrub starts 2024-03-26T04:13:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:43 smithi086 ceph-mon[32965]: 1.6 scrub ok 2024-03-26T04:13:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:43 smithi086 ceph-mon[32965]: 4.c scrub starts 2024-03-26T04:13:43.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:43 smithi086 ceph-mon[32965]: 4.c scrub ok 2024-03-26T04:13:43.653 INFO:teuthology.orchestra.run.smithi006.stdout:107374182723 2024-03-26T04:13:43.654 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T04:13:44.006 INFO:teuthology.orchestra.run.smithi006.stdout:55834575181 2024-03-26T04:13:44.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: pgmap v1229: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 2.1 scrub starts 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 2.1 scrub ok 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 3.13 deep-scrub starts 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 3.13 deep-scrub ok 2024-03-26T04:13:44.234 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 3.3 scrub starts 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[29446]: 3.3 scrub ok 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: pgmap v1229: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 2.1 scrub starts 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 2.1 scrub ok 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 3.13 deep-scrub starts 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 3.13 deep-scrub ok 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 3.3 scrub starts 2024-03-26T04:13:44.235 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-mon[37501]: 3.3 scrub ok 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: pgmap v1229: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 2.1 scrub starts 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 2.1 scrub ok 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 3.13 deep-scrub starts 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 3.13 deep-scrub ok 2024-03-26T04:13:44.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 3.3 scrub starts 2024-03-26T04:13:44.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:44 smithi086 ceph-mon[32965]: 3.3 scrub ok 2024-03-26T04:13:44.635 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:44] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:44.674 INFO:teuthology.orchestra.run.smithi006.stdout:253403070767 2024-03-26T04:13:44.674 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T04:13:45.332 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 4.1c deep-scrub starts 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 4.1c deep-scrub ok 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 5.1a deep-scrub starts 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 5.1a deep-scrub ok 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 3.a scrub starts 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[37501]: 3.a scrub ok 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 4.1c deep-scrub starts 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 4.1c deep-scrub ok 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 5.1a deep-scrub starts 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 5.1a deep-scrub ok 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:13:45.333 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 3.a scrub starts 2024-03-26T04:13:45.334 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:45 smithi006 ceph-mon[29446]: 3.a scrub ok 2024-03-26T04:13:45.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 4.1c deep-scrub starts 2024-03-26T04:13:45.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 4.1c deep-scrub ok 2024-03-26T04:13:45.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 5.1a deep-scrub starts 2024-03-26T04:13:45.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 5.1a deep-scrub ok 2024-03-26T04:13:45.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:13:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 3.a scrub starts 2024-03-26T04:13:45.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-mon[32965]: 3.a scrub ok 2024-03-26T04:13:45.620 INFO:teuthology.orchestra.run.smithi006.stdout:1623497637989 2024-03-26T04:13:45.620 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:13:45.664 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:45.665 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:45.666 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:45.674 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:46.057 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:45 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:46.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[37501]: pgmap v1231: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 363 B/s rd, 0 op/s 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[37501]: 4.1d scrub starts 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[37501]: 4.1d scrub ok 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[37501]: 3.1f scrub starts 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[37501]: 3.1f scrub ok 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[29446]: pgmap v1231: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 363 B/s rd, 0 op/s 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[29446]: 4.1d scrub starts 2024-03-26T04:13:46.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[29446]: 4.1d scrub ok 2024-03-26T04:13:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[29446]: 3.1f scrub starts 2024-03-26T04:13:46.141 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:46 smithi006 ceph-mon[29446]: 3.1f scrub ok 2024-03-26T04:13:46.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:46 smithi086 ceph-mon[32965]: pgmap v1231: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 363 B/s rd, 0 op/s 2024-03-26T04:13:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:46 smithi086 ceph-mon[32965]: 4.1d scrub starts 2024-03-26T04:13:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:46 smithi086 ceph-mon[32965]: 4.1d scrub ok 2024-03-26T04:13:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:46 smithi086 ceph-mon[32965]: 3.1f scrub starts 2024-03-26T04:13:46.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:46 smithi086 ceph-mon[32965]: 3.1f scrub ok 2024-03-26T04:13:46.662 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:46.664 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:46.665 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:47 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:47 smithi086 ceph-mon[32965]: 6.1a scrub starts 2024-03-26T04:13:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:47 smithi086 ceph-mon[32965]: 6.1a scrub ok 2024-03-26T04:13:47.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:47 smithi086 ceph-mon[32965]: 4.12 scrub starts 2024-03-26T04:13:47.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:47 smithi086 ceph-mon[32965]: 4.12 scrub ok 2024-03-26T04:13:47.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:47.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[37501]: 6.1a scrub starts 2024-03-26T04:13:47.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[37501]: 6.1a scrub ok 2024-03-26T04:13:47.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[37501]: 4.12 scrub starts 2024-03-26T04:13:47.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[37501]: 4.12 scrub ok 2024-03-26T04:13:47.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:47.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[29446]: 6.1a scrub starts 2024-03-26T04:13:47.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[29446]: 6.1a scrub ok 2024-03-26T04:13:47.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[29446]: 4.12 scrub starts 2024-03-26T04:13:47.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:47 smithi006 ceph-mon[29446]: 4.12 scrub ok 2024-03-26T04:13:48.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: pgmap v1232: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 3.2 scrub starts 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 3.2 scrub ok 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 5.1f scrub starts 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 5.1f scrub ok 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 5.b deep-scrub starts 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 5.b deep-scrub ok 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 4.7 scrub starts 2024-03-26T04:13:48.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:48 smithi086 ceph-mon[32965]: 4.7 scrub ok 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: pgmap v1232: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 3.2 scrub starts 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 3.2 scrub ok 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 5.1f scrub starts 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 5.1f scrub ok 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 5.b deep-scrub starts 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 5.b deep-scrub ok 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 4.7 scrub starts 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[37501]: 4.7 scrub ok 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: pgmap v1232: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1020 B/s rd, 0 op/s 2024-03-26T04:13:48.384 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 3.2 scrub starts 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 3.2 scrub ok 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 5.1f scrub starts 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 5.1f scrub ok 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 5.b deep-scrub starts 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 5.b deep-scrub ok 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 4.7 scrub starts 2024-03-26T04:13:48.385 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:48 smithi006 ceph-mon[29446]: 4.7 scrub ok 2024-03-26T04:13:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:49 smithi086 ceph-mon[32965]: 6.b scrub starts 2024-03-26T04:13:49.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:49 smithi086 ceph-mon[32965]: 6.b scrub ok 2024-03-26T04:13:49.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:49 smithi006 ceph-mon[37501]: 6.b scrub starts 2024-03-26T04:13:49.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:49 smithi006 ceph-mon[37501]: 6.b scrub ok 2024-03-26T04:13:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:49 smithi006 ceph-mon[29446]: 6.b scrub starts 2024-03-26T04:13:49.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:49 smithi006 ceph-mon[29446]: 6.b scrub ok 2024-03-26T04:13:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[29446]: pgmap v1233: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[29446]: 4.6 scrub starts 2024-03-26T04:13:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[29446]: 4.6 scrub ok 2024-03-26T04:13:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[29446]: 4.15 scrub starts 2024-03-26T04:13:50.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[29446]: 4.15 scrub ok 2024-03-26T04:13:50.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[37501]: pgmap v1233: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:50.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[37501]: 4.6 scrub starts 2024-03-26T04:13:50.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[37501]: 4.6 scrub ok 2024-03-26T04:13:50.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[37501]: 4.15 scrub starts 2024-03-26T04:13:50.141 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:50 smithi006 ceph-mon[37501]: 4.15 scrub ok 2024-03-26T04:13:50.215 INFO:teuthology.orchestra.run.smithi006.stdout:773094113496 2024-03-26T04:13:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:50 smithi086 ceph-mon[32965]: pgmap v1233: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:13:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:50 smithi086 ceph-mon[32965]: 4.6 scrub starts 2024-03-26T04:13:50.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:50 smithi086 ceph-mon[32965]: 4.6 scrub ok 2024-03-26T04:13:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:50 smithi086 ceph-mon[32965]: 4.15 scrub starts 2024-03-26T04:13:50.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:50 smithi086 ceph-mon[32965]: 4.15 scrub ok 2024-03-26T04:13:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:51 smithi086 ceph-mon[32965]: 4.11 scrub starts 2024-03-26T04:13:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:51 smithi086 ceph-mon[32965]: 4.11 scrub ok 2024-03-26T04:13:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:51 smithi086 ceph-mon[32965]: 5.6 scrub starts 2024-03-26T04:13:51.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:51 smithi086 ceph-mon[32965]: 5.6 scrub ok 2024-03-26T04:13:51.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:51 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2855045901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:13:51.346 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[29446]: 4.11 scrub starts 2024-03-26T04:13:51.346 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[29446]: 4.11 scrub ok 2024-03-26T04:13:51.346 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[29446]: 5.6 scrub starts 2024-03-26T04:13:51.346 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[29446]: 5.6 scrub ok 2024-03-26T04:13:51.346 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2855045901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:13:51.347 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[37501]: 4.11 scrub starts 2024-03-26T04:13:51.347 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[37501]: 4.11 scrub ok 2024-03-26T04:13:51.347 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[37501]: 5.6 scrub starts 2024-03-26T04:13:51.347 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[37501]: 5.6 scrub ok 2024-03-26T04:13:51.347 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:51 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2855045901' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:13:51.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 773094113494 got 773094113496 for osd.5 2024-03-26T04:13:51.348 DEBUG:teuthology.parallel:result is None 2024-03-26T04:13:52.192 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[37501]: pgmap v1234: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[37501]: 3.f scrub starts 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[37501]: 3.f scrub ok 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[37501]: 4.16 scrub starts 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[37501]: 4.16 scrub ok 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[29446]: pgmap v1234: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[29446]: 3.f scrub starts 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[29446]: 3.f scrub ok 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[29446]: 4.16 scrub starts 2024-03-26T04:13:52.193 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:52 smithi006 ceph-mon[29446]: 4.16 scrub ok 2024-03-26T04:13:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:52 smithi086 ceph-mon[32965]: pgmap v1234: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:13:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:52 smithi086 ceph-mon[32965]: 3.f scrub starts 2024-03-26T04:13:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:52 smithi086 ceph-mon[32965]: 3.f scrub ok 2024-03-26T04:13:52.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:52 smithi086 ceph-mon[32965]: 4.16 scrub starts 2024-03-26T04:13:52.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:52 smithi086 ceph-mon[32965]: 4.16 scrub ok 2024-03-26T04:13:53.269 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[37501]: pgmap v1235: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[37501]: 5.3 scrub starts 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[37501]: 5.3 scrub ok 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[37501]: 6.4 scrub starts 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[37501]: 6.4 scrub ok 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[29446]: pgmap v1235: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[29446]: 5.3 scrub starts 2024-03-26T04:13:54.223 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[29446]: 5.3 scrub ok 2024-03-26T04:13:54.224 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[29446]: 6.4 scrub starts 2024-03-26T04:13:54.224 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-mon[29446]: 6.4 scrub ok 2024-03-26T04:13:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:54 smithi086 ceph-mon[32965]: pgmap v1235: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:13:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:54 smithi086 ceph-mon[32965]: 5.3 scrub starts 2024-03-26T04:13:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:54 smithi086 ceph-mon[32965]: 5.3 scrub ok 2024-03-26T04:13:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:54 smithi086 ceph-mon[32965]: 6.4 scrub starts 2024-03-26T04:13:54.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:54 smithi086 ceph-mon[32965]: 6.4 scrub ok 2024-03-26T04:13:54.393 INFO:teuthology.orchestra.run.smithi006.stdout:137438953793 2024-03-26T04:13:54.402 INFO:teuthology.orchestra.run.smithi006.stdout:107374182726 2024-03-26T04:13:54.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:13:54 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:13:54] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:13:54.977 INFO:teuthology.orchestra.run.smithi006.stdout:253403070770 2024-03-26T04:13:55.248 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3009499864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:13:55.248 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1375986194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:13:55.248 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/207526502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:13:55.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3009499864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:13:55.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1375986194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:13:55.249 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:55 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/207526502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:13:55.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182723 got 107374182726 for osd.3 2024-03-26T04:13:55.320 DEBUG:teuthology.parallel:result is None 2024-03-26T04:13:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3009499864' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:13:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1375986194' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:13:55.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:55 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/207526502' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:13:56.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:13:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:13:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: pgmap v1236: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-26T04:13:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 1.4 scrub starts 2024-03-26T04:13:56.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 1.4 scrub ok 2024-03-26T04:13:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 2.0 deep-scrub starts 2024-03-26T04:13:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 2.0 deep-scrub ok 2024-03-26T04:13:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 1.3 scrub starts 2024-03-26T04:13:56.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:56 smithi086 ceph-mon[32965]: 1.3 scrub ok 2024-03-26T04:13:56.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: pgmap v1236: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-26T04:13:56.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 1.4 scrub starts 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 1.4 scrub ok 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 2.0 deep-scrub starts 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 2.0 deep-scrub ok 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 1.3 scrub starts 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[29446]: 1.3 scrub ok 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: pgmap v1236: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1021 B/s rd, 0 op/s 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 1.4 scrub starts 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 1.4 scrub ok 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 2.0 deep-scrub starts 2024-03-26T04:13:56.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 2.0 deep-scrub ok 2024-03-26T04:13:56.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:13:56.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 1.3 scrub starts 2024-03-26T04:13:56.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:56 smithi006 ceph-mon[37501]: 1.3 scrub ok 2024-03-26T04:13:57.291 INFO:teuthology.orchestra.run.smithi006.stdout:34359738710 2024-03-26T04:13:57.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[37501]: 4.d deep-scrub starts 2024-03-26T04:13:57.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[37501]: 4.d deep-scrub ok 2024-03-26T04:13:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[29446]: 4.d deep-scrub starts 2024-03-26T04:13:57.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:57 smithi006 ceph-mon[29446]: 4.d deep-scrub ok 2024-03-26T04:13:57.554 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070767 got 253403070770 for osd.7 2024-03-26T04:13:57.554 DEBUG:teuthology.parallel:result is None 2024-03-26T04:13:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:57 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:13:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:57 smithi086 ceph-mon[32965]: 4.d deep-scrub starts 2024-03-26T04:13:57.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:57 smithi086 ceph-mon[32965]: 4.d deep-scrub ok 2024-03-26T04:13:58.174 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738706 got 34359738710 for osd.0 2024-03-26T04:13:58.174 DEBUG:teuthology.parallel:result is None 2024-03-26T04:13:58.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: pgmap v1237: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 4.17 deep-scrub starts 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 4.17 deep-scrub ok 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/447184811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 6.d scrub starts 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 3.0 scrub starts 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 3.0 scrub ok 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[37501]: 6.d scrub ok 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: pgmap v1237: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:58.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 4.17 deep-scrub starts 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 4.17 deep-scrub ok 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/447184811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 6.d scrub starts 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 3.0 scrub starts 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 3.0 scrub ok 2024-03-26T04:13:58.391 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:58 smithi006 ceph-mon[29446]: 6.d scrub ok 2024-03-26T04:13:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: pgmap v1237: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:13:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 4.17 deep-scrub starts 2024-03-26T04:13:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 4.17 deep-scrub ok 2024-03-26T04:13:58.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/447184811' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:13:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 6.d scrub starts 2024-03-26T04:13:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 3.0 scrub starts 2024-03-26T04:13:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 3.0 scrub ok 2024-03-26T04:13:58.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:58 smithi086 ceph-mon[32965]: 6.d scrub ok 2024-03-26T04:13:59.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.f scrub starts 2024-03-26T04:13:59.389 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.f scrub ok 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.9 scrub starts 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.9 scrub ok 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.a scrub starts 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 6.a scrub ok 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 4.f scrub starts 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[29446]: 4.f scrub ok 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.f scrub starts 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.f scrub ok 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.9 scrub starts 2024-03-26T04:13:59.390 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.9 scrub ok 2024-03-26T04:13:59.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.a scrub starts 2024-03-26T04:13:59.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 6.a scrub ok 2024-03-26T04:13:59.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 4.f scrub starts 2024-03-26T04:13:59.391 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:13:59 smithi006 ceph-mon[37501]: 4.f scrub ok 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.f scrub starts 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.f scrub ok 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.9 scrub starts 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.9 scrub ok 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.a scrub starts 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 6.a scrub ok 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 4.f scrub starts 2024-03-26T04:13:59.489 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:13:59 smithi086 ceph-mon[32965]: 4.f scrub ok 2024-03-26T04:13:59.536 INFO:teuthology.orchestra.run.smithi006.stdout:55834575186 2024-03-26T04:14:00.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953790 got 137438953793 for osd.4 2024-03-26T04:14:00.025 DEBUG:teuthology.parallel:result is None 2024-03-26T04:14:00.330 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[37501]: pgmap v1238: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:00.330 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[37501]: 4.1a scrub starts 2024-03-26T04:14:00.330 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[37501]: 4.1a scrub ok 2024-03-26T04:14:00.330 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/2978545384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:14:00.330 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[29446]: pgmap v1238: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:00.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[29446]: 4.1a scrub starts 2024-03-26T04:14:00.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[29446]: 4.1a scrub ok 2024-03-26T04:14:00.331 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:00 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/2978545384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:14:00.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:00 smithi086 ceph-mon[32965]: pgmap v1238: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:00.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:00 smithi086 ceph-mon[32965]: 4.1a scrub starts 2024-03-26T04:14:00.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:00 smithi086 ceph-mon[32965]: 4.1a scrub ok 2024-03-26T04:14:00.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:00 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/2978545384' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:14:00.648 INFO:teuthology.orchestra.run.smithi006.stdout:214748365113 2024-03-26T04:14:01.246 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:01 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3374408514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:14:01.247 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:01 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3374408514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:14:01.539 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575181 got 55834575186 for osd.1 2024-03-26T04:14:01.539 DEBUG:teuthology.parallel:result is None 2024-03-26T04:14:01.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:01 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3374408514' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:14:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: pgmap v1239: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 5.14 scrub starts 2024-03-26T04:14:02.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 5.14 scrub ok 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 5.1c scrub starts 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 5.1c scrub ok 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 4.8 scrub starts 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[37501]: 4.8 scrub ok 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: pgmap v1239: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 5.14 scrub starts 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 5.14 scrub ok 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 5.1c scrub starts 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 5.1c scrub ok 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 4.8 scrub starts 2024-03-26T04:14:02.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:02 smithi006 ceph-mon[29446]: 4.8 scrub ok 2024-03-26T04:14:02.198 INFO:teuthology.orchestra.run.smithi006.stdout:1623497637992 2024-03-26T04:14:02.573 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365110 got 214748365113 for osd.6 2024-03-26T04:14:02.573 DEBUG:teuthology.parallel:result is None 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: pgmap v1239: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 5.14 scrub starts 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 5.14 scrub ok 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 5.1c scrub starts 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 5.1c scrub ok 2024-03-26T04:14:02.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 4.8 scrub starts 2024-03-26T04:14:02.588 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:02 smithi086 ceph-mon[32965]: 4.8 scrub ok 2024-03-26T04:14:03.225 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[37501]: 4.2 scrub starts 2024-03-26T04:14:03.225 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[37501]: 4.2 scrub ok 2024-03-26T04:14:03.226 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/1446234267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:14:03.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[29446]: 4.2 scrub starts 2024-03-26T04:14:03.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[29446]: 4.2 scrub ok 2024-03-26T04:14:03.226 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:03 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/1446234267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:14:03.256 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1623497637989 got 1623497637992 for osd.2 2024-03-26T04:14:03.256 DEBUG:teuthology.parallel:result is None 2024-03-26T04:14:03.256 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-26T04:14:03.256 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:14:03.507 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:03 smithi086 ceph-mon[32965]: 4.2 scrub starts 2024-03-26T04:14:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:03 smithi086 ceph-mon[32965]: 4.2 scrub ok 2024-03-26T04:14:03.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:03 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/1446234267' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:14:04.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:04 smithi006 ceph-mon[29446]: pgmap v1240: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:04.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:04 smithi006 ceph-mon[37501]: pgmap v1240: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:04.575 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:04 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:14:04] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:14:04.587 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:04 smithi086 ceph-mon[32965]: pgmap v1240: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:05.292 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:14:05.295 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:14:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:05 smithi006 ceph-mon[37501]: pgmap v1241: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:05.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:05 smithi006 ceph-mon[37501]: 6.e scrub starts 2024-03-26T04:14:05.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:05 smithi006 ceph-mon[29446]: pgmap v1241: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:05.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:05 smithi006 ceph-mon[29446]: 6.e scrub starts 2024-03-26T04:14:06.086 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1241,"stamp":"2024-03-26T04:14:04.686938+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1958,"num_read_kb":1678,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1099,"num_bytes_recovered":1210983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2860,"ondisk_log_size":2860,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6413564,"kb_used_data":15932,"kb_used_omap":20,"kb_used_meta":6397547,"kb_avail":743547652,"statfs":{"total":767960285184,"available":761392795648,"internally_reserved":0,"allocated":16314368,"data_stored":9618632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20885,"internal_metadata":6551088747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003339"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1106,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.124667+0000","last_change":"2024-03-26T04:13:46.124667+0000","last_active":"2024-03-26T04:13:46.124667+0000","last_peered":"2024-03-26T04:13:46.124667+0000","last_clean":"2024-03-26T04:13:46.124667+0000","last_became_active":"2024-03-26T04:12:29.655731+0000","last_became_peered":"2024-03-26T04:12:29.655731+0000","last_unstale":"2024-03-26T04:13:46.124667+0000","last_undegraded":"2024-03-26T04:13:46.124667+0000","last_fullsized":"2024-03-26T04:13:46.124667+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.124561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.124561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.622643+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1106,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725361+0000","last_change":"2024-03-26T04:13:27.537492+0000","last_active":"2024-03-26T04:13:44.725361+0000","last_peered":"2024-03-26T04:13:44.725361+0000","last_clean":"2024-03-26T04:13:44.725361+0000","last_became_active":"2024-03-26T04:12:08.689302+0000","last_became_peered":"2024-03-26T04:12:08.689302+0000","last_unstale":"2024-03-26T04:13:44.725361+0000","last_undegraded":"2024-03-26T04:13:44.725361+0000","last_fullsized":"2024-03-26T04:13:44.725361+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.537375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:09.336550+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.537375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:50.160214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":103,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.202688+0000","last_change":"2024-03-26T04:13:45.202709+0000","last_active":"2024-03-26T04:13:45.202688+0000","last_peered":"2024-03-26T04:13:45.202688+0000","last_clean":"2024-03-26T04:13:45.202688+0000","last_became_active":"2024-03-26T04:12:21.681921+0000","last_became_peered":"2024-03-26T04:12:21.681921+0000","last_unstale":"2024-03-26T04:13:45.202688+0000","last_undegraded":"2024-03-26T04:13:45.202688+0000","last_fullsized":"2024-03-26T04:13:45.202688+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:45.202631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:32.371775+0000","last_clean_scrub_stamp":"2024-03-26T04:13:45.202631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:48.674388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"546'32","reported_seq":1231,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663557+0000","last_change":"2024-03-26T04:13:05.914382+0000","last_active":"2024-03-26T04:13:44.663557+0000","last_peered":"2024-03-26T04:13:44.663557+0000","last_clean":"2024-03-26T04:13:44.663557+0000","last_became_active":"2024-03-26T04:12:15.670323+0000","last_became_peered":"2024-03-26T04:12:15.670323+0000","last_unstale":"2024-03-26T04:13:44.663557+0000","last_undegraded":"2024-03-26T04:13:44.663557+0000","last_fullsized":"2024-03-26T04:13:44.663557+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'32","last_scrub_stamp":"2024-03-26T04:13:05.914240+0000","last_deep_scrub":"546'32","last_deep_scrub_stamp":"2024-03-26T04:13:05.914240+0000","last_clean_scrub_stamp":"2024-03-26T04:13:05.914240+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.043842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665018+0000","last_change":"2024-03-26T04:12:46.080880+0000","last_active":"2024-03-26T04:13:44.665018+0000","last_peered":"2024-03-26T04:13:44.665018+0000","last_clean":"2024-03-26T04:13:44.665018+0000","last_became_active":"2024-03-26T04:12:15.649855+0000","last_became_peered":"2024-03-26T04:12:15.649855+0000","last_unstale":"2024-03-26T04:13:44.665018+0000","last_undegraded":"2024-03-26T04:13:44.665018+0000","last_fullsized":"2024-03-26T04:13:44.665018+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.080779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.080779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.688251+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1088,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663570+0000","last_change":"2024-03-26T04:13:23.957403+0000","last_active":"2024-03-26T04:13:44.663570+0000","last_peered":"2024-03-26T04:13:44.663570+0000","last_clean":"2024-03-26T04:13:44.663570+0000","last_became_active":"2024-03-26T04:12:15.669617+0000","last_became_peered":"2024-03-26T04:12:15.669617+0000","last_unstale":"2024-03-26T04:13:44.663570+0000","last_undegraded":"2024-03-26T04:13:44.663570+0000","last_fullsized":"2024-03-26T04:13:44.663570+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.957300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.957300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:37.008656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665966+0000","last_change":"2024-03-26T04:13:14.818369+0000","last_active":"2024-03-26T04:13:44.665966+0000","last_peered":"2024-03-26T04:13:44.665966+0000","last_clean":"2024-03-26T04:13:44.665966+0000","last_became_active":"2024-03-26T04:12:15.653430+0000","last_became_peered":"2024-03-26T04:12:15.653430+0000","last_unstale":"2024-03-26T04:13:44.665966+0000","last_undegraded":"2024-03-26T04:13:44.665966+0000","last_fullsized":"2024-03-26T04:13:44.665966+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:14.818268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:13:14.818268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:26.009674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"448'63","reported_seq":137,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665598+0000","last_change":"2024-03-26T04:13:19.760651+0000","last_active":"2024-03-26T04:13:44.665598+0000","last_peered":"2024-03-26T04:13:44.665598+0000","last_clean":"2024-03-26T04:13:44.665598+0000","last_became_active":"2024-03-26T04:12:11.891462+0000","last_became_peered":"2024-03-26T04:12:11.891462+0000","last_unstale":"2024-03-26T04:13:44.665598+0000","last_undegraded":"2024-03-26T04:13:44.665598+0000","last_fullsized":"2024-03-26T04:13:44.665598+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'63","last_scrub_stamp":"2024-03-26T04:13:19.760505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.760505+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:25.575667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":82,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665905+0000","last_change":"2024-03-26T04:13:25.009186+0000","last_active":"2024-03-26T04:13:44.665905+0000","last_peered":"2024-03-26T04:13:44.665905+0000","last_clean":"2024-03-26T04:13:44.665905+0000","last_became_active":"2024-03-26T04:12:21.682966+0000","last_became_peered":"2024-03-26T04:12:21.682966+0000","last_unstale":"2024-03-26T04:13:44.665905+0000","last_undegraded":"2024-03-26T04:13:44.665905+0000","last_fullsized":"2024-03-26T04:13:44.665905+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:25.009071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:11:48.080284+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.009071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.427573+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200156+0000","last_change":"2024-03-26T04:13:13.603271+0000","last_active":"2024-03-26T04:13:45.200156+0000","last_peered":"2024-03-26T04:13:45.200156+0000","last_clean":"2024-03-26T04:13:45.200156+0000","last_became_active":"2024-03-26T04:12:08.697573+0000","last_became_peered":"2024-03-26T04:12:08.697573+0000","last_unstale":"2024-03-26T04:13:45.200156+0000","last_undegraded":"2024-03-26T04:13:45.200156+0000","last_fullsized":"2024-03-26T04:13:45.200156+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:13.603173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:13:13.603173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:30.358135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201353+0000","last_change":"2024-03-26T04:13:23.528861+0000","last_active":"2024-03-26T04:13:45.201353+0000","last_peered":"2024-03-26T04:13:45.201353+0000","last_clean":"2024-03-26T04:13:45.201353+0000","last_became_active":"2024-03-26T04:12:15.664653+0000","last_became_peered":"2024-03-26T04:12:15.664653+0000","last_unstale":"2024-03-26T04:13:45.201353+0000","last_undegraded":"2024-03-26T04:13:45.201353+0000","last_fullsized":"2024-03-26T04:13:45.201353+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.528745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.528745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:41.877652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"448'34","reported_seq":1219,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:59.063075+0000","last_change":"2024-03-26T04:13:59.063115+0000","last_active":"2024-03-26T04:13:59.063075+0000","last_peered":"2024-03-26T04:13:59.063075+0000","last_clean":"2024-03-26T04:13:59.063075+0000","last_became_active":"2024-03-26T04:12:11.894786+0000","last_became_peered":"2024-03-26T04:12:11.894786+0000","last_unstale":"2024-03-26T04:13:59.063075+0000","last_undegraded":"2024-03-26T04:13:59.063075+0000","last_fullsized":"2024-03-26T04:13:59.063075+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'34","last_scrub_stamp":"2024-03-26T04:13:59.062991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:59.062991+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:22.044808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200435+0000","last_change":"2024-03-26T04:13:24.654316+0000","last_active":"2024-03-26T04:13:45.200435+0000","last_peered":"2024-03-26T04:13:45.200435+0000","last_clean":"2024-03-26T04:13:45.200435+0000","last_became_active":"2024-03-26T04:12:11.888977+0000","last_became_peered":"2024-03-26T04:12:11.888977+0000","last_unstale":"2024-03-26T04:13:45.200435+0000","last_undegraded":"2024-03-26T04:13:45.200435+0000","last_fullsized":"2024-03-26T04:13:45.200435+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.654180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.654180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.856932+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048742+0000","last_change":"2024-03-26T04:13:44.047533+0000","last_active":"2024-03-26T04:13:45.048742+0000","last_peered":"2024-03-26T04:13:45.048742+0000","last_clean":"2024-03-26T04:13:45.048742+0000","last_became_active":"2024-03-26T04:12:08.677750+0000","last_became_peered":"2024-03-26T04:12:08.677750+0000","last_unstale":"2024-03-26T04:13:45.048742+0000","last_undegraded":"2024-03-26T04:13:45.048742+0000","last_fullsized":"2024-03-26T04:13:45.048742+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:44.047433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:44.047433+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.047433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:44.780691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":129,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201043+0000","last_change":"2024-03-26T04:12:57.557592+0000","last_active":"2024-03-26T04:13:45.201043+0000","last_peered":"2024-03-26T04:13:45.201043+0000","last_clean":"2024-03-26T04:13:45.201043+0000","last_became_active":"2024-03-26T04:12:03.859469+0000","last_became_peered":"2024-03-26T04:12:03.859469+0000","last_unstale":"2024-03-26T04:13:45.201043+0000","last_undegraded":"2024-03-26T04:13:45.201043+0000","last_fullsized":"2024-03-26T04:13:45.201043+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:57.557503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:35.315076+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.557503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:08.411348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"448'18","reported_seq":1201,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664289+0000","last_change":"2024-03-26T04:13:15.014520+0000","last_active":"2024-03-26T04:13:44.664289+0000","last_peered":"2024-03-26T04:13:44.664289+0000","last_clean":"2024-03-26T04:13:44.664289+0000","last_became_active":"2024-03-26T04:12:29.654762+0000","last_became_peered":"2024-03-26T04:12:29.654762+0000","last_unstale":"2024-03-26T04:13:44.664289+0000","last_undegraded":"2024-03-26T04:13:44.664289+0000","last_fullsized":"2024-03-26T04:13:44.664289+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'18","last_scrub_stamp":"2024-03-26T04:13:15.014415+0000","last_deep_scrub":"241'16","last_deep_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_clean_scrub_stamp":"2024-03-26T04:13:15.014415+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.629458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1128,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664560+0000","last_change":"2024-03-26T04:13:39.241627+0000","last_active":"2024-03-26T04:13:44.664560+0000","last_peered":"2024-03-26T04:13:44.664560+0000","last_clean":"2024-03-26T04:13:44.664560+0000","last_became_active":"2024-03-26T04:12:15.672724+0000","last_became_peered":"2024-03-26T04:12:15.672724+0000","last_unstale":"2024-03-26T04:13:44.664560+0000","last_undegraded":"2024-03-26T04:13:44.664560+0000","last_fullsized":"2024-03-26T04:13:44.664560+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:39.241518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:39.241518+0000","last_clean_scrub_stamp":"2024-03-26T04:13:39.241518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:44.840603+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1072,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.726041+0000","last_change":"2024-03-26T04:13:18.353035+0000","last_active":"2024-03-26T04:13:44.726041+0000","last_peered":"2024-03-26T04:13:44.726041+0000","last_clean":"2024-03-26T04:13:44.726041+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:13:44.726041+0000","last_undegraded":"2024-03-26T04:13:44.726041+0000","last_fullsized":"2024-03-26T04:13:44.726041+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.352950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.352950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:34.865310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663254+0000","last_change":"2024-03-26T04:12:46.691394+0000","last_active":"2024-03-26T04:13:44.663254+0000","last_peered":"2024-03-26T04:13:44.663254+0000","last_clean":"2024-03-26T04:13:44.663254+0000","last_became_active":"2024-03-26T04:12:21.686772+0000","last_became_peered":"2024-03-26T04:12:21.686772+0000","last_unstale":"2024-03-26T04:13:44.663254+0000","last_undegraded":"2024-03-26T04:13:44.663254+0000","last_fullsized":"2024-03-26T04:13:44.663254+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.691275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.691275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:59.880321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"564'46","reported_seq":1181,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898677+0000","last_change":"2024-03-26T04:13:43.945698+0000","last_active":"2024-03-26T04:13:44.898677+0000","last_peered":"2024-03-26T04:13:44.898677+0000","last_clean":"2024-03-26T04:13:44.898677+0000","last_became_active":"2024-03-26T04:12:11.880784+0000","last_became_peered":"2024-03-26T04:12:11.880784+0000","last_unstale":"2024-03-26T04:13:44.898677+0000","last_undegraded":"2024-03-26T04:13:44.898677+0000","last_fullsized":"2024-03-26T04:13:44.898677+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'46","last_scrub_stamp":"2024-03-26T04:13:43.945600+0000","last_deep_scrub":"564'46","last_deep_scrub_stamp":"2024-03-26T04:13:43.945600+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.945600+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:05.848463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664902+0000","last_change":"2024-03-26T04:12:35.953331+0000","last_active":"2024-03-26T04:13:44.664902+0000","last_peered":"2024-03-26T04:13:44.664902+0000","last_clean":"2024-03-26T04:13:44.664902+0000","last_became_active":"2024-03-26T04:12:15.655603+0000","last_became_peered":"2024-03-26T04:12:15.655603+0000","last_unstale":"2024-03-26T04:13:44.664902+0000","last_undegraded":"2024-03-26T04:13:44.664902+0000","last_fullsized":"2024-03-26T04:13:44.664902+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:35.953232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:12:35.953232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:00.094640+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1104,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664176+0000","last_change":"2024-03-26T04:12:48.968164+0000","last_active":"2024-03-26T04:13:44.664176+0000","last_peered":"2024-03-26T04:13:44.664176+0000","last_clean":"2024-03-26T04:13:44.664176+0000","last_became_active":"2024-03-26T04:12:11.894669+0000","last_became_peered":"2024-03-26T04:12:11.894669+0000","last_unstale":"2024-03-26T04:13:44.664176+0000","last_undegraded":"2024-03-26T04:13:44.664176+0000","last_fullsized":"2024-03-26T04:13:44.664176+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:48.968060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:48.968060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1076,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663970+0000","last_change":"2024-03-26T04:12:32.963874+0000","last_active":"2024-03-26T04:13:44.663970+0000","last_peered":"2024-03-26T04:13:44.663970+0000","last_clean":"2024-03-26T04:13:44.663970+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:13:44.663970+0000","last_undegraded":"2024-03-26T04:13:44.663970+0000","last_fullsized":"2024-03-26T04:13:44.663970+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:32.963756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:32.963756+0000","last_clean_scrub_stamp":"2024-03-26T04:12:32.963756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:57.321907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"514'36","reported_seq":127,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.729086+0000","last_change":"2024-03-26T04:13:44.729119+0000","last_active":"2024-03-26T04:13:44.729086+0000","last_peered":"2024-03-26T04:13:44.729086+0000","last_clean":"2024-03-26T04:13:44.729086+0000","last_became_active":"2024-03-26T04:12:11.900079+0000","last_became_peered":"2024-03-26T04:12:11.900079+0000","last_unstale":"2024-03-26T04:13:44.729086+0000","last_undegraded":"2024-03-26T04:13:44.729086+0000","last_fullsized":"2024-03-26T04:13:44.729086+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'36","last_scrub_stamp":"2024-03-26T04:13:44.729030+0000","last_deep_scrub":"514'36","last_deep_scrub_stamp":"2024-03-26T04:12:08.309354+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.729030+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.535514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":69,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049746+0000","last_change":"2024-03-26T04:13:06.668683+0000","last_active":"2024-03-26T04:13:45.049746+0000","last_peered":"2024-03-26T04:13:45.049746+0000","last_clean":"2024-03-26T04:13:45.049746+0000","last_became_active":"2024-03-26T04:13:06.668313+0000","last_became_peered":"2024-03-26T04:13:06.668313+0000","last_unstale":"2024-03-26T04:13:45.049746+0000","last_undegraded":"2024-03-26T04:13:45.049746+0000","last_fullsized":"2024-03-26T04:13:45.049746+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:12:51.678255+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:12:51.678255+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:20.460831+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"525'16","reported_seq":138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394717+0000","last_change":"2024-03-26T04:13:46.796970+0000","last_active":"2024-03-26T04:13:54.394717+0000","last_peered":"2024-03-26T04:13:54.394717+0000","last_clean":"2024-03-26T04:13:54.394717+0000","last_became_active":"2024-03-26T04:12:15.665266+0000","last_became_peered":"2024-03-26T04:12:15.665266+0000","last_unstale":"2024-03-26T04:13:54.394717+0000","last_undegraded":"2024-03-26T04:13:54.394717+0000","last_fullsized":"2024-03-26T04:13:54.394717+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'16","last_scrub_stamp":"2024-03-26T04:13:46.796865+0000","last_deep_scrub":"523'15","last_deep_scrub_stamp":"2024-03-26T04:12:17.024905+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.796865+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.010999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":1142,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725264+0000","last_change":"2024-03-26T04:12:59.294279+0000","last_active":"2024-03-26T04:13:44.725264+0000","last_peered":"2024-03-26T04:13:44.725264+0000","last_clean":"2024-03-26T04:13:44.725264+0000","last_became_active":"2024-03-26T04:12:11.892352+0000","last_became_peered":"2024-03-26T04:12:11.892352+0000","last_unstale":"2024-03-26T04:13:44.725264+0000","last_undegraded":"2024-03-26T04:13:44.725264+0000","last_fullsized":"2024-03-26T04:13:44.725264+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:12:59.294189+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:12:59.294189+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:09.887787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"448'42","reported_seq":1276,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665865+0000","last_change":"2024-03-26T04:12:58.672638+0000","last_active":"2024-03-26T04:13:44.665865+0000","last_peered":"2024-03-26T04:13:44.665865+0000","last_clean":"2024-03-26T04:13:44.665865+0000","last_became_active":"2024-03-26T04:12:11.896329+0000","last_became_peered":"2024-03-26T04:12:11.896329+0000","last_unstale":"2024-03-26T04:13:44.665865+0000","last_undegraded":"2024-03-26T04:13:44.665865+0000","last_fullsized":"2024-03-26T04:13:44.665865+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'42","last_scrub_stamp":"2024-03-26T04:12:58.672508+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:12:58.672508+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:58.737522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1090,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725663+0000","last_change":"2024-03-26T04:13:34.758039+0000","last_active":"2024-03-26T04:13:44.725663+0000","last_peered":"2024-03-26T04:13:44.725663+0000","last_clean":"2024-03-26T04:13:44.725663+0000","last_became_active":"2024-03-26T04:12:03.850434+0000","last_became_peered":"2024-03-26T04:12:03.850434+0000","last_unstale":"2024-03-26T04:13:44.725663+0000","last_undegraded":"2024-03-26T04:13:44.725663+0000","last_fullsized":"2024-03-26T04:13:44.725663+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:34.757931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:34.757931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:55.041262+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1096,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665576+0000","last_change":"2024-03-26T04:13:38.088706+0000","last_active":"2024-03-26T04:13:44.665576+0000","last_peered":"2024-03-26T04:13:44.665576+0000","last_clean":"2024-03-26T04:13:44.665576+0000","last_became_active":"2024-03-26T04:12:08.677498+0000","last_became_peered":"2024-03-26T04:12:08.677498+0000","last_unstale":"2024-03-26T04:13:44.665576+0000","last_undegraded":"2024-03-26T04:13:44.665576+0000","last_fullsized":"2024-03-26T04:13:44.665576+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:38.088621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:13:38.088621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.365215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664156+0000","last_change":"2024-03-26T04:13:32.600753+0000","last_active":"2024-03-26T04:13:44.664156+0000","last_peered":"2024-03-26T04:13:44.664156+0000","last_clean":"2024-03-26T04:13:44.664156+0000","last_became_active":"2024-03-26T04:12:29.659001+0000","last_became_peered":"2024-03-26T04:12:29.659001+0000","last_unstale":"2024-03-26T04:13:44.664156+0000","last_undegraded":"2024-03-26T04:13:44.664156+0000","last_fullsized":"2024-03-26T04:13:44.664156+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:32.600650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:32.600650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.249173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"527'41","reported_seq":132,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734338+0000","last_change":"2024-03-26T04:13:25.824502+0000","last_active":"2024-03-26T04:13:44.734338+0000","last_peered":"2024-03-26T04:13:44.734338+0000","last_clean":"2024-03-26T04:13:44.734338+0000","last_became_active":"2024-03-26T04:12:15.660617+0000","last_became_peered":"2024-03-26T04:12:15.660617+0000","last_unstale":"2024-03-26T04:13:44.734338+0000","last_undegraded":"2024-03-26T04:13:44.734338+0000","last_fullsized":"2024-03-26T04:13:44.734338+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'41","last_scrub_stamp":"2024-03-26T04:13:25.824320+0000","last_deep_scrub":"527'41","last_deep_scrub_stamp":"2024-03-26T04:12:22.934443+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.824320+0000","objects_scrubbed":6,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:33.203418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048924+0000","last_change":"2024-03-26T04:13:19.156112+0000","last_active":"2024-03-26T04:13:45.048924+0000","last_peered":"2024-03-26T04:13:45.048924+0000","last_clean":"2024-03-26T04:13:45.048924+0000","last_became_active":"2024-03-26T04:12:08.683874+0000","last_became_peered":"2024-03-26T04:12:08.683874+0000","last_unstale":"2024-03-26T04:13:45.048924+0000","last_undegraded":"2024-03-26T04:13:45.048924+0000","last_fullsized":"2024-03-26T04:13:45.048924+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:19.155998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.155998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:26.990024+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":104,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734119+0000","last_change":"2024-03-26T04:13:00.108543+0000","last_active":"2024-03-26T04:13:44.734119+0000","last_peered":"2024-03-26T04:13:44.734119+0000","last_clean":"2024-03-26T04:13:44.734119+0000","last_became_active":"2024-03-26T04:12:15.661728+0000","last_became_peered":"2024-03-26T04:12:15.661728+0000","last_unstale":"2024-03-26T04:13:44.734119+0000","last_undegraded":"2024-03-26T04:13:44.734119+0000","last_fullsized":"2024-03-26T04:13:44.734119+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:00.108436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:00.108436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.300626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1129,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665056+0000","last_change":"2024-03-26T04:13:34.064391+0000","last_active":"2024-03-26T04:13:44.665056+0000","last_peered":"2024-03-26T04:13:44.665056+0000","last_clean":"2024-03-26T04:13:44.665056+0000","last_became_active":"2024-03-26T04:12:11.896146+0000","last_became_peered":"2024-03-26T04:12:11.896146+0000","last_unstale":"2024-03-26T04:13:44.665056+0000","last_undegraded":"2024-03-26T04:13:44.665056+0000","last_fullsized":"2024-03-26T04:13:44.665056+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:34.064307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:34.064307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:34.346295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"448'14","reported_seq":93,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200277+0000","last_change":"2024-03-26T04:13:40.424554+0000","last_active":"2024-03-26T04:13:45.200277+0000","last_peered":"2024-03-26T04:13:45.200277+0000","last_clean":"2024-03-26T04:13:45.200277+0000","last_became_active":"2024-03-26T04:12:27.733789+0000","last_became_peered":"2024-03-26T04:12:27.733789+0000","last_unstale":"2024-03-26T04:13:45.200277+0000","last_undegraded":"2024-03-26T04:13:45.200277+0000","last_fullsized":"2024-03-26T04:13:45.200277+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'14","last_scrub_stamp":"2024-03-26T04:13:40.424448+0000","last_deep_scrub":"448'14","last_deep_scrub_stamp":"2024-03-26T04:12:24.786427+0000","last_clean_scrub_stamp":"2024-03-26T04:13:40.424448+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.543859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":125,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665184+0000","last_change":"2024-03-26T04:12:27.977716+0000","last_active":"2024-03-26T04:13:44.665184+0000","last_peered":"2024-03-26T04:13:44.665184+0000","last_clean":"2024-03-26T04:13:44.665184+0000","last_became_active":"2024-03-26T04:12:21.854036+0000","last_became_peered":"2024-03-26T04:12:21.854036+0000","last_unstale":"2024-03-26T04:13:44.665184+0000","last_undegraded":"2024-03-26T04:13:44.665184+0000","last_fullsized":"2024-03-26T04:13:44.665184+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:27.977610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:12:27.977610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:56.604807+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049323+0000","last_change":"2024-03-26T04:12:40.947206+0000","last_active":"2024-03-26T04:13:45.049323+0000","last_peered":"2024-03-26T04:13:45.049323+0000","last_clean":"2024-03-26T04:13:45.049323+0000","last_became_active":"2024-03-26T04:12:15.659773+0000","last_became_peered":"2024-03-26T04:12:15.659773+0000","last_unstale":"2024-03-26T04:13:45.049323+0000","last_undegraded":"2024-03-26T04:13:45.049323+0000","last_fullsized":"2024-03-26T04:13:45.049323+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:40.947102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:40.947102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:50.092363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200939+0000","last_change":"2024-03-26T04:13:15.570101+0000","last_active":"2024-03-26T04:13:45.200939+0000","last_peered":"2024-03-26T04:13:45.200939+0000","last_clean":"2024-03-26T04:13:45.200939+0000","last_became_active":"2024-03-26T04:12:29.653443+0000","last_became_peered":"2024-03-26T04:12:29.653443+0000","last_unstale":"2024-03-26T04:13:45.200939+0000","last_undegraded":"2024-03-26T04:13:45.200939+0000","last_fullsized":"2024-03-26T04:13:45.200939+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:15.570014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:15.570014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.944021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"448'39","reported_seq":155,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.808701+0000","last_change":"2024-03-26T04:13:49.808763+0000","last_active":"2024-03-26T04:13:49.808701+0000","last_peered":"2024-03-26T04:13:49.808701+0000","last_clean":"2024-03-26T04:13:49.808701+0000","last_became_active":"2024-03-26T04:12:15.651591+0000","last_became_peered":"2024-03-26T04:12:15.651591+0000","last_unstale":"2024-03-26T04:13:49.808701+0000","last_undegraded":"2024-03-26T04:13:49.808701+0000","last_fullsized":"2024-03-26T04:13:49.808701+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:49.808602+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.808602+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.344595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1090,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664024+0000","last_change":"2024-03-26T04:12:45.797882+0000","last_active":"2024-03-26T04:13:44.664024+0000","last_peered":"2024-03-26T04:13:44.664024+0000","last_clean":"2024-03-26T04:13:44.664024+0000","last_became_active":"2024-03-26T04:12:03.854120+0000","last_became_peered":"2024-03-26T04:12:03.854120+0000","last_unstale":"2024-03-26T04:13:44.664024+0000","last_undegraded":"2024-03-26T04:13:44.664024+0000","last_fullsized":"2024-03-26T04:13:44.664024+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:45.797787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:45.797787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:10.096962+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665105+0000","last_change":"2024-03-26T04:13:38.656051+0000","last_active":"2024-03-26T04:13:44.665105+0000","last_peered":"2024-03-26T04:13:44.665105+0000","last_clean":"2024-03-26T04:13:44.665105+0000","last_became_active":"2024-03-26T04:12:11.891221+0000","last_became_peered":"2024-03-26T04:12:11.891221+0000","last_unstale":"2024-03-26T04:13:44.665105+0000","last_undegraded":"2024-03-26T04:13:44.665105+0000","last_fullsized":"2024-03-26T04:13:44.665105+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:38.655950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:38.655950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.096084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050175+0000","last_change":"2024-03-26T04:12:33.983181+0000","last_active":"2024-03-26T04:13:45.050175+0000","last_peered":"2024-03-26T04:13:45.050175+0000","last_clean":"2024-03-26T04:13:45.050175+0000","last_became_active":"2024-03-26T04:12:11.887377+0000","last_became_peered":"2024-03-26T04:12:11.887377+0000","last_unstale":"2024-03-26T04:13:45.050175+0000","last_undegraded":"2024-03-26T04:13:45.050175+0000","last_fullsized":"2024-03-26T04:13:45.050175+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:33.983081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:33.983081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"448'25","reported_seq":93,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.445861+0000","last_change":"2024-03-26T04:13:46.445909+0000","last_active":"2024-03-26T04:13:46.445861+0000","last_peered":"2024-03-26T04:13:46.445861+0000","last_clean":"2024-03-26T04:13:46.445861+0000","last_became_active":"2024-03-26T04:12:29.654852+0000","last_became_peered":"2024-03-26T04:12:29.654852+0000","last_unstale":"2024-03-26T04:13:46.445861+0000","last_undegraded":"2024-03-26T04:13:46.445861+0000","last_fullsized":"2024-03-26T04:13:46.445861+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'25","last_scrub_stamp":"2024-03-26T04:13:46.445788+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.445788+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.114454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664294+0000","last_change":"2024-03-26T04:12:41.955305+0000","last_active":"2024-03-26T04:13:44.664294+0000","last_peered":"2024-03-26T04:13:44.664294+0000","last_clean":"2024-03-26T04:13:44.664294+0000","last_became_active":"2024-03-26T04:12:11.890613+0000","last_became_peered":"2024-03-26T04:12:11.890613+0000","last_unstale":"2024-03-26T04:13:44.664294+0000","last_undegraded":"2024-03-26T04:13:44.664294+0000","last_fullsized":"2024-03-26T04:13:44.664294+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:41.955200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:12:41.955200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:58.969597+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1121,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725767+0000","last_change":"2024-03-26T04:12:56.335382+0000","last_active":"2024-03-26T04:13:44.725767+0000","last_peered":"2024-03-26T04:13:44.725767+0000","last_clean":"2024-03-26T04:13:44.725767+0000","last_became_active":"2024-03-26T04:12:11.892084+0000","last_became_peered":"2024-03-26T04:12:11.892084+0000","last_unstale":"2024-03-26T04:13:44.725767+0000","last_undegraded":"2024-03-26T04:13:44.725767+0000","last_fullsized":"2024-03-26T04:13:44.725767+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.335288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:56.335288+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.335288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:11.604175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1108,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664027+0000","last_change":"2024-03-26T04:13:18.053542+0000","last_active":"2024-03-26T04:13:44.664027+0000","last_peered":"2024-03-26T04:13:44.664027+0000","last_clean":"2024-03-26T04:13:44.664027+0000","last_became_active":"2024-03-26T04:12:15.670251+0000","last_became_peered":"2024-03-26T04:12:15.670251+0000","last_unstale":"2024-03-26T04:13:44.664027+0000","last_undegraded":"2024-03-26T04:13:44.664027+0000","last_fullsized":"2024-03-26T04:13:44.664027+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.053432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.053432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:29.722406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"448'27","reported_seq":1196,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663770+0000","last_change":"2024-03-26T04:13:20.055428+0000","last_active":"2024-03-26T04:13:44.663770+0000","last_peered":"2024-03-26T04:13:44.663770+0000","last_clean":"2024-03-26T04:13:44.663770+0000","last_became_active":"2024-03-26T04:12:15.938622+0000","last_became_peered":"2024-03-26T04:12:15.938622+0000","last_unstale":"2024-03-26T04:13:44.663770+0000","last_undegraded":"2024-03-26T04:13:44.663770+0000","last_fullsized":"2024-03-26T04:13:44.663770+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'27","last_scrub_stamp":"2024-03-26T04:13:20.055317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:20.055317+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:46.920139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665362+0000","last_change":"2024-03-26T04:13:31.111428+0000","last_active":"2024-03-26T04:13:44.665362+0000","last_peered":"2024-03-26T04:13:44.665362+0000","last_clean":"2024-03-26T04:13:44.665362+0000","last_became_active":"2024-03-26T04:12:21.682113+0000","last_became_peered":"2024-03-26T04:12:21.682113+0000","last_unstale":"2024-03-26T04:13:44.665362+0000","last_undegraded":"2024-03-26T04:13:44.665362+0000","last_fullsized":"2024-03-26T04:13:44.665362+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:31.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:13:31.111320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.860528+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"548'14","reported_seq":158,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394775+0000","last_change":"2024-03-26T04:13:08.596778+0000","last_active":"2024-03-26T04:13:54.394775+0000","last_peered":"2024-03-26T04:13:54.394775+0000","last_clean":"2024-03-26T04:13:54.394775+0000","last_became_active":"2024-03-26T04:12:03.839909+0000","last_became_peered":"2024-03-26T04:12:03.839909+0000","last_unstale":"2024-03-26T04:13:54.394775+0000","last_undegraded":"2024-03-26T04:13:54.394775+0000","last_fullsized":"2024-03-26T04:13:54.394775+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'14","last_scrub_stamp":"2024-03-26T04:13:08.596685+0000","last_deep_scrub":"548'14","last_deep_scrub_stamp":"2024-03-26T04:13:08.596685+0000","last_clean_scrub_stamp":"2024-03-26T04:13:08.596685+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.264648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048748+0000","last_change":"2024-03-26T04:13:43.051465+0000","last_active":"2024-03-26T04:13:45.048748+0000","last_peered":"2024-03-26T04:13:45.048748+0000","last_clean":"2024-03-26T04:13:45.048748+0000","last_became_active":"2024-03-26T04:12:08.685754+0000","last_became_peered":"2024-03-26T04:12:08.685754+0000","last_unstale":"2024-03-26T04:13:45.048748+0000","last_undegraded":"2024-03-26T04:13:45.048748+0000","last_fullsized":"2024-03-26T04:13:45.048748+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:43.051352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:43.051352+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.051352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.934527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"448'49","reported_seq":151,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666118+0000","last_change":"2024-03-26T04:13:29.690717+0000","last_active":"2024-03-26T04:13:44.666118+0000","last_peered":"2024-03-26T04:13:44.666118+0000","last_clean":"2024-03-26T04:13:44.666118+0000","last_became_active":"2024-03-26T04:12:29.658202+0000","last_became_peered":"2024-03-26T04:12:29.658202+0000","last_unstale":"2024-03-26T04:13:44.666118+0000","last_undegraded":"2024-03-26T04:13:44.666118+0000","last_fullsized":"2024-03-26T04:13:44.666118+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'49","last_scrub_stamp":"2024-03-26T04:13:29.690591+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:13:29.690591+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.971046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":53,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1085,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664783+0000","last_change":"2024-03-26T04:13:04.226284+0000","last_active":"2024-03-26T04:13:44.664783+0000","last_peered":"2024-03-26T04:13:44.664783+0000","last_clean":"2024-03-26T04:13:44.664783+0000","last_became_active":"2024-03-26T04:12:03.856175+0000","last_became_peered":"2024-03-26T04:12:03.856175+0000","last_unstale":"2024-03-26T04:13:44.664783+0000","last_undegraded":"2024-03-26T04:13:44.664783+0000","last_fullsized":"2024-03-26T04:13:44.664783+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.226167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.226167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:09.764223+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"520'17","reported_seq":147,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:00.865896+0000","last_change":"2024-03-26T04:14:00.865922+0000","last_active":"2024-03-26T04:14:00.865896+0000","last_peered":"2024-03-26T04:14:00.865896+0000","last_clean":"2024-03-26T04:14:00.865896+0000","last_became_active":"2024-03-26T04:12:11.900998+0000","last_became_peered":"2024-03-26T04:12:11.900998+0000","last_unstale":"2024-03-26T04:14:00.865896+0000","last_undegraded":"2024-03-26T04:14:00.865896+0000","last_fullsized":"2024-03-26T04:14:00.865896+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'17","last_scrub_stamp":"2024-03-26T04:14:00.865833+0000","last_deep_scrub":"520'17","last_deep_scrub_stamp":"2024-03-26T04:12:46.959732+0000","last_clean_scrub_stamp":"2024-03-26T04:14:00.865833+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:20.982097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":1185,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663314+0000","last_change":"2024-03-26T04:12:56.711797+0000","last_active":"2024-03-26T04:13:44.663314+0000","last_peered":"2024-03-26T04:13:44.663314+0000","last_clean":"2024-03-26T04:13:44.663314+0000","last_became_active":"2024-03-26T04:12:11.896040+0000","last_became_peered":"2024-03-26T04:12:11.896040+0000","last_unstale":"2024-03-26T04:13:44.663314+0000","last_undegraded":"2024-03-26T04:13:44.663314+0000","last_fullsized":"2024-03-26T04:13:44.663314+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:12:56.711651+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.711651+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.974213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":10776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"448'32","reported_seq":185,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.433171+0000","last_change":"2024-03-26T04:13:49.433220+0000","last_active":"2024-03-26T04:13:49.433171+0000","last_peered":"2024-03-26T04:13:49.433171+0000","last_clean":"2024-03-26T04:13:49.433171+0000","last_became_active":"2024-03-26T04:12:11.890563+0000","last_became_peered":"2024-03-26T04:12:11.890563+0000","last_unstale":"2024-03-26T04:13:49.433171+0000","last_undegraded":"2024-03-26T04:13:49.433171+0000","last_fullsized":"2024-03-26T04:13:49.433171+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'32","last_scrub_stamp":"2024-03-26T04:13:49.433092+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.433092+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.996306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":1039,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.897879+0000","last_change":"2024-03-26T04:13:21.739781+0000","last_active":"2024-03-26T04:13:44.897879+0000","last_peered":"2024-03-26T04:13:44.897879+0000","last_clean":"2024-03-26T04:13:44.897879+0000","last_became_active":"2024-03-26T04:12:08.696475+0000","last_became_peered":"2024-03-26T04:12:08.696475+0000","last_unstale":"2024-03-26T04:13:44.897879+0000","last_undegraded":"2024-03-26T04:13:44.897879+0000","last_fullsized":"2024-03-26T04:13:44.897879+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:13:21.739679+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:13:21.739679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:32.139054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664741+0000","last_change":"2024-03-26T04:13:18.779649+0000","last_active":"2024-03-26T04:13:44.664741+0000","last_peered":"2024-03-26T04:13:44.664741+0000","last_clean":"2024-03-26T04:13:44.664741+0000","last_became_active":"2024-03-26T04:12:11.895264+0000","last_became_peered":"2024-03-26T04:12:11.895264+0000","last_unstale":"2024-03-26T04:13:44.664741+0000","last_undegraded":"2024-03-26T04:13:44.664741+0000","last_fullsized":"2024-03-26T04:13:44.664741+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.779519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.779519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:46.985322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":745,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049538+0000","last_change":"2024-03-26T04:13:41.037320+0000","last_active":"2024-03-26T04:13:45.049538+0000","last_peered":"2024-03-26T04:13:45.049538+0000","last_clean":"2024-03-26T04:13:45.049538+0000","last_became_active":"2024-03-26T04:13:06.665935+0000","last_became_peered":"2024-03-26T04:13:06.665935+0000","last_unstale":"2024-03-26T04:13:45.049538+0000","last_undegraded":"2024-03-26T04:13:45.049538+0000","last_fullsized":"2024-03-26T04:13:45.049538+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:41.037177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.037177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.548033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"448'39","reported_seq":1304,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:51.106407+0000","last_change":"2024-03-26T04:13:51.106445+0000","last_active":"2024-03-26T04:13:51.106407+0000","last_peered":"2024-03-26T04:13:51.106407+0000","last_clean":"2024-03-26T04:13:51.106407+0000","last_became_active":"2024-03-26T04:12:11.896101+0000","last_became_peered":"2024-03-26T04:12:11.896101+0000","last_unstale":"2024-03-26T04:13:51.106407+0000","last_undegraded":"2024-03-26T04:13:51.106407+0000","last_fullsized":"2024-03-26T04:13:51.106407+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:51.106351+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:13:51.106351+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:20.310413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049984+0000","last_change":"2024-03-26T04:12:27.181211+0000","last_active":"2024-03-26T04:13:45.049984+0000","last_peered":"2024-03-26T04:13:45.049984+0000","last_clean":"2024-03-26T04:13:45.049984+0000","last_became_active":"2024-03-26T04:12:15.655426+0000","last_became_peered":"2024-03-26T04:12:15.655426+0000","last_unstale":"2024-03-26T04:13:45.049984+0000","last_undegraded":"2024-03-26T04:13:45.049984+0000","last_fullsized":"2024-03-26T04:13:45.049984+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:27.181104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:27.181104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:52.875486+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200866+0000","last_change":"2024-03-26T04:13:14.579516+0000","last_active":"2024-03-26T04:13:45.200866+0000","last_peered":"2024-03-26T04:13:45.200866+0000","last_clean":"2024-03-26T04:13:45.200866+0000","last_became_active":"2024-03-26T04:12:11.888430+0000","last_became_peered":"2024-03-26T04:12:11.888430+0000","last_unstale":"2024-03-26T04:13:45.200866+0000","last_undegraded":"2024-03-26T04:13:45.200866+0000","last_fullsized":"2024-03-26T04:13:45.200866+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:14.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:13:14.579417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:22.488941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":112,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.733684+0000","last_change":"2024-03-26T04:13:30.793991+0000","last_active":"2024-03-26T04:13:44.733684+0000","last_peered":"2024-03-26T04:13:44.733684+0000","last_clean":"2024-03-26T04:13:44.733684+0000","last_became_active":"2024-03-26T04:12:15.658406+0000","last_became_peered":"2024-03-26T04:12:15.658406+0000","last_unstale":"2024-03-26T04:13:44.733684+0000","last_undegraded":"2024-03-26T04:13:44.733684+0000","last_fullsized":"2024-03-26T04:13:44.733684+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:30.793880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.793880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.042956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"565'29","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:56.854282+0000","last_change":"2024-03-26T04:13:56.854325+0000","last_active":"2024-03-26T04:13:56.854282+0000","last_peered":"2024-03-26T04:13:56.854282+0000","last_clean":"2024-03-26T04:13:56.854282+0000","last_became_active":"2024-03-26T04:12:11.894373+0000","last_became_peered":"2024-03-26T04:12:11.894373+0000","last_unstale":"2024-03-26T04:13:56.854282+0000","last_undegraded":"2024-03-26T04:13:56.854282+0000","last_fullsized":"2024-03-26T04:13:56.854282+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'29","last_scrub_stamp":"2024-03-26T04:13:56.854206+0000","last_deep_scrub":"565'29","last_deep_scrub_stamp":"2024-03-26T04:13:56.854206+0000","last_clean_scrub_stamp":"2024-03-26T04:13:56.854206+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:12.464099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.585537+0000","last_change":"2024-03-26T04:13:58.585565+0000","last_active":"2024-03-26T04:13:58.585537+0000","last_peered":"2024-03-26T04:13:58.585537+0000","last_clean":"2024-03-26T04:13:58.585537+0000","last_became_active":"2024-03-26T04:12:15.667445+0000","last_became_peered":"2024-03-26T04:12:15.667445+0000","last_unstale":"2024-03-26T04:13:58.585537+0000","last_undegraded":"2024-03-26T04:13:58.585537+0000","last_fullsized":"2024-03-26T04:13:58.585537+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:58.585462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:39.714496+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.585462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:24.291714+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"446'4","reported_seq":132,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:51.104380+0000","last_change":"2024-03-26T04:13:51.104410+0000","last_active":"2024-03-26T04:13:51.104380+0000","last_peered":"2024-03-26T04:13:51.104380+0000","last_clean":"2024-03-26T04:13:51.104380+0000","last_became_active":"2024-03-26T04:12:08.682824+0000","last_became_peered":"2024-03-26T04:12:08.682824+0000","last_unstale":"2024-03-26T04:13:51.104380+0000","last_undegraded":"2024-03-26T04:13:51.104380+0000","last_fullsized":"2024-03-26T04:13:51.104380+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'4","last_scrub_stamp":"2024-03-26T04:13:51.104303+0000","last_deep_scrub":"446'4","last_deep_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_clean_scrub_stamp":"2024-03-26T04:13:51.104303+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:08.794984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"448'51","reported_seq":209,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:01.575584+0000","last_change":"2024-03-26T04:14:01.575628+0000","last_active":"2024-03-26T04:14:01.575584+0000","last_peered":"2024-03-26T04:14:01.575584+0000","last_clean":"2024-03-26T04:14:01.575584+0000","last_became_active":"2024-03-26T04:12:15.664505+0000","last_became_peered":"2024-03-26T04:12:15.664505+0000","last_unstale":"2024-03-26T04:14:01.575584+0000","last_undegraded":"2024-03-26T04:14:01.575584+0000","last_fullsized":"2024-03-26T04:14:01.575584+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'51","last_scrub_stamp":"2024-03-26T04:14:01.575511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:14:01.575511+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:17.469647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"563'17","reported_seq":174,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394746+0000","last_change":"2024-03-26T04:13:40.076237+0000","last_active":"2024-03-26T04:13:54.394746+0000","last_peered":"2024-03-26T04:13:54.394746+0000","last_clean":"2024-03-26T04:13:54.394746+0000","last_became_active":"2024-03-26T04:12:15.655610+0000","last_became_peered":"2024-03-26T04:12:15.655610+0000","last_unstale":"2024-03-26T04:13:54.394746+0000","last_undegraded":"2024-03-26T04:13:54.394746+0000","last_fullsized":"2024-03-26T04:13:54.394746+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'17","last_scrub_stamp":"2024-03-26T04:13:40.076131+0000","last_deep_scrub":"563'17","last_deep_scrub_stamp":"2024-03-26T04:13:40.076131+0000","last_clean_scrub_stamp":"2024-03-26T04:13:40.076131+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:49.540190+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":121,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:47.913732+0000","last_change":"2024-03-26T04:13:47.913757+0000","last_active":"2024-03-26T04:13:47.913732+0000","last_peered":"2024-03-26T04:13:47.913732+0000","last_clean":"2024-03-26T04:13:47.913732+0000","last_became_active":"2024-03-26T04:12:11.896186+0000","last_became_peered":"2024-03-26T04:12:11.896186+0000","last_unstale":"2024-03-26T04:13:47.913732+0000","last_undegraded":"2024-03-26T04:13:47.913732+0000","last_fullsized":"2024-03-26T04:13:47.913732+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:47.913656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:13:47.913656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:11.382588+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048824+0000","last_change":"2024-03-26T04:12:47.843902+0000","last_active":"2024-03-26T04:13:45.048824+0000","last_peered":"2024-03-26T04:13:45.048824+0000","last_clean":"2024-03-26T04:13:45.048824+0000","last_became_active":"2024-03-26T04:12:08.688119+0000","last_became_peered":"2024-03-26T04:12:08.688119+0000","last_unstale":"2024-03-26T04:13:45.048824+0000","last_undegraded":"2024-03-26T04:13:45.048824+0000","last_fullsized":"2024-03-26T04:13:45.048824+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:47.843527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_clean_scrub_stamp":"2024-03-26T04:12:47.843527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:16.558614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"561'43","reported_seq":1232,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663943+0000","last_change":"2024-03-26T04:13:31.066703+0000","last_active":"2024-03-26T04:13:44.663943+0000","last_peered":"2024-03-26T04:13:44.663943+0000","last_clean":"2024-03-26T04:13:44.663943+0000","last_became_active":"2024-03-26T04:12:11.894713+0000","last_became_peered":"2024-03-26T04:12:11.894713+0000","last_unstale":"2024-03-26T04:13:44.663943+0000","last_undegraded":"2024-03-26T04:13:44.663943+0000","last_fullsized":"2024-03-26T04:13:44.663943+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'43","last_scrub_stamp":"2024-03-26T04:13:31.066515+0000","last_deep_scrub":"561'43","last_deep_scrub_stamp":"2024-03-26T04:13:31.066515+0000","last_clean_scrub_stamp":"2024-03-26T04:13:31.066515+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:31.620870+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1001,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.897941+0000","last_change":"2024-03-26T04:13:35.920669+0000","last_active":"2024-03-26T04:13:44.897941+0000","last_peered":"2024-03-26T04:13:44.897941+0000","last_clean":"2024-03-26T04:13:44.897941+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:13:44.897941+0000","last_undegraded":"2024-03-26T04:13:44.897941+0000","last_fullsized":"2024-03-26T04:13:44.897941+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:35.920587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.920587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.727024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":915,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049144+0000","last_change":"2024-03-26T04:13:06.668482+0000","last_active":"2024-03-26T04:13:45.049144+0000","last_peered":"2024-03-26T04:13:45.049144+0000","last_clean":"2024-03-26T04:13:45.049144+0000","last_became_active":"2024-03-26T04:13:06.668137+0000","last_became_peered":"2024-03-26T04:13:06.668137+0000","last_unstale":"2024-03-26T04:13:45.049144+0000","last_undegraded":"2024-03-26T04:13:45.049144+0000","last_fullsized":"2024-03-26T04:13:45.049144+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.356757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.356757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.916940+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1020,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.872581+0000","last_change":"2024-03-26T04:13:46.872621+0000","last_active":"2024-03-26T04:13:46.872581+0000","last_peered":"2024-03-26T04:13:46.872581+0000","last_clean":"2024-03-26T04:13:46.872581+0000","last_became_active":"2024-03-26T04:12:03.866785+0000","last_became_peered":"2024-03-26T04:12:03.866785+0000","last_unstale":"2024-03-26T04:13:46.872581+0000","last_undegraded":"2024-03-26T04:13:46.872581+0000","last_fullsized":"2024-03-26T04:13:46.872581+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.872470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:46.872470+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.872470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:00.528375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":99,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050190+0000","last_change":"2024-03-26T04:13:09.037100+0000","last_active":"2024-03-26T04:13:45.050190+0000","last_peered":"2024-03-26T04:13:45.050190+0000","last_clean":"2024-03-26T04:13:45.050190+0000","last_became_active":"2024-03-26T04:13:06.666989+0000","last_became_peered":"2024-03-26T04:13:06.666989+0000","last_unstale":"2024-03-26T04:13:45.050190+0000","last_undegraded":"2024-03-26T04:13:45.050190+0000","last_fullsized":"2024-03-26T04:13:45.050190+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:09.036935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:09.036935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:20.043134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"448'67","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734119+0000","last_change":"2024-03-26T04:12:49.091599+0000","last_active":"2024-03-26T04:13:44.734119+0000","last_peered":"2024-03-26T04:13:44.734119+0000","last_clean":"2024-03-26T04:13:44.734119+0000","last_became_active":"2024-03-26T04:12:15.668519+0000","last_became_peered":"2024-03-26T04:12:15.668519+0000","last_unstale":"2024-03-26T04:13:44.734119+0000","last_undegraded":"2024-03-26T04:13:44.734119+0000","last_fullsized":"2024-03-26T04:13:44.734119+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'67","last_scrub_stamp":"2024-03-26T04:12:49.091464+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:12:49.091464+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.600520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1085,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.129345+0000","last_change":"2024-03-26T04:13:58.129374+0000","last_active":"2024-03-26T04:13:58.129345+0000","last_peered":"2024-03-26T04:13:58.129345+0000","last_clean":"2024-03-26T04:13:58.129345+0000","last_became_active":"2024-03-26T04:12:08.677824+0000","last_became_peered":"2024-03-26T04:12:08.677824+0000","last_unstale":"2024-03-26T04:13:58.129345+0000","last_undegraded":"2024-03-26T04:13:58.129345+0000","last_fullsized":"2024-03-26T04:13:58.129345+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:58.129266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.129266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:02.360285+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1016,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898324+0000","last_change":"2024-03-26T04:13:24.768873+0000","last_active":"2024-03-26T04:13:44.898324+0000","last_peered":"2024-03-26T04:13:44.898324+0000","last_clean":"2024-03-26T04:13:44.898324+0000","last_became_active":"2024-03-26T04:12:11.880288+0000","last_became_peered":"2024-03-26T04:12:11.880288+0000","last_unstale":"2024-03-26T04:13:44.898324+0000","last_undegraded":"2024-03-26T04:13:44.898324+0000","last_fullsized":"2024-03-26T04:13:44.898324+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.768785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.768785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:29.244873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":149,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200739+0000","last_change":"2024-03-26T04:13:26.515548+0000","last_active":"2024-03-26T04:13:45.200739+0000","last_peered":"2024-03-26T04:13:45.200739+0000","last_clean":"2024-03-26T04:13:45.200739+0000","last_became_active":"2024-03-26T04:12:15.663290+0000","last_became_peered":"2024-03-26T04:12:15.663290+0000","last_unstale":"2024-03-26T04:13:45.200739+0000","last_undegraded":"2024-03-26T04:13:45.200739+0000","last_fullsized":"2024-03-26T04:13:45.200739+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:26.515446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:13:26.515446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:45.969628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"448'27","reported_seq":1189,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665339+0000","last_change":"2024-03-26T04:13:28.213776+0000","last_active":"2024-03-26T04:13:44.665339+0000","last_peered":"2024-03-26T04:13:44.665339+0000","last_clean":"2024-03-26T04:13:44.665339+0000","last_became_active":"2024-03-26T04:13:28.207633+0000","last_became_peered":"2024-03-26T04:13:28.207633+0000","last_unstale":"2024-03-26T04:13:44.665339+0000","last_undegraded":"2024-03-26T04:13:44.665339+0000","last_fullsized":"2024-03-26T04:13:44.665339+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'27","last_scrub_stamp":"2024-03-26T04:13:25.942590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.942590+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.395509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":897,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200504+0000","last_change":"2024-03-26T04:13:30.506773+0000","last_active":"2024-03-26T04:13:45.200504+0000","last_peered":"2024-03-26T04:13:45.200504+0000","last_clean":"2024-03-26T04:13:45.200504+0000","last_became_active":"2024-03-26T04:12:06.659993+0000","last_became_peered":"2024-03-26T04:12:06.659993+0000","last_unstale":"2024-03-26T04:13:45.200504+0000","last_undegraded":"2024-03-26T04:13:45.200504+0000","last_fullsized":"2024-03-26T04:13:45.200504+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:30.506668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.506668+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.573206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666062+0000","last_change":"2024-03-26T04:13:13.868069+0000","last_active":"2024-03-26T04:13:44.666062+0000","last_peered":"2024-03-26T04:13:44.666062+0000","last_clean":"2024-03-26T04:13:44.666062+0000","last_became_active":"2024-03-26T04:12:11.894563+0000","last_became_peered":"2024-03-26T04:12:11.894563+0000","last_unstale":"2024-03-26T04:13:44.666062+0000","last_undegraded":"2024-03-26T04:13:44.666062+0000","last_fullsized":"2024-03-26T04:13:44.666062+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:13.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:13.867940+0000","last_clean_scrub_stamp":"2024-03-26T04:13:13.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:17.782005+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200447+0000","last_change":"2024-03-26T04:13:16.532509+0000","last_active":"2024-03-26T04:13:45.200447+0000","last_peered":"2024-03-26T04:13:45.200447+0000","last_clean":"2024-03-26T04:13:45.200447+0000","last_became_active":"2024-03-26T04:12:15.667426+0000","last_became_peered":"2024-03-26T04:12:15.667426+0000","last_unstale":"2024-03-26T04:13:45.200447+0000","last_undegraded":"2024-03-26T04:13:45.200447+0000","last_fullsized":"2024-03-26T04:13:45.200447+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:13:16.532403+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:13:16.532403+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.591110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1089,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664249+0000","last_change":"2024-03-26T04:13:43.062614+0000","last_active":"2024-03-26T04:13:44.664249+0000","last_peered":"2024-03-26T04:13:44.664249+0000","last_clean":"2024-03-26T04:13:44.664249+0000","last_became_active":"2024-03-26T04:12:15.673049+0000","last_became_peered":"2024-03-26T04:12:15.673049+0000","last_unstale":"2024-03-26T04:13:44.664249+0000","last_undegraded":"2024-03-26T04:13:44.664249+0000","last_fullsized":"2024-03-26T04:13:44.664249+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:43.062525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.062525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.018180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"520'67","reported_seq":1700,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394563+0000","last_change":"2024-03-26T04:13:39.448800+0000","last_active":"2024-03-26T04:13:54.394563+0000","last_peered":"2024-03-26T04:13:54.394563+0000","last_clean":"2024-03-26T04:13:54.394563+0000","last_became_active":"2024-03-26T04:12:11.893198+0000","last_became_peered":"2024-03-26T04:12:11.893198+0000","last_unstale":"2024-03-26T04:13:54.394563+0000","last_undegraded":"2024-03-26T04:13:54.394563+0000","last_fullsized":"2024-03-26T04:13:54.394563+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'67","last_scrub_stamp":"2024-03-26T04:13:39.448703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:39.448703+0000","objects_scrubbed":11,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:01.740182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":2506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664638+0000","last_change":"2024-03-26T04:13:27.898799+0000","last_active":"2024-03-26T04:13:44.664638+0000","last_peered":"2024-03-26T04:13:44.664638+0000","last_clean":"2024-03-26T04:13:44.664638+0000","last_became_active":"2024-03-26T04:12:21.684416+0000","last_became_peered":"2024-03-26T04:12:21.684416+0000","last_unstale":"2024-03-26T04:13:44.664638+0000","last_undegraded":"2024-03-26T04:13:44.664638+0000","last_fullsized":"2024-03-26T04:13:44.664638+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.898720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.898720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:52.356266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":864,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049656+0000","last_change":"2024-03-26T04:13:06.669980+0000","last_active":"2024-03-26T04:13:45.049656+0000","last_peered":"2024-03-26T04:13:45.049656+0000","last_clean":"2024-03-26T04:13:45.049656+0000","last_became_active":"2024-03-26T04:13:06.669607+0000","last_became_peered":"2024-03-26T04:13:06.669607+0000","last_unstale":"2024-03-26T04:13:45.049656+0000","last_undegraded":"2024-03-26T04:13:45.049656+0000","last_fullsized":"2024-03-26T04:13:45.049656+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:52.733697+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:12:52.733697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:13.030575+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050054+0000","last_change":"2024-03-26T04:13:35.146329+0000","last_active":"2024-03-26T04:13:45.050054+0000","last_peered":"2024-03-26T04:13:45.050054+0000","last_clean":"2024-03-26T04:13:45.050054+0000","last_became_active":"2024-03-26T04:12:15.654522+0000","last_became_peered":"2024-03-26T04:12:15.654522+0000","last_unstale":"2024-03-26T04:13:45.050054+0000","last_undegraded":"2024-03-26T04:13:45.050054+0000","last_fullsized":"2024-03-26T04:13:45.050054+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:35.146222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.146222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:47.967411+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1111,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.611205+0000","last_change":"2024-03-26T04:13:57.611240+0000","last_active":"2024-03-26T04:13:57.611205+0000","last_peered":"2024-03-26T04:13:57.611205+0000","last_clean":"2024-03-26T04:13:57.611205+0000","last_became_active":"2024-03-26T04:12:15.670120+0000","last_became_peered":"2024-03-26T04:12:15.670120+0000","last_unstale":"2024-03-26T04:13:57.611205+0000","last_undegraded":"2024-03-26T04:13:57.611205+0000","last_fullsized":"2024-03-26T04:13:57.611205+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.611137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.611137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:19.664946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1034,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898137+0000","last_change":"2024-03-26T04:13:42.909700+0000","last_active":"2024-03-26T04:13:44.898137+0000","last_peered":"2024-03-26T04:13:44.898137+0000","last_clean":"2024-03-26T04:13:44.898137+0000","last_became_active":"2024-03-26T04:12:11.879075+0000","last_became_peered":"2024-03-26T04:12:11.879075+0000","last_unstale":"2024-03-26T04:13:44.898137+0000","last_undegraded":"2024-03-26T04:13:44.898137+0000","last_fullsized":"2024-03-26T04:13:44.898137+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:42.909590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:25.140185+0000","last_clean_scrub_stamp":"2024-03-26T04:13:42.909590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:07.684065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"448'33","reported_seq":97,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:47.449302+0000","last_change":"2024-03-26T04:13:47.449340+0000","last_active":"2024-03-26T04:13:47.449302+0000","last_peered":"2024-03-26T04:13:47.449302+0000","last_clean":"2024-03-26T04:13:47.449302+0000","last_became_active":"2024-03-26T04:12:29.656239+0000","last_became_peered":"2024-03-26T04:12:29.656239+0000","last_unstale":"2024-03-26T04:13:47.449302+0000","last_undegraded":"2024-03-26T04:13:47.449302+0000","last_fullsized":"2024-03-26T04:13:47.449302+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'33","last_scrub_stamp":"2024-03-26T04:13:47.449223+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:13:47.449223+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:13.029397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1039,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.816449+0000","last_change":"2024-03-26T04:13:49.816472+0000","last_active":"2024-03-26T04:13:49.816449+0000","last_peered":"2024-03-26T04:13:49.816449+0000","last_clean":"2024-03-26T04:13:49.816449+0000","last_became_active":"2024-03-26T04:12:08.690699+0000","last_became_peered":"2024-03-26T04:12:08.690699+0000","last_unstale":"2024-03-26T04:13:49.816449+0000","last_undegraded":"2024-03-26T04:13:49.816449+0000","last_fullsized":"2024-03-26T04:13:49.816449+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:49.816391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.816391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.765706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201163+0000","last_change":"2024-03-26T04:13:04.711186+0000","last_active":"2024-03-26T04:13:45.201163+0000","last_peered":"2024-03-26T04:13:45.201163+0000","last_clean":"2024-03-26T04:13:45.201163+0000","last_became_active":"2024-03-26T04:12:11.888053+0000","last_became_peered":"2024-03-26T04:12:11.888053+0000","last_unstale":"2024-03-26T04:13:45.201163+0000","last_undegraded":"2024-03-26T04:13:45.201163+0000","last_fullsized":"2024-03-26T04:13:45.201163+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.711083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.711083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.655476+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":565,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049201+0000","last_change":"2024-03-26T04:13:28.086428+0000","last_active":"2024-03-26T04:13:45.049201+0000","last_peered":"2024-03-26T04:13:45.049201+0000","last_clean":"2024-03-26T04:13:45.049201+0000","last_became_active":"2024-03-26T04:12:18.665269+0000","last_became_peered":"2024-03-26T04:12:18.665269+0000","last_unstale":"2024-03-26T04:13:45.049201+0000","last_undegraded":"2024-03-26T04:13:45.049201+0000","last_fullsized":"2024-03-26T04:13:45.049201+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:28.086295+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:10:33.403777+0000","last_clean_scrub_stamp":"2024-03-26T04:13:28.086295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.249806+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":127,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.733685+0000","last_change":"2024-03-26T04:13:46.733706+0000","last_active":"2024-03-26T04:13:46.733685+0000","last_peered":"2024-03-26T04:13:46.733685+0000","last_clean":"2024-03-26T04:13:46.733685+0000","last_became_active":"2024-03-26T04:12:15.656802+0000","last_became_peered":"2024-03-26T04:12:15.656802+0000","last_unstale":"2024-03-26T04:13:46.733685+0000","last_undegraded":"2024-03-26T04:13:46.733685+0000","last_fullsized":"2024-03-26T04:13:46.733685+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.733623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.733623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.849645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"525'45","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394749+0000","last_change":"2024-03-26T04:13:04.100203+0000","last_active":"2024-03-26T04:13:54.394749+0000","last_peered":"2024-03-26T04:13:54.394749+0000","last_clean":"2024-03-26T04:13:54.394749+0000","last_became_active":"2024-03-26T04:12:15.665471+0000","last_became_peered":"2024-03-26T04:12:15.665471+0000","last_unstale":"2024-03-26T04:13:54.394749+0000","last_undegraded":"2024-03-26T04:13:54.394749+0000","last_fullsized":"2024-03-26T04:13:54.394749+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'45","last_scrub_stamp":"2024-03-26T04:13:04.100076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.100076+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.796896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":858,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049900+0000","last_change":"2024-03-26T04:13:23.013833+0000","last_active":"2024-03-26T04:13:45.049900+0000","last_peered":"2024-03-26T04:13:45.049900+0000","last_clean":"2024-03-26T04:13:45.049900+0000","last_became_active":"2024-03-26T04:13:06.669225+0000","last_became_peered":"2024-03-26T04:13:06.669225+0000","last_unstale":"2024-03-26T04:13:45.049900+0000","last_undegraded":"2024-03-26T04:13:45.049900+0000","last_fullsized":"2024-03-26T04:13:45.049900+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.013702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:22.075168+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.013702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.886857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":998,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:55.566341+0000","last_change":"2024-03-26T04:13:55.566379+0000","last_active":"2024-03-26T04:13:55.566341+0000","last_peered":"2024-03-26T04:13:55.566341+0000","last_clean":"2024-03-26T04:13:55.566341+0000","last_became_active":"2024-03-26T04:12:18.657768+0000","last_became_peered":"2024-03-26T04:12:18.657768+0000","last_unstale":"2024-03-26T04:13:55.566341+0000","last_undegraded":"2024-03-26T04:13:55.566341+0000","last_fullsized":"2024-03-26T04:13:55.566341+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:55.566263+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:12:39.440775+0000","last_clean_scrub_stamp":"2024-03-26T04:13:55.566263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:56.515493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":83,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:53.509387+0000","last_change":"2024-03-26T04:13:53.509408+0000","last_active":"2024-03-26T04:13:53.509387+0000","last_peered":"2024-03-26T04:13:53.509387+0000","last_clean":"2024-03-26T04:13:53.509387+0000","last_became_active":"2024-03-26T04:12:29.661693+0000","last_became_peered":"2024-03-26T04:12:29.661693+0000","last_unstale":"2024-03-26T04:13:53.509387+0000","last_undegraded":"2024-03-26T04:13:53.509387+0000","last_fullsized":"2024-03-26T04:13:53.509387+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:53.509326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:53.509326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.077901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663781+0000","last_change":"2024-03-26T04:13:41.153795+0000","last_active":"2024-03-26T04:13:44.663781+0000","last_peered":"2024-03-26T04:13:44.663781+0000","last_clean":"2024-03-26T04:13:44.663781+0000","last_became_active":"2024-03-26T04:12:11.895940+0000","last_became_peered":"2024-03-26T04:12:11.895940+0000","last_unstale":"2024-03-26T04:13:44.663781+0000","last_undegraded":"2024-03-26T04:13:44.663781+0000","last_fullsized":"2024-03-26T04:13:44.663781+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:41.153712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:41.153712+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.153712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.510402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"565'6","reported_seq":242,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.870105+0000","last_change":"2024-03-26T04:13:55.135310+0000","last_active":"2024-03-26T04:13:57.870105+0000","last_peered":"2024-03-26T04:13:57.870105+0000","last_clean":"2024-03-26T04:13:57.870105+0000","last_became_active":"2024-03-26T04:12:08.690101+0000","last_became_peered":"2024-03-26T04:12:08.690101+0000","last_unstale":"2024-03-26T04:13:57.870105+0000","last_undegraded":"2024-03-26T04:13:57.870105+0000","last_fullsized":"2024-03-26T04:13:57.870105+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'6","last_scrub_stamp":"2024-03-26T04:13:55.135201+0000","last_deep_scrub":"565'6","last_deep_scrub_stamp":"2024-03-26T04:13:55.135201+0000","last_clean_scrub_stamp":"2024-03-26T04:13:55.135201+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:15.797200+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"448'42","reported_seq":109,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.088052+0000","last_change":"2024-03-26T04:13:49.088102+0000","last_active":"2024-03-26T04:13:49.088052+0000","last_peered":"2024-03-26T04:13:49.088052+0000","last_clean":"2024-03-26T04:13:49.088052+0000","last_became_active":"2024-03-26T04:12:21.685659+0000","last_became_peered":"2024-03-26T04:12:21.685659+0000","last_unstale":"2024-03-26T04:13:49.088052+0000","last_undegraded":"2024-03-26T04:13:49.088052+0000","last_fullsized":"2024-03-26T04:13:49.088052+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'42","last_scrub_stamp":"2024-03-26T04:13:49.087962+0000","last_deep_scrub":"422'30","last_deep_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.087962+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:08.277379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":137,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201234+0000","last_change":"2024-03-26T04:13:22.598273+0000","last_active":"2024-03-26T04:13:45.201234+0000","last_peered":"2024-03-26T04:13:45.201234+0000","last_clean":"2024-03-26T04:13:45.201234+0000","last_became_active":"2024-03-26T04:12:03.846590+0000","last_became_peered":"2024-03-26T04:12:03.846590+0000","last_unstale":"2024-03-26T04:13:45.201234+0000","last_undegraded":"2024-03-26T04:13:45.201234+0000","last_fullsized":"2024-03-26T04:13:45.201234+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:13:22.598174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:37.397639+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":944,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734438+0000","last_change":"2024-03-26T04:13:41.829833+0000","last_active":"2024-03-26T04:13:44.734438+0000","last_peered":"2024-03-26T04:13:44.734438+0000","last_clean":"2024-03-26T04:13:44.734438+0000","last_became_active":"2024-03-26T04:12:18.661189+0000","last_became_peered":"2024-03-26T04:12:18.661189+0000","last_unstale":"2024-03-26T04:13:44.734438+0000","last_undegraded":"2024-03-26T04:13:44.734438+0000","last_fullsized":"2024-03-26T04:13:44.734438+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:41.829728+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:12:19.954807+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.829728+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.400447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1103,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725449+0000","last_change":"2024-03-26T04:13:25.547378+0000","last_active":"2024-03-26T04:13:44.725449+0000","last_peered":"2024-03-26T04:13:44.725449+0000","last_clean":"2024-03-26T04:13:44.725449+0000","last_became_active":"2024-03-26T04:12:15.675671+0000","last_became_peered":"2024-03-26T04:12:15.675671+0000","last_unstale":"2024-03-26T04:13:44.725449+0000","last_undegraded":"2024-03-26T04:13:44.725449+0000","last_fullsized":"2024-03-26T04:13:44.725449+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:25.547273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:25.547273+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.547273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:39.656810+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1112,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.726175+0000","last_change":"2024-03-26T04:13:30.590127+0000","last_active":"2024-03-26T04:13:44.726175+0000","last_peered":"2024-03-26T04:13:44.726175+0000","last_clean":"2024-03-26T04:13:44.726175+0000","last_became_active":"2024-03-26T04:12:03.859048+0000","last_became_peered":"2024-03-26T04:12:03.859048+0000","last_unstale":"2024-03-26T04:13:44.726175+0000","last_undegraded":"2024-03-26T04:13:44.726175+0000","last_fullsized":"2024-03-26T04:13:44.726175+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:30.590005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:30.590005+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.590005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:55.749006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"448'54","reported_seq":1307,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665546+0000","last_change":"2024-03-26T04:13:19.960173+0000","last_active":"2024-03-26T04:13:44.665546+0000","last_peered":"2024-03-26T04:13:44.665546+0000","last_clean":"2024-03-26T04:13:44.665546+0000","last_became_active":"2024-03-26T04:12:08.678078+0000","last_became_peered":"2024-03-26T04:12:08.678078+0000","last_unstale":"2024-03-26T04:13:44.665546+0000","last_undegraded":"2024-03-26T04:13:44.665546+0000","last_fullsized":"2024-03-26T04:13:44.665546+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'54","last_scrub_stamp":"2024-03-26T04:13:19.960073+0000","last_deep_scrub":"448'54","last_deep_scrub_stamp":"2024-03-26T04:11:45.215033+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.960073+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:23.105238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049530+0000","last_change":"2024-03-26T04:13:06.952445+0000","last_active":"2024-03-26T04:13:45.049530+0000","last_peered":"2024-03-26T04:13:45.049530+0000","last_clean":"2024-03-26T04:13:45.049530+0000","last_became_active":"2024-03-26T04:13:06.666213+0000","last_became_peered":"2024-03-26T04:13:06.666213+0000","last_unstale":"2024-03-26T04:13:45.049530+0000","last_undegraded":"2024-03-26T04:13:45.049530+0000","last_fullsized":"2024-03-26T04:13:45.049530+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:06.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_clean_scrub_stamp":"2024-03-26T04:13:06.952348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.925261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":779,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734270+0000","last_change":"2024-03-26T04:12:57.109436+0000","last_active":"2024-03-26T04:13:44.734270+0000","last_peered":"2024-03-26T04:13:44.734270+0000","last_clean":"2024-03-26T04:13:44.734270+0000","last_became_active":"2024-03-26T04:12:18.662592+0000","last_became_peered":"2024-03-26T04:12:18.662592+0000","last_unstale":"2024-03-26T04:13:44.734270+0000","last_undegraded":"2024-03-26T04:13:44.734270+0000","last_fullsized":"2024-03-26T04:13:44.734270+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:57.109323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.109323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.603411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1120,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664118+0000","last_change":"2024-03-26T04:13:36.159194+0000","last_active":"2024-03-26T04:13:44.664118+0000","last_peered":"2024-03-26T04:13:44.664118+0000","last_clean":"2024-03-26T04:13:44.664118+0000","last_became_active":"2024-03-26T04:12:11.903366+0000","last_became_peered":"2024-03-26T04:12:11.903366+0000","last_unstale":"2024-03-26T04:13:44.664118+0000","last_undegraded":"2024-03-26T04:13:44.664118+0000","last_fullsized":"2024-03-26T04:13:44.664118+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:36.159096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:36.159096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.574402+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201209+0000","last_change":"2024-03-26T04:12:56.581058+0000","last_active":"2024-03-26T04:13:45.201209+0000","last_peered":"2024-03-26T04:13:45.201209+0000","last_clean":"2024-03-26T04:13:45.201209+0000","last_became_active":"2024-03-26T04:12:11.888081+0000","last_became_peered":"2024-03-26T04:12:11.888081+0000","last_unstale":"2024-03-26T04:13:45.201209+0000","last_undegraded":"2024-03-26T04:13:45.201209+0000","last_fullsized":"2024-03-26T04:13:45.201209+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.580954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.580954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.537483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"448'36","reported_seq":105,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:01.684796+0000","last_change":"2024-03-26T04:14:01.684834+0000","last_active":"2024-03-26T04:14:01.684796+0000","last_peered":"2024-03-26T04:14:01.684796+0000","last_clean":"2024-03-26T04:14:01.684796+0000","last_became_active":"2024-03-26T04:12:29.656147+0000","last_became_peered":"2024-03-26T04:12:29.656147+0000","last_unstale":"2024-03-26T04:14:01.684796+0000","last_undegraded":"2024-03-26T04:14:01.684796+0000","last_fullsized":"2024-03-26T04:14:01.684796+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'36","last_scrub_stamp":"2024-03-26T04:14:01.684717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:14:01.684717+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:26.607349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"525'21","reported_seq":1505,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394483+0000","last_change":"2024-03-26T04:13:53.036323+0000","last_active":"2024-03-26T04:13:54.394483+0000","last_peered":"2024-03-26T04:13:54.394483+0000","last_clean":"2024-03-26T04:13:54.394483+0000","last_became_active":"2024-03-26T04:12:15.654133+0000","last_became_peered":"2024-03-26T04:12:15.654133+0000","last_unstale":"2024-03-26T04:13:54.394483+0000","last_undegraded":"2024-03-26T04:13:54.394483+0000","last_fullsized":"2024-03-26T04:13:54.394483+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'21","last_scrub_stamp":"2024-03-26T04:13:53.036229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:53.036229+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.858637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":594,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.845962+0000","last_change":"2024-03-26T04:13:54.845990+0000","last_active":"2024-03-26T04:13:54.845962+0000","last_peered":"2024-03-26T04:13:54.845962+0000","last_clean":"2024-03-26T04:13:54.845962+0000","last_became_active":"2024-03-26T04:12:14.663450+0000","last_became_peered":"2024-03-26T04:12:14.663450+0000","last_unstale":"2024-03-26T04:13:54.845962+0000","last_undegraded":"2024-03-26T04:13:54.845962+0000","last_fullsized":"2024-03-26T04:13:54.845962+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:54.845896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:13:54.845896+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.640626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049674+0000","last_change":"2024-03-26T04:13:04.089423+0000","last_active":"2024-03-26T04:13:45.049674+0000","last_peered":"2024-03-26T04:13:45.049674+0000","last_clean":"2024-03-26T04:13:45.049674+0000","last_became_active":"2024-03-26T04:12:15.654149+0000","last_became_peered":"2024-03-26T04:12:15.654149+0000","last_unstale":"2024-03-26T04:13:45.049674+0000","last_undegraded":"2024-03-26T04:13:45.049674+0000","last_fullsized":"2024-03-26T04:13:45.049674+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.089322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:04.089322+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.089322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.627171+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":1086,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665965+0000","last_change":"2024-03-26T04:13:28.942893+0000","last_active":"2024-03-26T04:13:44.665965+0000","last_peered":"2024-03-26T04:13:44.665965+0000","last_clean":"2024-03-26T04:13:44.665965+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:13:44.665965+0000","last_undegraded":"2024-03-26T04:13:44.665965+0000","last_fullsized":"2024-03-26T04:13:44.665965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:13:28.942794+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:13:28.942794+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.356439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"448'48","reported_seq":1277,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663956+0000","last_change":"2024-03-26T04:13:29.003162+0000","last_active":"2024-03-26T04:13:44.663956+0000","last_peered":"2024-03-26T04:13:44.663956+0000","last_clean":"2024-03-26T04:13:44.663956+0000","last_became_active":"2024-03-26T04:12:15.668826+0000","last_became_peered":"2024-03-26T04:12:15.668826+0000","last_unstale":"2024-03-26T04:13:44.663956+0000","last_undegraded":"2024-03-26T04:13:44.663956+0000","last_fullsized":"2024-03-26T04:13:44.663956+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'48","last_scrub_stamp":"2024-03-26T04:13:29.003026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:29.003026+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.528033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"520'14","reported_seq":139,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394459+0000","last_change":"2024-03-26T04:13:06.899411+0000","last_active":"2024-03-26T04:13:54.394459+0000","last_peered":"2024-03-26T04:13:54.394459+0000","last_clean":"2024-03-26T04:13:54.394459+0000","last_became_active":"2024-03-26T04:12:11.900417+0000","last_became_peered":"2024-03-26T04:12:11.900417+0000","last_unstale":"2024-03-26T04:13:54.394459+0000","last_undegraded":"2024-03-26T04:13:54.394459+0000","last_fullsized":"2024-03-26T04:13:54.394459+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'14","last_scrub_stamp":"2024-03-26T04:13:06.899326+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:13:06.899326+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:08.668844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":942,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664742+0000","last_change":"2024-03-26T04:12:54.914516+0000","last_active":"2024-03-26T04:13:44.664742+0000","last_peered":"2024-03-26T04:13:44.664742+0000","last_clean":"2024-03-26T04:13:44.664742+0000","last_became_active":"2024-03-26T04:12:18.661143+0000","last_became_peered":"2024-03-26T04:12:18.661143+0000","last_unstale":"2024-03-26T04:13:44.664742+0000","last_undegraded":"2024-03-26T04:13:44.664742+0000","last_fullsized":"2024-03-26T04:13:44.664742+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:54.914405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:54.914405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:01.121205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1100,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664522+0000","last_change":"2024-03-26T04:12:31.998898+0000","last_active":"2024-03-26T04:13:44.664522+0000","last_peered":"2024-03-26T04:13:44.664522+0000","last_clean":"2024-03-26T04:13:44.664522+0000","last_became_active":"2024-03-26T04:12:11.882387+0000","last_became_peered":"2024-03-26T04:12:11.882387+0000","last_unstale":"2024-03-26T04:13:44.664522+0000","last_undegraded":"2024-03-26T04:13:44.664522+0000","last_fullsized":"2024-03-26T04:13:44.664522+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:31.998779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:31.998779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:54.836641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665007+0000","last_change":"2024-03-26T04:12:58.920908+0000","last_active":"2024-03-26T04:13:44.665007+0000","last_peered":"2024-03-26T04:13:44.665007+0000","last_clean":"2024-03-26T04:13:44.665007+0000","last_became_active":"2024-03-26T04:12:11.899519+0000","last_became_peered":"2024-03-26T04:12:11.899519+0000","last_unstale":"2024-03-26T04:13:44.665007+0000","last_undegraded":"2024-03-26T04:13:44.665007+0000","last_fullsized":"2024-03-26T04:13:44.665007+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:58.920789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:58.920789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:13.165271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"448'64","reported_seq":171,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664095+0000","last_change":"2024-03-26T04:13:37.607925+0000","last_active":"2024-03-26T04:13:44.664095+0000","last_peered":"2024-03-26T04:13:44.664095+0000","last_clean":"2024-03-26T04:13:44.664095+0000","last_became_active":"2024-03-26T04:12:21.683241+0000","last_became_peered":"2024-03-26T04:12:21.683241+0000","last_unstale":"2024-03-26T04:13:44.664095+0000","last_undegraded":"2024-03-26T04:13:44.664095+0000","last_fullsized":"2024-03-26T04:13:44.664095+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'64","last_scrub_stamp":"2024-03-26T04:13:37.607799+0000","last_deep_scrub":"448'64","last_deep_scrub_stamp":"2024-03-26T04:12:22.799227+0000","last_clean_scrub_stamp":"2024-03-26T04:13:37.607799+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.297848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1115,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663701+0000","last_change":"2024-03-26T04:13:24.919940+0000","last_active":"2024-03-26T04:13:44.663701+0000","last_peered":"2024-03-26T04:13:44.663701+0000","last_clean":"2024-03-26T04:13:44.663701+0000","last_became_active":"2024-03-26T04:12:11.881608+0000","last_became_peered":"2024-03-26T04:12:11.881608+0000","last_unstale":"2024-03-26T04:13:44.663701+0000","last_undegraded":"2024-03-26T04:13:44.663701+0000","last_fullsized":"2024-03-26T04:13:44.663701+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.919819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.919819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.479562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":433,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200893+0000","last_change":"2024-03-26T04:13:17.559995+0000","last_active":"2024-03-26T04:13:45.200893+0000","last_peered":"2024-03-26T04:13:45.200893+0000","last_clean":"2024-03-26T04:13:45.200893+0000","last_became_active":"2024-03-26T04:12:18.659657+0000","last_became_peered":"2024-03-26T04:12:18.659657+0000","last_unstale":"2024-03-26T04:13:45.200893+0000","last_undegraded":"2024-03-26T04:13:45.200893+0000","last_fullsized":"2024-03-26T04:13:45.200893+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:17.559860+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:13:17.559860+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:45.570707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":153,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.665830+0000","last_change":"2024-03-26T04:13:57.665880+0000","last_active":"2024-03-26T04:13:57.665830+0000","last_peered":"2024-03-26T04:13:57.665830+0000","last_clean":"2024-03-26T04:13:57.665830+0000","last_became_active":"2024-03-26T04:12:29.656864+0000","last_became_peered":"2024-03-26T04:12:29.656864+0000","last_unstale":"2024-03-26T04:13:57.665830+0000","last_undegraded":"2024-03-26T04:13:57.665830+0000","last_fullsized":"2024-03-26T04:13:57.665830+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.665730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.665730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:13.842790+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"510'14","reported_seq":1415,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394743+0000","last_change":"2024-03-26T04:12:57.939664+0000","last_active":"2024-03-26T04:13:54.394743+0000","last_peered":"2024-03-26T04:13:54.394743+0000","last_clean":"2024-03-26T04:13:54.394743+0000","last_became_active":"2024-03-26T04:12:03.838321+0000","last_became_peered":"2024-03-26T04:12:03.838321+0000","last_unstale":"2024-03-26T04:13:54.394743+0000","last_undegraded":"2024-03-26T04:13:54.394743+0000","last_fullsized":"2024-03-26T04:13:54.394743+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'14","last_scrub_stamp":"2024-03-26T04:12:57.939567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.939567+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.031183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664359+0000","last_change":"2024-03-26T04:12:49.876477+0000","last_active":"2024-03-26T04:13:44.664359+0000","last_peered":"2024-03-26T04:13:44.664359+0000","last_clean":"2024-03-26T04:13:44.664359+0000","last_became_active":"2024-03-26T04:12:11.902422+0000","last_became_peered":"2024-03-26T04:12:11.902422+0000","last_unstale":"2024-03-26T04:13:44.664359+0000","last_undegraded":"2024-03-26T04:13:44.664359+0000","last_fullsized":"2024-03-26T04:13:44.664359+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:49.876374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:12:49.876374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.259627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"448'39","reported_seq":1300,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.610278+0000","last_change":"2024-03-26T04:13:58.610330+0000","last_active":"2024-03-26T04:13:58.610278+0000","last_peered":"2024-03-26T04:13:58.610278+0000","last_clean":"2024-03-26T04:13:58.610278+0000","last_became_active":"2024-03-26T04:12:11.892439+0000","last_became_peered":"2024-03-26T04:12:11.892439+0000","last_unstale":"2024-03-26T04:13:58.610278+0000","last_undegraded":"2024-03-26T04:13:58.610278+0000","last_fullsized":"2024-03-26T04:13:58.610278+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:58.610208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.610208+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:17.322978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666170+0000","last_change":"2024-03-26T04:12:42.869017+0000","last_active":"2024-03-26T04:13:44.666170+0000","last_peered":"2024-03-26T04:13:44.666170+0000","last_clean":"2024-03-26T04:13:44.666170+0000","last_became_active":"2024-03-26T04:12:15.655609+0000","last_became_peered":"2024-03-26T04:12:15.655609+0000","last_unstale":"2024-03-26T04:13:44.666170+0000","last_undegraded":"2024-03-26T04:13:44.666170+0000","last_fullsized":"2024-03-26T04:13:44.666170+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:42.868898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_clean_scrub_stamp":"2024-03-26T04:12:42.868898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:51.336079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201041+0000","last_change":"2024-03-26T04:13:27.482011+0000","last_active":"2024-03-26T04:13:45.201041+0000","last_peered":"2024-03-26T04:13:45.201041+0000","last_clean":"2024-03-26T04:13:45.201041+0000","last_became_active":"2024-03-26T04:12:15.664011+0000","last_became_peered":"2024-03-26T04:12:15.664011+0000","last_unstale":"2024-03-26T04:13:45.201041+0000","last_undegraded":"2024-03-26T04:13:45.201041+0000","last_fullsized":"2024-03-26T04:13:45.201041+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.481912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.481912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:31.004240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1089,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663688+0000","last_change":"2024-03-26T04:12:56.978763+0000","last_active":"2024-03-26T04:13:44.663688+0000","last_peered":"2024-03-26T04:13:44.663688+0000","last_clean":"2024-03-26T04:13:44.663688+0000","last_became_active":"2024-03-26T04:12:08.682229+0000","last_became_peered":"2024-03-26T04:12:08.682229+0000","last_unstale":"2024-03-26T04:13:44.663688+0000","last_undegraded":"2024-03-26T04:13:44.663688+0000","last_fullsized":"2024-03-26T04:13:44.663688+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.978663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.978663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:06.093627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"448'33","reported_seq":1364,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664582+0000","last_change":"2024-03-26T04:13:35.055258+0000","last_active":"2024-03-26T04:13:44.664582+0000","last_peered":"2024-03-26T04:13:44.664582+0000","last_clean":"2024-03-26T04:13:44.664582+0000","last_became_active":"2024-03-26T04:12:29.654543+0000","last_became_peered":"2024-03-26T04:12:29.654543+0000","last_unstale":"2024-03-26T04:13:44.664582+0000","last_undegraded":"2024-03-26T04:13:44.664582+0000","last_fullsized":"2024-03-26T04:13:44.664582+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'33","last_scrub_stamp":"2024-03-26T04:13:35.055131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.055131+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.214551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":1210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":510,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200557+0000","last_change":"2024-03-26T04:12:47.630769+0000","last_active":"2024-03-26T04:13:45.200557+0000","last_peered":"2024-03-26T04:13:45.200557+0000","last_clean":"2024-03-26T04:13:45.200557+0000","last_became_active":"2024-03-26T04:12:14.661582+0000","last_became_peered":"2024-03-26T04:12:14.661582+0000","last_unstale":"2024-03-26T04:13:45.200557+0000","last_undegraded":"2024-03-26T04:13:45.200557+0000","last_fullsized":"2024-03-26T04:13:45.200557+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:47.630674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:47.630674+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:07.766240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1032,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.874573+0000","last_change":"2024-03-26T04:13:57.874596+0000","last_active":"2024-03-26T04:13:57.874573+0000","last_peered":"2024-03-26T04:13:57.874573+0000","last_clean":"2024-03-26T04:13:57.874573+0000","last_became_active":"2024-03-26T04:12:11.878355+0000","last_became_peered":"2024-03-26T04:12:11.878355+0000","last_unstale":"2024-03-26T04:13:57.874573+0000","last_undegraded":"2024-03-26T04:13:57.874573+0000","last_fullsized":"2024-03-26T04:13:57.874573+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.874497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:37.500679+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.874497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.148212+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1073,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725953+0000","last_change":"2024-03-26T04:13:03.292505+0000","last_active":"2024-03-26T04:13:44.725953+0000","last_peered":"2024-03-26T04:13:44.725953+0000","last_clean":"2024-03-26T04:13:44.725953+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:13:44.725953+0000","last_undegraded":"2024-03-26T04:13:44.725953+0000","last_fullsized":"2024-03-26T04:13:44.725953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:03.292382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:13:03.292382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:25.987480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734561+0000","last_change":"2024-03-26T04:13:44.734583+0000","last_active":"2024-03-26T04:13:44.734561+0000","last_peered":"2024-03-26T04:13:44.734561+0000","last_clean":"2024-03-26T04:13:44.734561+0000","last_became_active":"2024-03-26T04:12:29.654073+0000","last_became_peered":"2024-03-26T04:12:29.654073+0000","last_unstale":"2024-03-26T04:13:44.734561+0000","last_undegraded":"2024-03-26T04:13:44.734561+0000","last_fullsized":"2024-03-26T04:13:44.734561+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:44.734499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.734499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.687565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"565'53","reported_seq":1159,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:56.981469+0000","last_change":"2024-03-26T04:13:56.981536+0000","last_active":"2024-03-26T04:13:56.981469+0000","last_peered":"2024-03-26T04:13:56.981469+0000","last_clean":"2024-03-26T04:13:56.981469+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:13:56.981469+0000","last_undegraded":"2024-03-26T04:13:56.981469+0000","last_fullsized":"2024-03-26T04:13:56.981469+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'53","last_scrub_stamp":"2024-03-26T04:13:56.981397+0000","last_deep_scrub":"565'53","last_deep_scrub_stamp":"2024-03-26T04:13:56.981397+0000","last_clean_scrub_stamp":"2024-03-26T04:13:56.981397+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:23.609533+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1019,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049648+0000","last_change":"2024-03-26T04:12:52.803427+0000","last_active":"2024-03-26T04:13:45.049648+0000","last_peered":"2024-03-26T04:13:45.049648+0000","last_clean":"2024-03-26T04:13:45.049648+0000","last_became_active":"2024-03-26T04:12:14.661744+0000","last_became_peered":"2024-03-26T04:12:14.661744+0000","last_unstale":"2024-03-26T04:13:45.049648+0000","last_undegraded":"2024-03-26T04:13:45.049648+0000","last_fullsized":"2024-03-26T04:13:45.049648+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:52.803327+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:12:52.803327+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.096349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1083,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664428+0000","last_change":"2024-03-26T04:12:48.090953+0000","last_active":"2024-03-26T04:13:44.664428+0000","last_peered":"2024-03-26T04:13:44.664428+0000","last_clean":"2024-03-26T04:13:44.664428+0000","last_became_active":"2024-03-26T04:12:29.653566+0000","last_became_peered":"2024-03-26T04:12:29.653566+0000","last_unstale":"2024-03-26T04:13:44.664428+0000","last_undegraded":"2024-03-26T04:13:44.664428+0000","last_fullsized":"2024-03-26T04:13:44.664428+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:48.090845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:12:48.090845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:52.933804+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"515'16","reported_seq":1344,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394484+0000","last_change":"2024-03-26T04:12:55.531644+0000","last_active":"2024-03-26T04:13:54.394484+0000","last_peered":"2024-03-26T04:13:54.394484+0000","last_clean":"2024-03-26T04:13:54.394484+0000","last_became_active":"2024-03-26T04:12:08.696335+0000","last_became_peered":"2024-03-26T04:12:08.696335+0000","last_unstale":"2024-03-26T04:13:54.394484+0000","last_undegraded":"2024-03-26T04:13:54.394484+0000","last_fullsized":"2024-03-26T04:13:54.394484+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'16","last_scrub_stamp":"2024-03-26T04:12:55.531545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:55.531545+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.410810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665265+0000","last_change":"2024-03-26T04:13:07.871715+0000","last_active":"2024-03-26T04:13:44.665265+0000","last_peered":"2024-03-26T04:13:44.665265+0000","last_clean":"2024-03-26T04:13:44.665265+0000","last_became_active":"2024-03-26T04:12:11.902647+0000","last_became_peered":"2024-03-26T04:12:11.902647+0000","last_unstale":"2024-03-26T04:13:44.665265+0000","last_undegraded":"2024-03-26T04:13:44.665265+0000","last_fullsized":"2024-03-26T04:13:44.665265+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:07.871615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:07.871615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.672879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"448'30","reported_seq":1261,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664059+0000","last_change":"2024-03-26T04:13:42.106430+0000","last_active":"2024-03-26T04:13:44.664059+0000","last_peered":"2024-03-26T04:13:44.664059+0000","last_clean":"2024-03-26T04:13:44.664059+0000","last_became_active":"2024-03-26T04:12:15.669351+0000","last_became_peered":"2024-03-26T04:12:15.669351+0000","last_unstale":"2024-03-26T04:13:44.664059+0000","last_undegraded":"2024-03-26T04:13:44.664059+0000","last_fullsized":"2024-03-26T04:13:44.664059+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'30","last_scrub_stamp":"2024-03-26T04:13:42.106315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:42.106315+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.243456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":112,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1817,"num_read_kb":1542,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":17884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1298,"ondisk_log_size":1298,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070772,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012028,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733124,"statfs":{"total":95995035648,"available":94958718976,"internally_reserved":0,"allocated":2150400,"data_stored":1309827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1034156490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365115,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37084,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708068,"statfs":{"total":95995035648,"available":95957061632,"internally_reserved":0,"allocated":1789952,"data_stored":979735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575187,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188884,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":1187070,"kb_avail":92556268,"statfs":{"total":95995035648,"available":94777618432,"internally_reserved":0,"allocated":1847296,"data_stored":981300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1215560133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738712,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295740,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":1293950,"kb_avail":92449412,"statfs":{"total":95995035648,"available":94668197888,"internally_reserved":0,"allocated":1822720,"data_stored":981876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1325005253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637993,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666280,"kb_used_data":2016,"kb_used_omap":6,"kb_used_meta":664249,"kb_avail":93078872,"statfs":{"total":95995035648,"available":95312764928,"internally_reserved":0,"allocated":2064384,"data_stored":1242771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":680191269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182729,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565260,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93179892,"statfs":{"total":95995035648,"available":95416209408,"internally_reserved":0,"allocated":2428928,"data_stored":1572341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":576387527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953795,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161212,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":1159422,"kb_avail":92583940,"statfs":{"total":95995035648,"available":94805954560,"internally_reserved":0,"allocated":1822720,"data_stored":980353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1187248581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113499,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487076,"kb_used_data":2332,"kb_used_omap":4,"kb_used_meta":484731,"kb_avail":93258076,"statfs":{"total":95995035648,"available":95496269824,"internally_reserved":0,"allocated":2387968,"data_stored":1570429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":496365216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:14:06.092 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:14:06.093 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:05 smithi086 ceph-mon[32965]: pgmap v1241: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:06.093 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:05 smithi086 ceph-mon[32965]: 6.e scrub starts 2024-03-26T04:14:06.093 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:14:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:14:06.343 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:06.738 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[37501]: 6.e scrub ok 2024-03-26T04:14:06.738 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[37501]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:06.738 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[37501]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:14:06.739 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[29446]: 6.e scrub ok 2024-03-26T04:14:06.739 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[29446]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:06.739 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:06 smithi006 ceph-mon[29446]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:14:07.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:06 smithi086 ceph-mon[32965]: 6.e scrub ok 2024-03-26T04:14:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:06 smithi086 ceph-mon[32965]: from='client.16218 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:07.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:06 smithi086 ceph-mon[32965]: from='client.24421 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:14:07.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[37501]: pgmap v1242: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:07.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:07.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[37501]: 6.2 deep-scrub starts 2024-03-26T04:14:07.765 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[37501]: 6.2 deep-scrub ok 2024-03-26T04:14:07.765 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[29446]: pgmap v1242: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:07.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:07.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[29446]: 6.2 deep-scrub starts 2024-03-26T04:14:07.766 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:07 smithi006 ceph-mon[29446]: 6.2 deep-scrub ok 2024-03-26T04:14:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:07 smithi086 ceph-mon[32965]: pgmap v1242: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:07 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:07 smithi086 ceph-mon[32965]: 6.2 deep-scrub starts 2024-03-26T04:14:08.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:07 smithi086 ceph-mon[32965]: 6.2 deep-scrub ok 2024-03-26T04:14:08.372 INFO:teuthology.orchestra.run.smithi006.stdout: 2024-03-26T04:14:08.375 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2024-03-26T04:14:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:08 smithi086 ceph-mon[32965]: 6.c scrub starts 2024-03-26T04:14:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:08 smithi086 ceph-mon[32965]: 6.c scrub ok 2024-03-26T04:14:08.837 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:08 smithi086 ceph-mon[32965]: 3.1a scrub starts 2024-03-26T04:14:08.838 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:08 smithi086 ceph-mon[32965]: 3.1a scrub ok 2024-03-26T04:14:08.919 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[37501]: 6.c scrub starts 2024-03-26T04:14:08.919 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[37501]: 6.c scrub ok 2024-03-26T04:14:08.919 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[37501]: 3.1a scrub starts 2024-03-26T04:14:08.919 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[37501]: 3.1a scrub ok 2024-03-26T04:14:08.919 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[29446]: 6.c scrub starts 2024-03-26T04:14:08.920 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[29446]: 6.c scrub ok 2024-03-26T04:14:08.920 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[29446]: 3.1a scrub starts 2024-03-26T04:14:08.920 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:08 smithi006 ceph-mon[29446]: 3.1a scrub ok 2024-03-26T04:14:09.367 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":1242,"stamp":"2024-03-26T04:14:06.687477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1963,"num_read_kb":1683,"num_write":1200,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1099,"num_bytes_recovered":1210983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2860,"ondisk_log_size":2860,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6413564,"kb_used_data":15932,"kb_used_omap":20,"kb_used_meta":6397547,"kb_avail":743547652,"statfs":{"total":767960285184,"available":761392795648,"internally_reserved":0,"allocated":16314368,"data_stored":9618632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20885,"internal_metadata":6551088747},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003417"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":1106,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.124667+0000","last_change":"2024-03-26T04:13:46.124667+0000","last_active":"2024-03-26T04:13:46.124667+0000","last_peered":"2024-03-26T04:13:46.124667+0000","last_clean":"2024-03-26T04:13:46.124667+0000","last_became_active":"2024-03-26T04:12:29.655731+0000","last_became_peered":"2024-03-26T04:12:29.655731+0000","last_unstale":"2024-03-26T04:13:46.124667+0000","last_undegraded":"2024-03-26T04:13:46.124667+0000","last_fullsized":"2024-03-26T04:13:46.124667+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.124561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.124561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.622643+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1106,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725361+0000","last_change":"2024-03-26T04:13:27.537492+0000","last_active":"2024-03-26T04:13:44.725361+0000","last_peered":"2024-03-26T04:13:44.725361+0000","last_clean":"2024-03-26T04:13:44.725361+0000","last_became_active":"2024-03-26T04:12:08.689302+0000","last_became_peered":"2024-03-26T04:12:08.689302+0000","last_unstale":"2024-03-26T04:13:44.725361+0000","last_undegraded":"2024-03-26T04:13:44.725361+0000","last_fullsized":"2024-03-26T04:13:44.725361+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.537375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:09.336550+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.537375+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:50.160214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":103,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.202688+0000","last_change":"2024-03-26T04:13:45.202709+0000","last_active":"2024-03-26T04:13:45.202688+0000","last_peered":"2024-03-26T04:13:45.202688+0000","last_clean":"2024-03-26T04:13:45.202688+0000","last_became_active":"2024-03-26T04:12:21.681921+0000","last_became_peered":"2024-03-26T04:12:21.681921+0000","last_unstale":"2024-03-26T04:13:45.202688+0000","last_undegraded":"2024-03-26T04:13:45.202688+0000","last_fullsized":"2024-03-26T04:13:45.202688+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:45.202631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:32.371775+0000","last_clean_scrub_stamp":"2024-03-26T04:13:45.202631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:48.674388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"546'32","reported_seq":1231,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663557+0000","last_change":"2024-03-26T04:13:05.914382+0000","last_active":"2024-03-26T04:13:44.663557+0000","last_peered":"2024-03-26T04:13:44.663557+0000","last_clean":"2024-03-26T04:13:44.663557+0000","last_became_active":"2024-03-26T04:12:15.670323+0000","last_became_peered":"2024-03-26T04:12:15.670323+0000","last_unstale":"2024-03-26T04:13:44.663557+0000","last_undegraded":"2024-03-26T04:13:44.663557+0000","last_fullsized":"2024-03-26T04:13:44.663557+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"546'32","last_scrub_stamp":"2024-03-26T04:13:05.914240+0000","last_deep_scrub":"546'32","last_deep_scrub_stamp":"2024-03-26T04:13:05.914240+0000","last_clean_scrub_stamp":"2024-03-26T04:13:05.914240+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.043842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665018+0000","last_change":"2024-03-26T04:12:46.080880+0000","last_active":"2024-03-26T04:13:44.665018+0000","last_peered":"2024-03-26T04:13:44.665018+0000","last_clean":"2024-03-26T04:13:44.665018+0000","last_became_active":"2024-03-26T04:12:15.649855+0000","last_became_peered":"2024-03-26T04:12:15.649855+0000","last_unstale":"2024-03-26T04:13:44.665018+0000","last_undegraded":"2024-03-26T04:13:44.665018+0000","last_fullsized":"2024-03-26T04:13:44.665018+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.080779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:35.321382+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.080779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.688251+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1088,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663570+0000","last_change":"2024-03-26T04:13:23.957403+0000","last_active":"2024-03-26T04:13:44.663570+0000","last_peered":"2024-03-26T04:13:44.663570+0000","last_clean":"2024-03-26T04:13:44.663570+0000","last_became_active":"2024-03-26T04:12:15.669617+0000","last_became_peered":"2024-03-26T04:12:15.669617+0000","last_unstale":"2024-03-26T04:13:44.663570+0000","last_undegraded":"2024-03-26T04:13:44.663570+0000","last_fullsized":"2024-03-26T04:13:44.663570+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.957300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.957300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:37.008656+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665966+0000","last_change":"2024-03-26T04:13:14.818369+0000","last_active":"2024-03-26T04:13:44.665966+0000","last_peered":"2024-03-26T04:13:44.665966+0000","last_clean":"2024-03-26T04:13:44.665966+0000","last_became_active":"2024-03-26T04:12:15.653430+0000","last_became_peered":"2024-03-26T04:12:15.653430+0000","last_unstale":"2024-03-26T04:13:44.665966+0000","last_undegraded":"2024-03-26T04:13:44.665966+0000","last_fullsized":"2024-03-26T04:13:44.665966+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:14.818268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:32.445390+0000","last_clean_scrub_stamp":"2024-03-26T04:13:14.818268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:26.009674+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"448'63","reported_seq":137,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665598+0000","last_change":"2024-03-26T04:13:19.760651+0000","last_active":"2024-03-26T04:13:44.665598+0000","last_peered":"2024-03-26T04:13:44.665598+0000","last_clean":"2024-03-26T04:13:44.665598+0000","last_became_active":"2024-03-26T04:12:11.891462+0000","last_became_peered":"2024-03-26T04:12:11.891462+0000","last_unstale":"2024-03-26T04:13:44.665598+0000","last_undegraded":"2024-03-26T04:13:44.665598+0000","last_fullsized":"2024-03-26T04:13:44.665598+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'63","last_scrub_stamp":"2024-03-26T04:13:19.760505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.760505+0000","objects_scrubbed":11,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:25.575667+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":95,"num_read_kb":82,"num_write":64,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665905+0000","last_change":"2024-03-26T04:13:25.009186+0000","last_active":"2024-03-26T04:13:44.665905+0000","last_peered":"2024-03-26T04:13:44.665905+0000","last_clean":"2024-03-26T04:13:44.665905+0000","last_became_active":"2024-03-26T04:12:21.682966+0000","last_became_peered":"2024-03-26T04:12:21.682966+0000","last_unstale":"2024-03-26T04:13:44.665905+0000","last_undegraded":"2024-03-26T04:13:44.665905+0000","last_fullsized":"2024-03-26T04:13:44.665905+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:25.009071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:11:48.080284+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.009071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.427573+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200156+0000","last_change":"2024-03-26T04:13:13.603271+0000","last_active":"2024-03-26T04:13:45.200156+0000","last_peered":"2024-03-26T04:13:45.200156+0000","last_clean":"2024-03-26T04:13:45.200156+0000","last_became_active":"2024-03-26T04:12:08.697573+0000","last_became_peered":"2024-03-26T04:12:08.697573+0000","last_unstale":"2024-03-26T04:13:45.200156+0000","last_undegraded":"2024-03-26T04:13:45.200156+0000","last_fullsized":"2024-03-26T04:13:45.200156+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:13.603173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:00:25.328566+0000","last_clean_scrub_stamp":"2024-03-26T04:13:13.603173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:30.358135+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201353+0000","last_change":"2024-03-26T04:13:23.528861+0000","last_active":"2024-03-26T04:13:45.201353+0000","last_peered":"2024-03-26T04:13:45.201353+0000","last_clean":"2024-03-26T04:13:45.201353+0000","last_became_active":"2024-03-26T04:12:15.664653+0000","last_became_peered":"2024-03-26T04:12:15.664653+0000","last_unstale":"2024-03-26T04:13:45.201353+0000","last_undegraded":"2024-03-26T04:13:45.201353+0000","last_fullsized":"2024-03-26T04:13:45.201353+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.528745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.528745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:41.877652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"448'34","reported_seq":1219,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:59.063075+0000","last_change":"2024-03-26T04:13:59.063115+0000","last_active":"2024-03-26T04:13:59.063075+0000","last_peered":"2024-03-26T04:13:59.063075+0000","last_clean":"2024-03-26T04:13:59.063075+0000","last_became_active":"2024-03-26T04:12:11.894786+0000","last_became_peered":"2024-03-26T04:12:11.894786+0000","last_unstale":"2024-03-26T04:13:59.063075+0000","last_undegraded":"2024-03-26T04:13:59.063075+0000","last_fullsized":"2024-03-26T04:13:59.063075+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'34","last_scrub_stamp":"2024-03-26T04:13:59.062991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:59.062991+0000","objects_scrubbed":6,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:22.044808+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200435+0000","last_change":"2024-03-26T04:13:24.654316+0000","last_active":"2024-03-26T04:13:45.200435+0000","last_peered":"2024-03-26T04:13:45.200435+0000","last_clean":"2024-03-26T04:13:45.200435+0000","last_became_active":"2024-03-26T04:12:11.888977+0000","last_became_peered":"2024-03-26T04:12:11.888977+0000","last_unstale":"2024-03-26T04:13:45.200435+0000","last_undegraded":"2024-03-26T04:13:45.200435+0000","last_fullsized":"2024-03-26T04:13:45.200435+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.654180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.654180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.856932+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048742+0000","last_change":"2024-03-26T04:13:44.047533+0000","last_active":"2024-03-26T04:13:45.048742+0000","last_peered":"2024-03-26T04:13:45.048742+0000","last_clean":"2024-03-26T04:13:45.048742+0000","last_became_active":"2024-03-26T04:12:08.677750+0000","last_became_peered":"2024-03-26T04:12:08.677750+0000","last_unstale":"2024-03-26T04:13:45.048742+0000","last_undegraded":"2024-03-26T04:13:45.048742+0000","last_fullsized":"2024-03-26T04:13:45.048742+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:44.047433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:44.047433+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.047433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:44.780691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":129,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201043+0000","last_change":"2024-03-26T04:12:57.557592+0000","last_active":"2024-03-26T04:13:45.201043+0000","last_peered":"2024-03-26T04:13:45.201043+0000","last_clean":"2024-03-26T04:13:45.201043+0000","last_became_active":"2024-03-26T04:12:03.859469+0000","last_became_peered":"2024-03-26T04:12:03.859469+0000","last_unstale":"2024-03-26T04:13:45.201043+0000","last_undegraded":"2024-03-26T04:13:45.201043+0000","last_fullsized":"2024-03-26T04:13:45.201043+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:57.557503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:10:35.315076+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.557503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:08.411348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"448'18","reported_seq":1201,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664289+0000","last_change":"2024-03-26T04:13:15.014520+0000","last_active":"2024-03-26T04:13:44.664289+0000","last_peered":"2024-03-26T04:13:44.664289+0000","last_clean":"2024-03-26T04:13:44.664289+0000","last_became_active":"2024-03-26T04:12:29.654762+0000","last_became_peered":"2024-03-26T04:12:29.654762+0000","last_unstale":"2024-03-26T04:13:44.664289+0000","last_undegraded":"2024-03-26T04:13:44.664289+0000","last_fullsized":"2024-03-26T04:13:44.664289+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'18","last_scrub_stamp":"2024-03-26T04:13:15.014415+0000","last_deep_scrub":"241'16","last_deep_scrub_stamp":"2024-03-26T04:08:33.351097+0000","last_clean_scrub_stamp":"2024-03-26T04:13:15.014415+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.629458+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":1128,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664560+0000","last_change":"2024-03-26T04:13:39.241627+0000","last_active":"2024-03-26T04:13:44.664560+0000","last_peered":"2024-03-26T04:13:44.664560+0000","last_clean":"2024-03-26T04:13:44.664560+0000","last_became_active":"2024-03-26T04:12:15.672724+0000","last_became_peered":"2024-03-26T04:12:15.672724+0000","last_unstale":"2024-03-26T04:13:44.664560+0000","last_undegraded":"2024-03-26T04:13:44.664560+0000","last_fullsized":"2024-03-26T04:13:44.664560+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:39.241518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:39.241518+0000","last_clean_scrub_stamp":"2024-03-26T04:13:39.241518+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:44.840603+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1072,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.726041+0000","last_change":"2024-03-26T04:13:18.353035+0000","last_active":"2024-03-26T04:13:44.726041+0000","last_peered":"2024-03-26T04:13:44.726041+0000","last_clean":"2024-03-26T04:13:44.726041+0000","last_became_active":"2024-03-26T03:48:48.025184+0000","last_became_peered":"2024-03-26T03:48:48.025184+0000","last_unstale":"2024-03-26T04:13:44.726041+0000","last_undegraded":"2024-03-26T04:13:44.726041+0000","last_fullsized":"2024-03-26T04:13:44.726041+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.352950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.352950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:34.865310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663254+0000","last_change":"2024-03-26T04:12:46.691394+0000","last_active":"2024-03-26T04:13:44.663254+0000","last_peered":"2024-03-26T04:13:44.663254+0000","last_clean":"2024-03-26T04:13:44.663254+0000","last_became_active":"2024-03-26T04:12:21.686772+0000","last_became_peered":"2024-03-26T04:12:21.686772+0000","last_unstale":"2024-03-26T04:13:44.663254+0000","last_undegraded":"2024-03-26T04:13:44.663254+0000","last_fullsized":"2024-03-26T04:13:44.663254+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.691275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.691275+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:59.880321+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"564'46","reported_seq":1181,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898677+0000","last_change":"2024-03-26T04:13:43.945698+0000","last_active":"2024-03-26T04:13:44.898677+0000","last_peered":"2024-03-26T04:13:44.898677+0000","last_clean":"2024-03-26T04:13:44.898677+0000","last_became_active":"2024-03-26T04:12:11.880784+0000","last_became_peered":"2024-03-26T04:12:11.880784+0000","last_unstale":"2024-03-26T04:13:44.898677+0000","last_undegraded":"2024-03-26T04:13:44.898677+0000","last_fullsized":"2024-03-26T04:13:44.898677+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"564'46","last_scrub_stamp":"2024-03-26T04:13:43.945600+0000","last_deep_scrub":"564'46","last_deep_scrub_stamp":"2024-03-26T04:13:43.945600+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.945600+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:05.848463+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664902+0000","last_change":"2024-03-26T04:12:35.953331+0000","last_active":"2024-03-26T04:13:44.664902+0000","last_peered":"2024-03-26T04:13:44.664902+0000","last_clean":"2024-03-26T04:13:44.664902+0000","last_became_active":"2024-03-26T04:12:15.655603+0000","last_became_peered":"2024-03-26T04:12:15.655603+0000","last_unstale":"2024-03-26T04:13:44.664902+0000","last_undegraded":"2024-03-26T04:13:44.664902+0000","last_fullsized":"2024-03-26T04:13:44.664902+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:35.953232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:38.222375+0000","last_clean_scrub_stamp":"2024-03-26T04:12:35.953232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:00.094640+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1112,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:01.052908+0000","last_change":"2024-03-26T04:14:01.052937+0000","last_active":"2024-03-26T04:14:01.052908+0000","last_peered":"2024-03-26T04:14:01.052908+0000","last_clean":"2024-03-26T04:14:01.052908+0000","last_became_active":"2024-03-26T04:12:11.894669+0000","last_became_peered":"2024-03-26T04:12:11.894669+0000","last_unstale":"2024-03-26T04:14:01.052908+0000","last_undegraded":"2024-03-26T04:14:01.052908+0000","last_fullsized":"2024-03-26T04:14:01.052908+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:14:01.052818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:14:01.052818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:15.558894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":1076,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663970+0000","last_change":"2024-03-26T04:12:32.963874+0000","last_active":"2024-03-26T04:13:44.663970+0000","last_peered":"2024-03-26T04:13:44.663970+0000","last_clean":"2024-03-26T04:13:44.663970+0000","last_became_active":"2024-03-26T04:05:29.520346+0000","last_became_peered":"2024-03-26T04:05:29.520346+0000","last_unstale":"2024-03-26T04:13:44.663970+0000","last_undegraded":"2024-03-26T04:13:44.663970+0000","last_fullsized":"2024-03-26T04:13:44.663970+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:32.963756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:32.963756+0000","last_clean_scrub_stamp":"2024-03-26T04:12:32.963756+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:57.321907+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"514'36","reported_seq":127,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.729086+0000","last_change":"2024-03-26T04:13:44.729119+0000","last_active":"2024-03-26T04:13:44.729086+0000","last_peered":"2024-03-26T04:13:44.729086+0000","last_clean":"2024-03-26T04:13:44.729086+0000","last_became_active":"2024-03-26T04:12:11.900079+0000","last_became_peered":"2024-03-26T04:12:11.900079+0000","last_unstale":"2024-03-26T04:13:44.729086+0000","last_undegraded":"2024-03-26T04:13:44.729086+0000","last_fullsized":"2024-03-26T04:13:44.729086+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"514'36","last_scrub_stamp":"2024-03-26T04:13:44.729030+0000","last_deep_scrub":"514'36","last_deep_scrub_stamp":"2024-03-26T04:12:08.309354+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.729030+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.535514+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"87'2","reported_seq":69,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049746+0000","last_change":"2024-03-26T04:13:06.668683+0000","last_active":"2024-03-26T04:13:45.049746+0000","last_peered":"2024-03-26T04:13:45.049746+0000","last_clean":"2024-03-26T04:13:45.049746+0000","last_became_active":"2024-03-26T04:13:06.668313+0000","last_became_peered":"2024-03-26T04:13:06.668313+0000","last_unstale":"2024-03-26T04:13:45.049746+0000","last_undegraded":"2024-03-26T04:13:45.049746+0000","last_fullsized":"2024-03-26T04:13:45.049746+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'2","last_scrub_stamp":"2024-03-26T04:12:51.678255+0000","last_deep_scrub":"87'2","last_deep_scrub_stamp":"2024-03-26T03:50:02.740545+0000","last_clean_scrub_stamp":"2024-03-26T04:12:51.678255+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:20.460831+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"525'16","reported_seq":138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394717+0000","last_change":"2024-03-26T04:13:46.796970+0000","last_active":"2024-03-26T04:13:54.394717+0000","last_peered":"2024-03-26T04:13:54.394717+0000","last_clean":"2024-03-26T04:13:54.394717+0000","last_became_active":"2024-03-26T04:12:15.665266+0000","last_became_peered":"2024-03-26T04:12:15.665266+0000","last_unstale":"2024-03-26T04:13:54.394717+0000","last_undegraded":"2024-03-26T04:13:54.394717+0000","last_fullsized":"2024-03-26T04:13:54.394717+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'16","last_scrub_stamp":"2024-03-26T04:13:46.796865+0000","last_deep_scrub":"523'15","last_deep_scrub_stamp":"2024-03-26T04:12:17.024905+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.796865+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.010999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"255'2","reported_seq":1142,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725264+0000","last_change":"2024-03-26T04:12:59.294279+0000","last_active":"2024-03-26T04:13:44.725264+0000","last_peered":"2024-03-26T04:13:44.725264+0000","last_clean":"2024-03-26T04:13:44.725264+0000","last_became_active":"2024-03-26T04:12:11.892352+0000","last_became_peered":"2024-03-26T04:12:11.892352+0000","last_unstale":"2024-03-26T04:13:44.725264+0000","last_undegraded":"2024-03-26T04:13:44.725264+0000","last_fullsized":"2024-03-26T04:13:44.725264+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"255'2","last_scrub_stamp":"2024-03-26T04:12:59.294189+0000","last_deep_scrub":"255'2","last_deep_scrub_stamp":"2024-03-26T03:59:26.324998+0000","last_clean_scrub_stamp":"2024-03-26T04:12:59.294189+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:09.887787+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"448'42","reported_seq":1276,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665865+0000","last_change":"2024-03-26T04:12:58.672638+0000","last_active":"2024-03-26T04:13:44.665865+0000","last_peered":"2024-03-26T04:13:44.665865+0000","last_clean":"2024-03-26T04:13:44.665865+0000","last_became_active":"2024-03-26T04:12:11.896329+0000","last_became_peered":"2024-03-26T04:12:11.896329+0000","last_unstale":"2024-03-26T04:13:44.665865+0000","last_undegraded":"2024-03-26T04:13:44.665865+0000","last_fullsized":"2024-03-26T04:13:44.665865+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'42","last_scrub_stamp":"2024-03-26T04:12:58.672508+0000","last_deep_scrub":"356'32","last_deep_scrub_stamp":"2024-03-26T04:04:05.344849+0000","last_clean_scrub_stamp":"2024-03-26T04:12:58.672508+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:58.737522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":1090,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725663+0000","last_change":"2024-03-26T04:13:34.758039+0000","last_active":"2024-03-26T04:13:44.725663+0000","last_peered":"2024-03-26T04:13:44.725663+0000","last_clean":"2024-03-26T04:13:44.725663+0000","last_became_active":"2024-03-26T04:12:03.850434+0000","last_became_peered":"2024-03-26T04:12:03.850434+0000","last_unstale":"2024-03-26T04:13:44.725663+0000","last_undegraded":"2024-03-26T04:13:44.725663+0000","last_fullsized":"2024-03-26T04:13:44.725663+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:34.757931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:34.757931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:55.041262+0000","scrub_duration":249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1096,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665576+0000","last_change":"2024-03-26T04:13:38.088706+0000","last_active":"2024-03-26T04:13:44.665576+0000","last_peered":"2024-03-26T04:13:44.665576+0000","last_clean":"2024-03-26T04:13:44.665576+0000","last_became_active":"2024-03-26T04:12:08.677498+0000","last_became_peered":"2024-03-26T04:12:08.677498+0000","last_unstale":"2024-03-26T04:13:44.665576+0000","last_undegraded":"2024-03-26T04:13:44.665576+0000","last_fullsized":"2024-03-26T04:13:44.665576+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:38.088621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:51:33.548192+0000","last_clean_scrub_stamp":"2024-03-26T04:13:38.088621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.365215+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":125,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664156+0000","last_change":"2024-03-26T04:13:32.600753+0000","last_active":"2024-03-26T04:13:44.664156+0000","last_peered":"2024-03-26T04:13:44.664156+0000","last_clean":"2024-03-26T04:13:44.664156+0000","last_became_active":"2024-03-26T04:12:29.659001+0000","last_became_peered":"2024-03-26T04:12:29.659001+0000","last_unstale":"2024-03-26T04:13:44.664156+0000","last_undegraded":"2024-03-26T04:13:44.664156+0000","last_fullsized":"2024-03-26T04:13:44.664156+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:32.600650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:32.600650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.249173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"527'41","reported_seq":132,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734338+0000","last_change":"2024-03-26T04:13:25.824502+0000","last_active":"2024-03-26T04:13:44.734338+0000","last_peered":"2024-03-26T04:13:44.734338+0000","last_clean":"2024-03-26T04:13:44.734338+0000","last_became_active":"2024-03-26T04:12:15.660617+0000","last_became_peered":"2024-03-26T04:12:15.660617+0000","last_unstale":"2024-03-26T04:13:44.734338+0000","last_undegraded":"2024-03-26T04:13:44.734338+0000","last_fullsized":"2024-03-26T04:13:44.734338+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"527'41","last_scrub_stamp":"2024-03-26T04:13:25.824320+0000","last_deep_scrub":"527'41","last_deep_scrub_stamp":"2024-03-26T04:12:22.934443+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.824320+0000","objects_scrubbed":6,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:33.203418+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":42,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048924+0000","last_change":"2024-03-26T04:13:19.156112+0000","last_active":"2024-03-26T04:13:45.048924+0000","last_peered":"2024-03-26T04:13:45.048924+0000","last_clean":"2024-03-26T04:13:45.048924+0000","last_became_active":"2024-03-26T04:12:08.683874+0000","last_became_peered":"2024-03-26T04:12:08.683874+0000","last_unstale":"2024-03-26T04:13:45.048924+0000","last_undegraded":"2024-03-26T04:13:45.048924+0000","last_fullsized":"2024-03-26T04:13:45.048924+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:19.155998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:42.011134+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.155998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:26.990024+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":104,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734119+0000","last_change":"2024-03-26T04:13:00.108543+0000","last_active":"2024-03-26T04:13:44.734119+0000","last_peered":"2024-03-26T04:13:44.734119+0000","last_clean":"2024-03-26T04:13:44.734119+0000","last_became_active":"2024-03-26T04:12:15.661728+0000","last_became_peered":"2024-03-26T04:12:15.661728+0000","last_unstale":"2024-03-26T04:13:44.734119+0000","last_undegraded":"2024-03-26T04:13:44.734119+0000","last_fullsized":"2024-03-26T04:13:44.734119+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:00.108436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:00.108436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.300626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1129,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665056+0000","last_change":"2024-03-26T04:13:34.064391+0000","last_active":"2024-03-26T04:13:44.665056+0000","last_peered":"2024-03-26T04:13:44.665056+0000","last_clean":"2024-03-26T04:13:44.665056+0000","last_became_active":"2024-03-26T04:12:11.896146+0000","last_became_peered":"2024-03-26T04:12:11.896146+0000","last_unstale":"2024-03-26T04:13:44.665056+0000","last_undegraded":"2024-03-26T04:13:44.665056+0000","last_fullsized":"2024-03-26T04:13:44.665056+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:34.064307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:34.064307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:34.346295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"448'14","reported_seq":93,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200277+0000","last_change":"2024-03-26T04:13:40.424554+0000","last_active":"2024-03-26T04:13:45.200277+0000","last_peered":"2024-03-26T04:13:45.200277+0000","last_clean":"2024-03-26T04:13:45.200277+0000","last_became_active":"2024-03-26T04:12:27.733789+0000","last_became_peered":"2024-03-26T04:12:27.733789+0000","last_unstale":"2024-03-26T04:13:45.200277+0000","last_undegraded":"2024-03-26T04:13:45.200277+0000","last_fullsized":"2024-03-26T04:13:45.200277+0000","mapping_epoch":530,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":531,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'14","last_scrub_stamp":"2024-03-26T04:13:40.424448+0000","last_deep_scrub":"448'14","last_deep_scrub_stamp":"2024-03-26T04:12:24.786427+0000","last_clean_scrub_stamp":"2024-03-26T04:13:40.424448+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.543859+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":125,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665184+0000","last_change":"2024-03-26T04:12:27.977716+0000","last_active":"2024-03-26T04:13:44.665184+0000","last_peered":"2024-03-26T04:13:44.665184+0000","last_clean":"2024-03-26T04:13:44.665184+0000","last_became_active":"2024-03-26T04:12:21.854036+0000","last_became_peered":"2024-03-26T04:12:21.854036+0000","last_unstale":"2024-03-26T04:13:44.665184+0000","last_undegraded":"2024-03-26T04:13:44.665184+0000","last_fullsized":"2024-03-26T04:13:44.665184+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:27.977610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:55:29.311918+0000","last_clean_scrub_stamp":"2024-03-26T04:12:27.977610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:56.604807+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049323+0000","last_change":"2024-03-26T04:12:40.947206+0000","last_active":"2024-03-26T04:13:45.049323+0000","last_peered":"2024-03-26T04:13:45.049323+0000","last_clean":"2024-03-26T04:13:45.049323+0000","last_became_active":"2024-03-26T04:12:15.659773+0000","last_became_peered":"2024-03-26T04:12:15.659773+0000","last_unstale":"2024-03-26T04:13:45.049323+0000","last_undegraded":"2024-03-26T04:13:45.049323+0000","last_fullsized":"2024-03-26T04:13:45.049323+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:40.947102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:40.947102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:50.092363+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":147,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200939+0000","last_change":"2024-03-26T04:13:15.570101+0000","last_active":"2024-03-26T04:13:45.200939+0000","last_peered":"2024-03-26T04:13:45.200939+0000","last_clean":"2024-03-26T04:13:45.200939+0000","last_became_active":"2024-03-26T04:12:29.653443+0000","last_became_peered":"2024-03-26T04:12:29.653443+0000","last_unstale":"2024-03-26T04:13:45.200939+0000","last_undegraded":"2024-03-26T04:13:45.200939+0000","last_fullsized":"2024-03-26T04:13:45.200939+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:15.570014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:15.570014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.944021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"448'39","reported_seq":155,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.808701+0000","last_change":"2024-03-26T04:13:49.808763+0000","last_active":"2024-03-26T04:13:49.808701+0000","last_peered":"2024-03-26T04:13:49.808701+0000","last_clean":"2024-03-26T04:13:49.808701+0000","last_became_active":"2024-03-26T04:12:15.651591+0000","last_became_peered":"2024-03-26T04:12:15.651591+0000","last_unstale":"2024-03-26T04:13:49.808701+0000","last_undegraded":"2024-03-26T04:13:49.808701+0000","last_fullsized":"2024-03-26T04:13:49.808701+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:49.808602+0000","last_deep_scrub":"89'23","last_deep_scrub_stamp":"2024-03-26T03:50:20.135174+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.808602+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.344595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1090,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664024+0000","last_change":"2024-03-26T04:12:45.797882+0000","last_active":"2024-03-26T04:13:44.664024+0000","last_peered":"2024-03-26T04:13:44.664024+0000","last_clean":"2024-03-26T04:13:44.664024+0000","last_became_active":"2024-03-26T04:12:03.854120+0000","last_became_peered":"2024-03-26T04:12:03.854120+0000","last_unstale":"2024-03-26T04:13:44.664024+0000","last_undegraded":"2024-03-26T04:13:44.664024+0000","last_fullsized":"2024-03-26T04:13:44.664024+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:45.797787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:45.797787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:10.096962+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665105+0000","last_change":"2024-03-26T04:13:38.656051+0000","last_active":"2024-03-26T04:13:44.665105+0000","last_peered":"2024-03-26T04:13:44.665105+0000","last_clean":"2024-03-26T04:13:44.665105+0000","last_became_active":"2024-03-26T04:12:11.891221+0000","last_became_peered":"2024-03-26T04:12:11.891221+0000","last_unstale":"2024-03-26T04:13:44.665105+0000","last_undegraded":"2024-03-26T04:13:44.665105+0000","last_fullsized":"2024-03-26T04:13:44.665105+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:38.655950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:38.655950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.096084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050175+0000","last_change":"2024-03-26T04:12:33.983181+0000","last_active":"2024-03-26T04:13:45.050175+0000","last_peered":"2024-03-26T04:13:45.050175+0000","last_clean":"2024-03-26T04:13:45.050175+0000","last_became_active":"2024-03-26T04:12:11.887377+0000","last_became_peered":"2024-03-26T04:12:11.887377+0000","last_unstale":"2024-03-26T04:13:45.050175+0000","last_undegraded":"2024-03-26T04:13:45.050175+0000","last_fullsized":"2024-03-26T04:13:45.050175+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:33.983081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:33.983081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"448'25","reported_seq":93,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.445861+0000","last_change":"2024-03-26T04:13:46.445909+0000","last_active":"2024-03-26T04:13:46.445861+0000","last_peered":"2024-03-26T04:13:46.445861+0000","last_clean":"2024-03-26T04:13:46.445861+0000","last_became_active":"2024-03-26T04:12:29.654852+0000","last_became_peered":"2024-03-26T04:12:29.654852+0000","last_unstale":"2024-03-26T04:13:46.445861+0000","last_undegraded":"2024-03-26T04:13:46.445861+0000","last_fullsized":"2024-03-26T04:13:46.445861+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'25","last_scrub_stamp":"2024-03-26T04:13:46.445788+0000","last_deep_scrub":"152'13","last_deep_scrub_stamp":"2024-03-26T03:54:26.861674+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.445788+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.114454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664294+0000","last_change":"2024-03-26T04:12:41.955305+0000","last_active":"2024-03-26T04:13:44.664294+0000","last_peered":"2024-03-26T04:13:44.664294+0000","last_clean":"2024-03-26T04:13:44.664294+0000","last_became_active":"2024-03-26T04:12:11.890613+0000","last_became_peered":"2024-03-26T04:12:11.890613+0000","last_unstale":"2024-03-26T04:13:44.664294+0000","last_undegraded":"2024-03-26T04:13:44.664294+0000","last_fullsized":"2024-03-26T04:13:44.664294+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:41.955200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:34.804652+0000","last_clean_scrub_stamp":"2024-03-26T04:12:41.955200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:58.969597+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1121,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725767+0000","last_change":"2024-03-26T04:12:56.335382+0000","last_active":"2024-03-26T04:13:44.725767+0000","last_peered":"2024-03-26T04:13:44.725767+0000","last_clean":"2024-03-26T04:13:44.725767+0000","last_became_active":"2024-03-26T04:12:11.892084+0000","last_became_peered":"2024-03-26T04:12:11.892084+0000","last_unstale":"2024-03-26T04:13:44.725767+0000","last_undegraded":"2024-03-26T04:13:44.725767+0000","last_fullsized":"2024-03-26T04:13:44.725767+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.335288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:56.335288+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.335288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:11.604175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":1108,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664027+0000","last_change":"2024-03-26T04:13:18.053542+0000","last_active":"2024-03-26T04:13:44.664027+0000","last_peered":"2024-03-26T04:13:44.664027+0000","last_clean":"2024-03-26T04:13:44.664027+0000","last_became_active":"2024-03-26T04:12:15.670251+0000","last_became_peered":"2024-03-26T04:12:15.670251+0000","last_unstale":"2024-03-26T04:13:44.664027+0000","last_undegraded":"2024-03-26T04:13:44.664027+0000","last_fullsized":"2024-03-26T04:13:44.664027+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.053432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:35.104690+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.053432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:29.722406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"448'27","reported_seq":1196,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663770+0000","last_change":"2024-03-26T04:13:20.055428+0000","last_active":"2024-03-26T04:13:44.663770+0000","last_peered":"2024-03-26T04:13:44.663770+0000","last_clean":"2024-03-26T04:13:44.663770+0000","last_became_active":"2024-03-26T04:12:15.938622+0000","last_became_peered":"2024-03-26T04:12:15.938622+0000","last_unstale":"2024-03-26T04:13:44.663770+0000","last_undegraded":"2024-03-26T04:13:44.663770+0000","last_fullsized":"2024-03-26T04:13:44.663770+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'27","last_scrub_stamp":"2024-03-26T04:13:20.055317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:20.055317+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:46.920139+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665362+0000","last_change":"2024-03-26T04:13:31.111428+0000","last_active":"2024-03-26T04:13:44.665362+0000","last_peered":"2024-03-26T04:13:44.665362+0000","last_clean":"2024-03-26T04:13:44.665362+0000","last_became_active":"2024-03-26T04:12:21.682113+0000","last_became_peered":"2024-03-26T04:12:21.682113+0000","last_unstale":"2024-03-26T04:13:44.665362+0000","last_undegraded":"2024-03-26T04:13:44.665362+0000","last_fullsized":"2024-03-26T04:13:44.665362+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:31.111320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:58.704813+0000","last_clean_scrub_stamp":"2024-03-26T04:13:31.111320+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.860528+0000","scrub_duration":103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"548'14","reported_seq":158,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394775+0000","last_change":"2024-03-26T04:13:08.596778+0000","last_active":"2024-03-26T04:13:54.394775+0000","last_peered":"2024-03-26T04:13:54.394775+0000","last_clean":"2024-03-26T04:13:54.394775+0000","last_became_active":"2024-03-26T04:12:03.839909+0000","last_became_peered":"2024-03-26T04:12:03.839909+0000","last_unstale":"2024-03-26T04:13:54.394775+0000","last_undegraded":"2024-03-26T04:13:54.394775+0000","last_fullsized":"2024-03-26T04:13:54.394775+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"548'14","last_scrub_stamp":"2024-03-26T04:13:08.596685+0000","last_deep_scrub":"548'14","last_deep_scrub_stamp":"2024-03-26T04:13:08.596685+0000","last_clean_scrub_stamp":"2024-03-26T04:13:08.596685+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.264648+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048748+0000","last_change":"2024-03-26T04:13:43.051465+0000","last_active":"2024-03-26T04:13:45.048748+0000","last_peered":"2024-03-26T04:13:45.048748+0000","last_clean":"2024-03-26T04:13:45.048748+0000","last_became_active":"2024-03-26T04:12:08.685754+0000","last_became_peered":"2024-03-26T04:12:08.685754+0000","last_unstale":"2024-03-26T04:13:45.048748+0000","last_undegraded":"2024-03-26T04:13:45.048748+0000","last_fullsized":"2024-03-26T04:13:45.048748+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:43.051352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:43.051352+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.051352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.934527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"448'49","reported_seq":151,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666118+0000","last_change":"2024-03-26T04:13:29.690717+0000","last_active":"2024-03-26T04:13:44.666118+0000","last_peered":"2024-03-26T04:13:44.666118+0000","last_clean":"2024-03-26T04:13:44.666118+0000","last_became_active":"2024-03-26T04:12:29.658202+0000","last_became_peered":"2024-03-26T04:12:29.658202+0000","last_unstale":"2024-03-26T04:13:44.666118+0000","last_undegraded":"2024-03-26T04:13:44.666118+0000","last_fullsized":"2024-03-26T04:13:44.666118+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'49","last_scrub_stamp":"2024-03-26T04:13:29.690591+0000","last_deep_scrub":"422'39","last_deep_scrub_stamp":"2024-03-26T04:07:42.462212+0000","last_clean_scrub_stamp":"2024-03-26T04:13:29.690591+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.971046+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":53,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":1085,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664783+0000","last_change":"2024-03-26T04:13:04.226284+0000","last_active":"2024-03-26T04:13:44.664783+0000","last_peered":"2024-03-26T04:13:44.664783+0000","last_clean":"2024-03-26T04:13:44.664783+0000","last_became_active":"2024-03-26T04:12:03.856175+0000","last_became_peered":"2024-03-26T04:12:03.856175+0000","last_unstale":"2024-03-26T04:13:44.664783+0000","last_undegraded":"2024-03-26T04:13:44.664783+0000","last_fullsized":"2024-03-26T04:13:44.664783+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.226167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.226167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:09.764223+0000","scrub_duration":270,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"520'17","reported_seq":147,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:00.865896+0000","last_change":"2024-03-26T04:14:00.865922+0000","last_active":"2024-03-26T04:14:00.865896+0000","last_peered":"2024-03-26T04:14:00.865896+0000","last_clean":"2024-03-26T04:14:00.865896+0000","last_became_active":"2024-03-26T04:12:11.900998+0000","last_became_peered":"2024-03-26T04:12:11.900998+0000","last_unstale":"2024-03-26T04:14:00.865896+0000","last_undegraded":"2024-03-26T04:14:00.865896+0000","last_fullsized":"2024-03-26T04:14:00.865896+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'17","last_scrub_stamp":"2024-03-26T04:14:00.865833+0000","last_deep_scrub":"520'17","last_deep_scrub_stamp":"2024-03-26T04:12:46.959732+0000","last_clean_scrub_stamp":"2024-03-26T04:14:00.865833+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:20.982097+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"92'4","reported_seq":1185,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663314+0000","last_change":"2024-03-26T04:12:56.711797+0000","last_active":"2024-03-26T04:13:44.663314+0000","last_peered":"2024-03-26T04:13:44.663314+0000","last_clean":"2024-03-26T04:13:44.663314+0000","last_became_active":"2024-03-26T04:12:11.896040+0000","last_became_peered":"2024-03-26T04:12:11.896040+0000","last_unstale":"2024-03-26T04:13:44.663314+0000","last_undegraded":"2024-03-26T04:13:44.663314+0000","last_fullsized":"2024-03-26T04:13:44.663314+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'4","last_scrub_stamp":"2024-03-26T04:12:56.711651+0000","last_deep_scrub":"92'4","last_deep_scrub_stamp":"2024-03-26T04:04:17.538770+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.711651+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.974213+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1347,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":10776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"448'32","reported_seq":185,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.433171+0000","last_change":"2024-03-26T04:13:49.433220+0000","last_active":"2024-03-26T04:13:49.433171+0000","last_peered":"2024-03-26T04:13:49.433171+0000","last_clean":"2024-03-26T04:13:49.433171+0000","last_became_active":"2024-03-26T04:12:11.890563+0000","last_became_peered":"2024-03-26T04:12:11.890563+0000","last_unstale":"2024-03-26T04:13:49.433171+0000","last_undegraded":"2024-03-26T04:13:49.433171+0000","last_fullsized":"2024-03-26T04:13:49.433171+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'32","last_scrub_stamp":"2024-03-26T04:13:49.433092+0000","last_deep_scrub":"89'20","last_deep_scrub_stamp":"2024-03-26T03:53:27.547507+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.433092+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:58.996306+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"89'2","reported_seq":1039,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.897879+0000","last_change":"2024-03-26T04:13:21.739781+0000","last_active":"2024-03-26T04:13:44.897879+0000","last_peered":"2024-03-26T04:13:44.897879+0000","last_clean":"2024-03-26T04:13:44.897879+0000","last_became_active":"2024-03-26T04:12:08.696475+0000","last_became_peered":"2024-03-26T04:12:08.696475+0000","last_unstale":"2024-03-26T04:13:44.897879+0000","last_undegraded":"2024-03-26T04:13:44.897879+0000","last_fullsized":"2024-03-26T04:13:44.897879+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'2","last_scrub_stamp":"2024-03-26T04:13:21.739679+0000","last_deep_scrub":"89'2","last_deep_scrub_stamp":"2024-03-26T03:50:16.037958+0000","last_clean_scrub_stamp":"2024-03-26T04:13:21.739679+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:32.139054+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664741+0000","last_change":"2024-03-26T04:13:18.779649+0000","last_active":"2024-03-26T04:13:44.664741+0000","last_peered":"2024-03-26T04:13:44.664741+0000","last_clean":"2024-03-26T04:13:44.664741+0000","last_became_active":"2024-03-26T04:12:11.895264+0000","last_became_peered":"2024-03-26T04:12:11.895264+0000","last_unstale":"2024-03-26T04:13:44.664741+0000","last_undegraded":"2024-03-26T04:13:44.664741+0000","last_fullsized":"2024-03-26T04:13:44.664741+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:18.779519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:18.779519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:46.985322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":745,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049538+0000","last_change":"2024-03-26T04:13:41.037320+0000","last_active":"2024-03-26T04:13:45.049538+0000","last_peered":"2024-03-26T04:13:45.049538+0000","last_clean":"2024-03-26T04:13:45.049538+0000","last_became_active":"2024-03-26T04:13:06.665935+0000","last_became_peered":"2024-03-26T04:13:06.665935+0000","last_unstale":"2024-03-26T04:13:45.049538+0000","last_undegraded":"2024-03-26T04:13:45.049538+0000","last_fullsized":"2024-03-26T04:13:45.049538+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:41.037177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.037177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.548033+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"448'39","reported_seq":1304,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:51.106407+0000","last_change":"2024-03-26T04:13:51.106445+0000","last_active":"2024-03-26T04:13:51.106407+0000","last_peered":"2024-03-26T04:13:51.106407+0000","last_clean":"2024-03-26T04:13:51.106407+0000","last_became_active":"2024-03-26T04:12:11.896101+0000","last_became_peered":"2024-03-26T04:12:11.896101+0000","last_unstale":"2024-03-26T04:13:51.106407+0000","last_undegraded":"2024-03-26T04:13:51.106407+0000","last_fullsized":"2024-03-26T04:13:51.106407+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:51.106351+0000","last_deep_scrub":"422'33","last_deep_scrub_stamp":"2024-03-26T04:07:45.584130+0000","last_clean_scrub_stamp":"2024-03-26T04:13:51.106351+0000","objects_scrubbed":6,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:20.310413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049984+0000","last_change":"2024-03-26T04:12:27.181211+0000","last_active":"2024-03-26T04:13:45.049984+0000","last_peered":"2024-03-26T04:13:45.049984+0000","last_clean":"2024-03-26T04:13:45.049984+0000","last_became_active":"2024-03-26T04:12:15.655426+0000","last_became_peered":"2024-03-26T04:12:15.655426+0000","last_unstale":"2024-03-26T04:13:45.049984+0000","last_undegraded":"2024-03-26T04:13:45.049984+0000","last_fullsized":"2024-03-26T04:13:45.049984+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:27.181104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:27.181104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:52.875486+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200866+0000","last_change":"2024-03-26T04:13:14.579516+0000","last_active":"2024-03-26T04:13:45.200866+0000","last_peered":"2024-03-26T04:13:45.200866+0000","last_clean":"2024-03-26T04:13:45.200866+0000","last_became_active":"2024-03-26T04:12:11.888430+0000","last_became_peered":"2024-03-26T04:12:11.888430+0000","last_unstale":"2024-03-26T04:13:45.200866+0000","last_undegraded":"2024-03-26T04:13:45.200866+0000","last_fullsized":"2024-03-26T04:13:45.200866+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:14.579417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:10.223350+0000","last_clean_scrub_stamp":"2024-03-26T04:13:14.579417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:22.488941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":112,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.733684+0000","last_change":"2024-03-26T04:13:30.793991+0000","last_active":"2024-03-26T04:13:44.733684+0000","last_peered":"2024-03-26T04:13:44.733684+0000","last_clean":"2024-03-26T04:13:44.733684+0000","last_became_active":"2024-03-26T04:12:15.658406+0000","last_became_peered":"2024-03-26T04:12:15.658406+0000","last_unstale":"2024-03-26T04:13:44.733684+0000","last_undegraded":"2024-03-26T04:13:44.733684+0000","last_fullsized":"2024-03-26T04:13:44.733684+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:30.793880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:52.564972+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.793880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.042956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"565'29","reported_seq":135,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:56.854282+0000","last_change":"2024-03-26T04:13:56.854325+0000","last_active":"2024-03-26T04:13:56.854282+0000","last_peered":"2024-03-26T04:13:56.854282+0000","last_clean":"2024-03-26T04:13:56.854282+0000","last_became_active":"2024-03-26T04:12:11.894373+0000","last_became_peered":"2024-03-26T04:12:11.894373+0000","last_unstale":"2024-03-26T04:13:56.854282+0000","last_undegraded":"2024-03-26T04:13:56.854282+0000","last_fullsized":"2024-03-26T04:13:56.854282+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'29","last_scrub_stamp":"2024-03-26T04:13:56.854206+0000","last_deep_scrub":"565'29","last_deep_scrub_stamp":"2024-03-26T04:13:56.854206+0000","last_clean_scrub_stamp":"2024-03-26T04:13:56.854206+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:12.464099+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.585537+0000","last_change":"2024-03-26T04:13:58.585565+0000","last_active":"2024-03-26T04:13:58.585537+0000","last_peered":"2024-03-26T04:13:58.585537+0000","last_clean":"2024-03-26T04:13:58.585537+0000","last_became_active":"2024-03-26T04:12:15.667445+0000","last_became_peered":"2024-03-26T04:12:15.667445+0000","last_unstale":"2024-03-26T04:13:58.585537+0000","last_undegraded":"2024-03-26T04:13:58.585537+0000","last_fullsized":"2024-03-26T04:13:58.585537+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:58.585462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:39.714496+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.585462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:24.291714+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"446'4","reported_seq":132,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:51.104380+0000","last_change":"2024-03-26T04:13:51.104410+0000","last_active":"2024-03-26T04:13:51.104380+0000","last_peered":"2024-03-26T04:13:51.104380+0000","last_clean":"2024-03-26T04:13:51.104380+0000","last_became_active":"2024-03-26T04:12:08.682824+0000","last_became_peered":"2024-03-26T04:12:08.682824+0000","last_unstale":"2024-03-26T04:13:51.104380+0000","last_undegraded":"2024-03-26T04:13:51.104380+0000","last_fullsized":"2024-03-26T04:13:51.104380+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'4","last_scrub_stamp":"2024-03-26T04:13:51.104303+0000","last_deep_scrub":"446'4","last_deep_scrub_stamp":"2024-03-26T04:08:31.098553+0000","last_clean_scrub_stamp":"2024-03-26T04:13:51.104303+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:08.794984+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"448'51","reported_seq":209,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:01.575584+0000","last_change":"2024-03-26T04:14:01.575628+0000","last_active":"2024-03-26T04:14:01.575584+0000","last_peered":"2024-03-26T04:14:01.575584+0000","last_clean":"2024-03-26T04:14:01.575584+0000","last_became_active":"2024-03-26T04:12:15.664505+0000","last_became_peered":"2024-03-26T04:12:15.664505+0000","last_unstale":"2024-03-26T04:14:01.575584+0000","last_undegraded":"2024-03-26T04:14:01.575584+0000","last_fullsized":"2024-03-26T04:14:01.575584+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'51","last_scrub_stamp":"2024-03-26T04:14:01.575511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:14:01.575511+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:17.469647+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":67,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"563'17","reported_seq":175,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:59.395658+0000","last_change":"2024-03-26T04:13:40.076237+0000","last_active":"2024-03-26T04:13:59.395658+0000","last_peered":"2024-03-26T04:13:59.395658+0000","last_clean":"2024-03-26T04:13:59.395658+0000","last_became_active":"2024-03-26T04:12:15.655610+0000","last_became_peered":"2024-03-26T04:12:15.655610+0000","last_unstale":"2024-03-26T04:13:59.395658+0000","last_undegraded":"2024-03-26T04:13:59.395658+0000","last_fullsized":"2024-03-26T04:13:59.395658+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"563'17","last_scrub_stamp":"2024-03-26T04:13:40.076131+0000","last_deep_scrub":"563'17","last_deep_scrub_stamp":"2024-03-26T04:13:40.076131+0000","last_clean_scrub_stamp":"2024-03-26T04:13:40.076131+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:49.540190+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":121,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:47.913732+0000","last_change":"2024-03-26T04:13:47.913757+0000","last_active":"2024-03-26T04:13:47.913732+0000","last_peered":"2024-03-26T04:13:47.913732+0000","last_clean":"2024-03-26T04:13:47.913732+0000","last_became_active":"2024-03-26T04:12:11.896186+0000","last_became_peered":"2024-03-26T04:12:11.896186+0000","last_unstale":"2024-03-26T04:13:47.913732+0000","last_undegraded":"2024-03-26T04:13:47.913732+0000","last_fullsized":"2024-03-26T04:13:47.913732+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:47.913656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:02:52.851175+0000","last_clean_scrub_stamp":"2024-03-26T04:13:47.913656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:11.382588+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.048824+0000","last_change":"2024-03-26T04:12:47.843902+0000","last_active":"2024-03-26T04:13:45.048824+0000","last_peered":"2024-03-26T04:13:45.048824+0000","last_clean":"2024-03-26T04:13:45.048824+0000","last_became_active":"2024-03-26T04:12:08.688119+0000","last_became_peered":"2024-03-26T04:12:08.688119+0000","last_unstale":"2024-03-26T04:13:45.048824+0000","last_undegraded":"2024-03-26T04:13:45.048824+0000","last_fullsized":"2024-03-26T04:13:45.048824+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:47.843527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.104519+0000","last_clean_scrub_stamp":"2024-03-26T04:12:47.843527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:16.558614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"561'43","reported_seq":1232,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663943+0000","last_change":"2024-03-26T04:13:31.066703+0000","last_active":"2024-03-26T04:13:44.663943+0000","last_peered":"2024-03-26T04:13:44.663943+0000","last_clean":"2024-03-26T04:13:44.663943+0000","last_became_active":"2024-03-26T04:12:11.894713+0000","last_became_peered":"2024-03-26T04:12:11.894713+0000","last_unstale":"2024-03-26T04:13:44.663943+0000","last_undegraded":"2024-03-26T04:13:44.663943+0000","last_fullsized":"2024-03-26T04:13:44.663943+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"561'43","last_scrub_stamp":"2024-03-26T04:13:31.066515+0000","last_deep_scrub":"561'43","last_deep_scrub_stamp":"2024-03-26T04:13:31.066515+0000","last_clean_scrub_stamp":"2024-03-26T04:13:31.066515+0000","objects_scrubbed":7,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:31.620870+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":50,"num_write":38,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1001,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.897941+0000","last_change":"2024-03-26T04:13:35.920669+0000","last_active":"2024-03-26T04:13:44.897941+0000","last_peered":"2024-03-26T04:13:44.897941+0000","last_clean":"2024-03-26T04:13:44.897941+0000","last_became_active":"2024-03-26T04:05:29.533533+0000","last_became_peered":"2024-03-26T04:05:29.533533+0000","last_unstale":"2024-03-26T04:13:44.897941+0000","last_undegraded":"2024-03-26T04:13:44.897941+0000","last_fullsized":"2024-03-26T04:13:44.897941+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:35.920587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.920587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.727024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":915,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049144+0000","last_change":"2024-03-26T04:13:06.668482+0000","last_active":"2024-03-26T04:13:45.049144+0000","last_peered":"2024-03-26T04:13:45.049144+0000","last_clean":"2024-03-26T04:13:45.049144+0000","last_became_active":"2024-03-26T04:13:06.668137+0000","last_became_peered":"2024-03-26T04:13:06.668137+0000","last_unstale":"2024-03-26T04:13:45.049144+0000","last_undegraded":"2024-03-26T04:13:45.049144+0000","last_fullsized":"2024-03-26T04:13:45.049144+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:46.356757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:46.356757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.916940+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1020,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.872581+0000","last_change":"2024-03-26T04:13:46.872621+0000","last_active":"2024-03-26T04:13:46.872581+0000","last_peered":"2024-03-26T04:13:46.872581+0000","last_clean":"2024-03-26T04:13:46.872581+0000","last_became_active":"2024-03-26T04:12:03.866785+0000","last_became_peered":"2024-03-26T04:12:03.866785+0000","last_unstale":"2024-03-26T04:13:46.872581+0000","last_undegraded":"2024-03-26T04:13:46.872581+0000","last_fullsized":"2024-03-26T04:13:46.872581+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.872470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:46.872470+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.872470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:00.528375+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":99,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050190+0000","last_change":"2024-03-26T04:13:09.037100+0000","last_active":"2024-03-26T04:13:45.050190+0000","last_peered":"2024-03-26T04:13:45.050190+0000","last_clean":"2024-03-26T04:13:45.050190+0000","last_became_active":"2024-03-26T04:13:06.666989+0000","last_became_peered":"2024-03-26T04:13:06.666989+0000","last_unstale":"2024-03-26T04:13:45.050190+0000","last_undegraded":"2024-03-26T04:13:45.050190+0000","last_fullsized":"2024-03-26T04:13:45.050190+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:09.036935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:09.036935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:20.043134+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"448'67","reported_seq":126,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734119+0000","last_change":"2024-03-26T04:12:49.091599+0000","last_active":"2024-03-26T04:13:44.734119+0000","last_peered":"2024-03-26T04:13:44.734119+0000","last_clean":"2024-03-26T04:13:44.734119+0000","last_became_active":"2024-03-26T04:12:15.668519+0000","last_became_peered":"2024-03-26T04:12:15.668519+0000","last_unstale":"2024-03-26T04:13:44.734119+0000","last_undegraded":"2024-03-26T04:13:44.734119+0000","last_fullsized":"2024-03-26T04:13:44.734119+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'67","last_scrub_stamp":"2024-03-26T04:12:49.091464+0000","last_deep_scrub":"356'51","last_deep_scrub_stamp":"2024-03-26T04:04:03.371335+0000","last_clean_scrub_stamp":"2024-03-26T04:12:49.091464+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.600520+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":88,"num_read_kb":76,"num_write":58,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1085,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.129345+0000","last_change":"2024-03-26T04:13:58.129374+0000","last_active":"2024-03-26T04:13:58.129345+0000","last_peered":"2024-03-26T04:13:58.129345+0000","last_clean":"2024-03-26T04:13:58.129345+0000","last_became_active":"2024-03-26T04:12:08.677824+0000","last_became_peered":"2024-03-26T04:12:08.677824+0000","last_unstale":"2024-03-26T04:13:58.129345+0000","last_undegraded":"2024-03-26T04:13:58.129345+0000","last_fullsized":"2024-03-26T04:13:58.129345+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:58.129266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.129266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:02.360285+0000","scrub_duration":192,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1016,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898324+0000","last_change":"2024-03-26T04:13:24.768873+0000","last_active":"2024-03-26T04:13:44.898324+0000","last_peered":"2024-03-26T04:13:44.898324+0000","last_clean":"2024-03-26T04:13:44.898324+0000","last_became_active":"2024-03-26T04:12:11.880288+0000","last_became_peered":"2024-03-26T04:12:11.880288+0000","last_unstale":"2024-03-26T04:13:44.898324+0000","last_undegraded":"2024-03-26T04:13:44.898324+0000","last_fullsized":"2024-03-26T04:13:44.898324+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.768785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:59.255770+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.768785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:29.244873+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":149,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200739+0000","last_change":"2024-03-26T04:13:26.515548+0000","last_active":"2024-03-26T04:13:45.200739+0000","last_peered":"2024-03-26T04:13:45.200739+0000","last_clean":"2024-03-26T04:13:45.200739+0000","last_became_active":"2024-03-26T04:12:15.663290+0000","last_became_peered":"2024-03-26T04:12:15.663290+0000","last_unstale":"2024-03-26T04:13:45.200739+0000","last_undegraded":"2024-03-26T04:13:45.200739+0000","last_fullsized":"2024-03-26T04:13:45.200739+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:26.515446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:57.098828+0000","last_clean_scrub_stamp":"2024-03-26T04:13:26.515446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:45.969628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"448'27","reported_seq":1189,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665339+0000","last_change":"2024-03-26T04:13:28.213776+0000","last_active":"2024-03-26T04:13:44.665339+0000","last_peered":"2024-03-26T04:13:44.665339+0000","last_clean":"2024-03-26T04:13:44.665339+0000","last_became_active":"2024-03-26T04:13:28.207633+0000","last_became_peered":"2024-03-26T04:13:28.207633+0000","last_unstale":"2024-03-26T04:13:44.665339+0000","last_undegraded":"2024-03-26T04:13:44.665339+0000","last_fullsized":"2024-03-26T04:13:44.665339+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'27","last_scrub_stamp":"2024-03-26T04:13:25.942590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.942590+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.395509+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"79'128","reported_seq":897,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200504+0000","last_change":"2024-03-26T04:13:30.506773+0000","last_active":"2024-03-26T04:13:45.200504+0000","last_peered":"2024-03-26T04:13:45.200504+0000","last_clean":"2024-03-26T04:13:45.200504+0000","last_became_active":"2024-03-26T04:12:06.659993+0000","last_became_peered":"2024-03-26T04:12:06.659993+0000","last_unstale":"2024-03-26T04:13:45.200504+0000","last_undegraded":"2024-03-26T04:13:45.200504+0000","last_fullsized":"2024-03-26T04:13:45.200504+0000","mapping_epoch":512,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":513,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:30.506668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.506668+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.573206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666062+0000","last_change":"2024-03-26T04:13:13.868069+0000","last_active":"2024-03-26T04:13:44.666062+0000","last_peered":"2024-03-26T04:13:44.666062+0000","last_clean":"2024-03-26T04:13:44.666062+0000","last_became_active":"2024-03-26T04:12:11.894563+0000","last_became_peered":"2024-03-26T04:12:11.894563+0000","last_unstale":"2024-03-26T04:13:44.666062+0000","last_undegraded":"2024-03-26T04:13:44.666062+0000","last_fullsized":"2024-03-26T04:13:44.666062+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:13.867940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:13.867940+0000","last_clean_scrub_stamp":"2024-03-26T04:13:13.867940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:17.782005+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"256'3","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200447+0000","last_change":"2024-03-26T04:13:16.532509+0000","last_active":"2024-03-26T04:13:45.200447+0000","last_peered":"2024-03-26T04:13:45.200447+0000","last_clean":"2024-03-26T04:13:45.200447+0000","last_became_active":"2024-03-26T04:12:15.667426+0000","last_became_peered":"2024-03-26T04:12:15.667426+0000","last_unstale":"2024-03-26T04:13:45.200447+0000","last_undegraded":"2024-03-26T04:13:45.200447+0000","last_fullsized":"2024-03-26T04:13:45.200447+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'3","last_scrub_stamp":"2024-03-26T04:13:16.532403+0000","last_deep_scrub":"256'3","last_deep_scrub_stamp":"2024-03-26T03:59:27.539528+0000","last_clean_scrub_stamp":"2024-03-26T04:13:16.532403+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.591110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00068707399999999997,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":1089,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664249+0000","last_change":"2024-03-26T04:13:43.062614+0000","last_active":"2024-03-26T04:13:44.664249+0000","last_peered":"2024-03-26T04:13:44.664249+0000","last_clean":"2024-03-26T04:13:44.664249+0000","last_became_active":"2024-03-26T04:12:15.673049+0000","last_became_peered":"2024-03-26T04:12:15.673049+0000","last_unstale":"2024-03-26T04:13:44.664249+0000","last_undegraded":"2024-03-26T04:13:44.664249+0000","last_fullsized":"2024-03-26T04:13:44.664249+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:43.062525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:50:22.903348+0000","last_clean_scrub_stamp":"2024-03-26T04:13:43.062525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.018180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"520'67","reported_seq":1700,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394563+0000","last_change":"2024-03-26T04:13:39.448800+0000","last_active":"2024-03-26T04:13:54.394563+0000","last_peered":"2024-03-26T04:13:54.394563+0000","last_clean":"2024-03-26T04:13:54.394563+0000","last_became_active":"2024-03-26T04:12:11.893198+0000","last_became_peered":"2024-03-26T04:12:11.893198+0000","last_unstale":"2024-03-26T04:13:54.394563+0000","last_undegraded":"2024-03-26T04:13:54.394563+0000","last_fullsized":"2024-03-26T04:13:54.394563+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'67","last_scrub_stamp":"2024-03-26T04:13:39.448703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:39.448703+0000","objects_scrubbed":11,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:01.740182+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":93,"num_read_kb":77,"num_write":56,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":2506,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664638+0000","last_change":"2024-03-26T04:13:27.898799+0000","last_active":"2024-03-26T04:13:44.664638+0000","last_peered":"2024-03-26T04:13:44.664638+0000","last_clean":"2024-03-26T04:13:44.664638+0000","last_became_active":"2024-03-26T04:12:21.684416+0000","last_became_peered":"2024-03-26T04:12:21.684416+0000","last_unstale":"2024-03-26T04:13:44.664638+0000","last_undegraded":"2024-03-26T04:13:44.664638+0000","last_fullsized":"2024-03-26T04:13:44.664638+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.898720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.898720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:52.356266+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"79'128","reported_seq":864,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049656+0000","last_change":"2024-03-26T04:13:06.669980+0000","last_active":"2024-03-26T04:13:45.049656+0000","last_peered":"2024-03-26T04:13:45.049656+0000","last_clean":"2024-03-26T04:13:45.049656+0000","last_became_active":"2024-03-26T04:13:06.669607+0000","last_became_peered":"2024-03-26T04:13:06.669607+0000","last_unstale":"2024-03-26T04:13:45.049656+0000","last_undegraded":"2024-03-26T04:13:45.049656+0000","last_fullsized":"2024-03-26T04:13:45.049656+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:52.733697+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:34.281069+0000","last_clean_scrub_stamp":"2024-03-26T04:12:52.733697+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:13.030575+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.050054+0000","last_change":"2024-03-26T04:13:35.146329+0000","last_active":"2024-03-26T04:13:45.050054+0000","last_peered":"2024-03-26T04:13:45.050054+0000","last_clean":"2024-03-26T04:13:45.050054+0000","last_became_active":"2024-03-26T04:12:15.654522+0000","last_became_peered":"2024-03-26T04:12:15.654522+0000","last_unstale":"2024-03-26T04:13:45.050054+0000","last_undegraded":"2024-03-26T04:13:45.050054+0000","last_fullsized":"2024-03-26T04:13:45.050054+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:35.146222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:07:38.023066+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.146222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:47.967411+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":1111,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.611205+0000","last_change":"2024-03-26T04:13:57.611240+0000","last_active":"2024-03-26T04:13:57.611205+0000","last_peered":"2024-03-26T04:13:57.611205+0000","last_clean":"2024-03-26T04:13:57.611205+0000","last_became_active":"2024-03-26T04:12:15.670120+0000","last_became_peered":"2024-03-26T04:12:15.670120+0000","last_unstale":"2024-03-26T04:13:57.611205+0000","last_undegraded":"2024-03-26T04:13:57.611205+0000","last_fullsized":"2024-03-26T04:13:57.611205+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.611137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.611137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:19.664946+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1034,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.898137+0000","last_change":"2024-03-26T04:13:42.909700+0000","last_active":"2024-03-26T04:13:44.898137+0000","last_peered":"2024-03-26T04:13:44.898137+0000","last_clean":"2024-03-26T04:13:44.898137+0000","last_became_active":"2024-03-26T04:12:11.879075+0000","last_became_peered":"2024-03-26T04:12:11.879075+0000","last_unstale":"2024-03-26T04:13:44.898137+0000","last_undegraded":"2024-03-26T04:13:44.898137+0000","last_fullsized":"2024-03-26T04:13:44.898137+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:42.909590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:25.140185+0000","last_clean_scrub_stamp":"2024-03-26T04:13:42.909590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:07.684065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023469239999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"448'33","reported_seq":97,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:47.449302+0000","last_change":"2024-03-26T04:13:47.449340+0000","last_active":"2024-03-26T04:13:47.449302+0000","last_peered":"2024-03-26T04:13:47.449302+0000","last_clean":"2024-03-26T04:13:47.449302+0000","last_became_active":"2024-03-26T04:12:29.656239+0000","last_became_peered":"2024-03-26T04:12:29.656239+0000","last_unstale":"2024-03-26T04:13:47.449302+0000","last_undegraded":"2024-03-26T04:13:47.449302+0000","last_fullsized":"2024-03-26T04:13:47.449302+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'33","last_scrub_stamp":"2024-03-26T04:13:47.449223+0000","last_deep_scrub":"371'29","last_deep_scrub_stamp":"2024-03-26T04:04:59.217868+0000","last_clean_scrub_stamp":"2024-03-26T04:13:47.449223+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:13.029397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":50,"num_read_kb":38,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1039,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.816449+0000","last_change":"2024-03-26T04:13:49.816472+0000","last_active":"2024-03-26T04:13:49.816449+0000","last_peered":"2024-03-26T04:13:49.816449+0000","last_clean":"2024-03-26T04:13:49.816449+0000","last_became_active":"2024-03-26T04:12:08.690699+0000","last_became_peered":"2024-03-26T04:12:08.690699+0000","last_unstale":"2024-03-26T04:13:49.816449+0000","last_undegraded":"2024-03-26T04:13:49.816449+0000","last_fullsized":"2024-03-26T04:13:49.816449+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:49.816391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.816391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.765706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201163+0000","last_change":"2024-03-26T04:13:04.711186+0000","last_active":"2024-03-26T04:13:45.201163+0000","last_peered":"2024-03-26T04:13:45.201163+0000","last_clean":"2024-03-26T04:13:45.201163+0000","last_became_active":"2024-03-26T04:12:11.888053+0000","last_became_peered":"2024-03-26T04:12:11.888053+0000","last_unstale":"2024-03-26T04:13:45.201163+0000","last_undegraded":"2024-03-26T04:13:45.201163+0000","last_fullsized":"2024-03-26T04:13:45.201163+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.711083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:02.403784+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.711083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.655476+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"79'128","reported_seq":565,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049201+0000","last_change":"2024-03-26T04:13:28.086428+0000","last_active":"2024-03-26T04:13:45.049201+0000","last_peered":"2024-03-26T04:13:45.049201+0000","last_clean":"2024-03-26T04:13:45.049201+0000","last_became_active":"2024-03-26T04:12:18.665269+0000","last_became_peered":"2024-03-26T04:12:18.665269+0000","last_unstale":"2024-03-26T04:13:45.049201+0000","last_undegraded":"2024-03-26T04:13:45.049201+0000","last_fullsized":"2024-03-26T04:13:45.049201+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:28.086295+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:10:33.403777+0000","last_clean_scrub_stamp":"2024-03-26T04:13:28.086295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.249806+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":127,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:46.733685+0000","last_change":"2024-03-26T04:13:46.733706+0000","last_active":"2024-03-26T04:13:46.733685+0000","last_peered":"2024-03-26T04:13:46.733685+0000","last_clean":"2024-03-26T04:13:46.733685+0000","last_became_active":"2024-03-26T04:12:15.656802+0000","last_became_peered":"2024-03-26T04:12:15.656802+0000","last_unstale":"2024-03-26T04:13:46.733685+0000","last_undegraded":"2024-03-26T04:13:46.733685+0000","last_fullsized":"2024-03-26T04:13:46.733685+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:46.733623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:46.733623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.849645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"525'45","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394749+0000","last_change":"2024-03-26T04:13:04.100203+0000","last_active":"2024-03-26T04:13:54.394749+0000","last_peered":"2024-03-26T04:13:54.394749+0000","last_clean":"2024-03-26T04:13:54.394749+0000","last_became_active":"2024-03-26T04:12:15.665471+0000","last_became_peered":"2024-03-26T04:12:15.665471+0000","last_unstale":"2024-03-26T04:13:54.394749+0000","last_undegraded":"2024-03-26T04:13:54.394749+0000","last_fullsized":"2024-03-26T04:13:54.394749+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'45","last_scrub_stamp":"2024-03-26T04:13:04.100076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.100076+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.796896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":59,"num_read_kb":49,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":858,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049900+0000","last_change":"2024-03-26T04:13:23.013833+0000","last_active":"2024-03-26T04:13:45.049900+0000","last_peered":"2024-03-26T04:13:45.049900+0000","last_clean":"2024-03-26T04:13:45.049900+0000","last_became_active":"2024-03-26T04:13:06.669225+0000","last_became_peered":"2024-03-26T04:13:06.669225+0000","last_unstale":"2024-03-26T04:13:45.049900+0000","last_undegraded":"2024-03-26T04:13:45.049900+0000","last_fullsized":"2024-03-26T04:13:45.049900+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:23.013702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:22.075168+0000","last_clean_scrub_stamp":"2024-03-26T04:13:23.013702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:42.886857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"79'128","reported_seq":998,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:55.566341+0000","last_change":"2024-03-26T04:13:55.566379+0000","last_active":"2024-03-26T04:13:55.566341+0000","last_peered":"2024-03-26T04:13:55.566341+0000","last_clean":"2024-03-26T04:13:55.566341+0000","last_became_active":"2024-03-26T04:12:18.657768+0000","last_became_peered":"2024-03-26T04:12:18.657768+0000","last_unstale":"2024-03-26T04:13:55.566341+0000","last_undegraded":"2024-03-26T04:13:55.566341+0000","last_fullsized":"2024-03-26T04:13:55.566341+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:55.566263+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:12:39.440775+0000","last_clean_scrub_stamp":"2024-03-26T04:13:55.566263+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:56.515493+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":83,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:53.509387+0000","last_change":"2024-03-26T04:13:53.509408+0000","last_active":"2024-03-26T04:13:53.509387+0000","last_peered":"2024-03-26T04:13:53.509387+0000","last_clean":"2024-03-26T04:13:53.509387+0000","last_became_active":"2024-03-26T04:12:29.661693+0000","last_became_peered":"2024-03-26T04:12:29.661693+0000","last_unstale":"2024-03-26T04:13:53.509387+0000","last_undegraded":"2024-03-26T04:13:53.509387+0000","last_fullsized":"2024-03-26T04:13:53.509387+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:53.509326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:53.509326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:59.077901+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1138,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663781+0000","last_change":"2024-03-26T04:13:41.153795+0000","last_active":"2024-03-26T04:13:44.663781+0000","last_peered":"2024-03-26T04:13:44.663781+0000","last_clean":"2024-03-26T04:13:44.663781+0000","last_became_active":"2024-03-26T04:12:11.895940+0000","last_became_peered":"2024-03-26T04:12:11.895940+0000","last_unstale":"2024-03-26T04:13:44.663781+0000","last_undegraded":"2024-03-26T04:13:44.663781+0000","last_fullsized":"2024-03-26T04:13:44.663781+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:41.153712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:41.153712+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.153712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:51.510402+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"565'6","reported_seq":247,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:02.878504+0000","last_change":"2024-03-26T04:13:55.135310+0000","last_active":"2024-03-26T04:14:02.878504+0000","last_peered":"2024-03-26T04:14:02.878504+0000","last_clean":"2024-03-26T04:14:02.878504+0000","last_became_active":"2024-03-26T04:12:08.690101+0000","last_became_peered":"2024-03-26T04:12:08.690101+0000","last_unstale":"2024-03-26T04:14:02.878504+0000","last_undegraded":"2024-03-26T04:14:02.878504+0000","last_fullsized":"2024-03-26T04:14:02.878504+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'6","last_scrub_stamp":"2024-03-26T04:13:55.135201+0000","last_deep_scrub":"565'6","last_deep_scrub_stamp":"2024-03-26T04:13:55.135201+0000","last_clean_scrub_stamp":"2024-03-26T04:13:55.135201+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:15.797200+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.001082256,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"448'42","reported_seq":109,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:49.088052+0000","last_change":"2024-03-26T04:13:49.088102+0000","last_active":"2024-03-26T04:13:49.088052+0000","last_peered":"2024-03-26T04:13:49.088052+0000","last_clean":"2024-03-26T04:13:49.088052+0000","last_became_active":"2024-03-26T04:12:21.685659+0000","last_became_peered":"2024-03-26T04:12:21.685659+0000","last_unstale":"2024-03-26T04:13:49.088052+0000","last_undegraded":"2024-03-26T04:13:49.088052+0000","last_fullsized":"2024-03-26T04:13:49.088052+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'42","last_scrub_stamp":"2024-03-26T04:13:49.087962+0000","last_deep_scrub":"422'30","last_deep_scrub_stamp":"2024-03-26T04:07:49.546733+0000","last_clean_scrub_stamp":"2024-03-26T04:13:49.087962+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:08.277379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":137,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201234+0000","last_change":"2024-03-26T04:13:22.598273+0000","last_active":"2024-03-26T04:13:45.201234+0000","last_peered":"2024-03-26T04:13:45.201234+0000","last_clean":"2024-03-26T04:13:45.201234+0000","last_became_active":"2024-03-26T04:12:03.846590+0000","last_became_peered":"2024-03-26T04:12:03.846590+0000","last_unstale":"2024-03-26T04:13:45.201234+0000","last_undegraded":"2024-03-26T04:13:45.201234+0000","last_fullsized":"2024-03-26T04:13:45.201234+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:05:01.160427+0000","last_clean_scrub_stamp":"2024-03-26T04:13:22.598174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:37.397639+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"79'128","reported_seq":944,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734438+0000","last_change":"2024-03-26T04:13:41.829833+0000","last_active":"2024-03-26T04:13:44.734438+0000","last_peered":"2024-03-26T04:13:44.734438+0000","last_clean":"2024-03-26T04:13:44.734438+0000","last_became_active":"2024-03-26T04:12:18.661189+0000","last_became_peered":"2024-03-26T04:12:18.661189+0000","last_unstale":"2024-03-26T04:13:44.734438+0000","last_undegraded":"2024-03-26T04:13:44.734438+0000","last_fullsized":"2024-03-26T04:13:44.734438+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:41.829728+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:12:19.954807+0000","last_clean_scrub_stamp":"2024-03-26T04:13:41.829728+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.400447+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1103,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725449+0000","last_change":"2024-03-26T04:13:25.547378+0000","last_active":"2024-03-26T04:13:44.725449+0000","last_peered":"2024-03-26T04:13:44.725449+0000","last_clean":"2024-03-26T04:13:44.725449+0000","last_became_active":"2024-03-26T04:12:15.675671+0000","last_became_peered":"2024-03-26T04:12:15.675671+0000","last_unstale":"2024-03-26T04:13:44.725449+0000","last_undegraded":"2024-03-26T04:13:44.725449+0000","last_fullsized":"2024-03-26T04:13:44.725449+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:25.547273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:25.547273+0000","last_clean_scrub_stamp":"2024-03-26T04:13:25.547273+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:39.656810+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":1112,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.726175+0000","last_change":"2024-03-26T04:13:30.590127+0000","last_active":"2024-03-26T04:13:44.726175+0000","last_peered":"2024-03-26T04:13:44.726175+0000","last_clean":"2024-03-26T04:13:44.726175+0000","last_became_active":"2024-03-26T04:12:03.859048+0000","last_became_peered":"2024-03-26T04:12:03.859048+0000","last_unstale":"2024-03-26T04:13:44.726175+0000","last_undegraded":"2024-03-26T04:13:44.726175+0000","last_fullsized":"2024-03-26T04:13:44.726175+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:30.590005+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:30.590005+0000","last_clean_scrub_stamp":"2024-03-26T04:13:30.590005+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:55.749006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"448'54","reported_seq":1307,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665546+0000","last_change":"2024-03-26T04:13:19.960173+0000","last_active":"2024-03-26T04:13:44.665546+0000","last_peered":"2024-03-26T04:13:44.665546+0000","last_clean":"2024-03-26T04:13:44.665546+0000","last_became_active":"2024-03-26T04:12:08.678078+0000","last_became_peered":"2024-03-26T04:12:08.678078+0000","last_unstale":"2024-03-26T04:13:44.665546+0000","last_undegraded":"2024-03-26T04:13:44.665546+0000","last_fullsized":"2024-03-26T04:13:44.665546+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'54","last_scrub_stamp":"2024-03-26T04:13:19.960073+0000","last_deep_scrub":"448'54","last_deep_scrub_stamp":"2024-03-26T04:11:45.215033+0000","last_clean_scrub_stamp":"2024-03-26T04:13:19.960073+0000","objects_scrubbed":9,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:23.105238+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":78,"num_read_kb":63,"num_write":48,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":1650,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049530+0000","last_change":"2024-03-26T04:13:06.952445+0000","last_active":"2024-03-26T04:13:45.049530+0000","last_peered":"2024-03-26T04:13:45.049530+0000","last_clean":"2024-03-26T04:13:45.049530+0000","last_became_active":"2024-03-26T04:13:06.666213+0000","last_became_peered":"2024-03-26T04:13:06.666213+0000","last_unstale":"2024-03-26T04:13:45.049530+0000","last_undegraded":"2024-03-26T04:13:45.049530+0000","last_fullsized":"2024-03-26T04:13:45.049530+0000","mapping_epoch":546,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":547,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:06.952348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:34.576640+0000","last_clean_scrub_stamp":"2024-03-26T04:13:06.952348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.925261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"79'128","reported_seq":779,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734270+0000","last_change":"2024-03-26T04:12:57.109436+0000","last_active":"2024-03-26T04:13:44.734270+0000","last_peered":"2024-03-26T04:13:44.734270+0000","last_clean":"2024-03-26T04:13:44.734270+0000","last_became_active":"2024-03-26T04:12:18.662592+0000","last_became_peered":"2024-03-26T04:12:18.662592+0000","last_unstale":"2024-03-26T04:13:44.734270+0000","last_undegraded":"2024-03-26T04:13:44.734270+0000","last_fullsized":"2024-03-26T04:13:44.734270+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:57.109323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.109323+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.603411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1120,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664118+0000","last_change":"2024-03-26T04:13:36.159194+0000","last_active":"2024-03-26T04:13:44.664118+0000","last_peered":"2024-03-26T04:13:44.664118+0000","last_clean":"2024-03-26T04:13:44.664118+0000","last_became_active":"2024-03-26T04:12:11.903366+0000","last_became_peered":"2024-03-26T04:12:11.903366+0000","last_unstale":"2024-03-26T04:13:44.664118+0000","last_undegraded":"2024-03-26T04:13:44.664118+0000","last_fullsized":"2024-03-26T04:13:44.664118+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:36.159096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:13:36.159096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.574402+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":141,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201209+0000","last_change":"2024-03-26T04:12:56.581058+0000","last_active":"2024-03-26T04:13:45.201209+0000","last_peered":"2024-03-26T04:13:45.201209+0000","last_clean":"2024-03-26T04:13:45.201209+0000","last_became_active":"2024-03-26T04:12:11.888081+0000","last_became_peered":"2024-03-26T04:12:11.888081+0000","last_unstale":"2024-03-26T04:13:45.201209+0000","last_undegraded":"2024-03-26T04:13:45.201209+0000","last_fullsized":"2024-03-26T04:13:45.201209+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.580954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.580954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.537483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"448'36","reported_seq":105,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:14:01.684796+0000","last_change":"2024-03-26T04:14:01.684834+0000","last_active":"2024-03-26T04:14:01.684796+0000","last_peered":"2024-03-26T04:14:01.684796+0000","last_clean":"2024-03-26T04:14:01.684796+0000","last_became_active":"2024-03-26T04:12:29.656147+0000","last_became_peered":"2024-03-26T04:12:29.656147+0000","last_unstale":"2024-03-26T04:14:01.684796+0000","last_undegraded":"2024-03-26T04:14:01.684796+0000","last_fullsized":"2024-03-26T04:14:01.684796+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'36","last_scrub_stamp":"2024-03-26T04:14:01.684717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:14:01.684717+0000","objects_scrubbed":6,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:26.607349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"525'21","reported_seq":1505,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394483+0000","last_change":"2024-03-26T04:13:53.036323+0000","last_active":"2024-03-26T04:13:54.394483+0000","last_peered":"2024-03-26T04:13:54.394483+0000","last_clean":"2024-03-26T04:13:54.394483+0000","last_became_active":"2024-03-26T04:12:15.654133+0000","last_became_peered":"2024-03-26T04:12:15.654133+0000","last_unstale":"2024-03-26T04:13:54.394483+0000","last_undegraded":"2024-03-26T04:13:54.394483+0000","last_fullsized":"2024-03-26T04:13:54.394483+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"525'21","last_scrub_stamp":"2024-03-26T04:13:53.036229+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:13:53.036229+0000","objects_scrubbed":1,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:54.858637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"79'128","reported_seq":594,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.845962+0000","last_change":"2024-03-26T04:13:54.845990+0000","last_active":"2024-03-26T04:13:54.845962+0000","last_peered":"2024-03-26T04:13:54.845962+0000","last_clean":"2024-03-26T04:13:54.845962+0000","last_became_active":"2024-03-26T04:12:14.663450+0000","last_became_peered":"2024-03-26T04:12:14.663450+0000","last_unstale":"2024-03-26T04:13:54.845962+0000","last_undegraded":"2024-03-26T04:13:54.845962+0000","last_fullsized":"2024-03-26T04:13:54.845962+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:54.845896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:13:54.845896+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:04.640626+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":655360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":124,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049674+0000","last_change":"2024-03-26T04:13:04.089423+0000","last_active":"2024-03-26T04:13:45.049674+0000","last_peered":"2024-03-26T04:13:45.049674+0000","last_clean":"2024-03-26T04:13:45.049674+0000","last_became_active":"2024-03-26T04:12:15.654149+0000","last_became_peered":"2024-03-26T04:12:15.654149+0000","last_unstale":"2024-03-26T04:13:45.049674+0000","last_undegraded":"2024-03-26T04:13:45.049674+0000","last_fullsized":"2024-03-26T04:13:45.049674+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:04.089322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:13:04.089322+0000","last_clean_scrub_stamp":"2024-03-26T04:13:04.089322+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.627171+0000","scrub_duration":147,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"292'2","reported_seq":1086,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665965+0000","last_change":"2024-03-26T04:13:28.942893+0000","last_active":"2024-03-26T04:13:44.665965+0000","last_peered":"2024-03-26T04:13:44.665965+0000","last_clean":"2024-03-26T04:13:44.665965+0000","last_became_active":"2024-03-26T03:48:44.010661+0000","last_became_peered":"2024-03-26T03:48:44.010661+0000","last_unstale":"2024-03-26T04:13:44.665965+0000","last_undegraded":"2024-03-26T04:13:44.665965+0000","last_fullsized":"2024-03-26T04:13:44.665965+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"292'2","last_scrub_stamp":"2024-03-26T04:13:28.942794+0000","last_deep_scrub":"292'2","last_deep_scrub_stamp":"2024-03-26T04:01:34.494617+0000","last_clean_scrub_stamp":"2024-03-26T04:13:28.942794+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.356439+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"448'48","reported_seq":1277,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663956+0000","last_change":"2024-03-26T04:13:29.003162+0000","last_active":"2024-03-26T04:13:44.663956+0000","last_peered":"2024-03-26T04:13:44.663956+0000","last_clean":"2024-03-26T04:13:44.663956+0000","last_became_active":"2024-03-26T04:12:15.668826+0000","last_became_peered":"2024-03-26T04:12:15.668826+0000","last_unstale":"2024-03-26T04:13:44.663956+0000","last_undegraded":"2024-03-26T04:13:44.663956+0000","last_fullsized":"2024-03-26T04:13:44.663956+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'48","last_scrub_stamp":"2024-03-26T04:13:29.003026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:29.003026+0000","objects_scrubbed":8,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:57.528033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":72,"num_read_kb":64,"num_write":48,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"520'14","reported_seq":139,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394459+0000","last_change":"2024-03-26T04:13:06.899411+0000","last_active":"2024-03-26T04:13:54.394459+0000","last_peered":"2024-03-26T04:13:54.394459+0000","last_clean":"2024-03-26T04:13:54.394459+0000","last_became_active":"2024-03-26T04:12:11.900417+0000","last_became_peered":"2024-03-26T04:12:11.900417+0000","last_unstale":"2024-03-26T04:13:54.394459+0000","last_undegraded":"2024-03-26T04:13:54.394459+0000","last_fullsized":"2024-03-26T04:13:54.394459+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"520'14","last_scrub_stamp":"2024-03-26T04:13:06.899326+0000","last_deep_scrub":"154'10","last_deep_scrub_stamp":"2024-03-26T03:54:34.943428+0000","last_clean_scrub_stamp":"2024-03-26T04:13:06.899326+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:08.668844+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"79'128","reported_seq":942,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664742+0000","last_change":"2024-03-26T04:12:54.914516+0000","last_active":"2024-03-26T04:13:44.664742+0000","last_peered":"2024-03-26T04:13:44.664742+0000","last_clean":"2024-03-26T04:13:44.664742+0000","last_became_active":"2024-03-26T04:12:18.661143+0000","last_became_peered":"2024-03-26T04:12:18.661143+0000","last_unstale":"2024-03-26T04:13:44.664742+0000","last_undegraded":"2024-03-26T04:13:44.664742+0000","last_fullsized":"2024-03-26T04:13:44.664742+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":3,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:54.914405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:54.914405+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:01.121205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":1100,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664522+0000","last_change":"2024-03-26T04:12:31.998898+0000","last_active":"2024-03-26T04:13:44.664522+0000","last_peered":"2024-03-26T04:13:44.664522+0000","last_clean":"2024-03-26T04:13:44.664522+0000","last_became_active":"2024-03-26T04:12:11.882387+0000","last_became_peered":"2024-03-26T04:12:11.882387+0000","last_unstale":"2024-03-26T04:13:44.664522+0000","last_undegraded":"2024-03-26T04:13:44.664522+0000","last_fullsized":"2024-03-26T04:13:44.664522+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:31.998779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:49.010280+0000","last_clean_scrub_stamp":"2024-03-26T04:12:31.998779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:54.836641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665007+0000","last_change":"2024-03-26T04:12:58.920908+0000","last_active":"2024-03-26T04:13:44.665007+0000","last_peered":"2024-03-26T04:13:44.665007+0000","last_clean":"2024-03-26T04:13:44.665007+0000","last_became_active":"2024-03-26T04:12:11.899519+0000","last_became_peered":"2024-03-26T04:12:11.899519+0000","last_unstale":"2024-03-26T04:13:44.665007+0000","last_undegraded":"2024-03-26T04:13:44.665007+0000","last_fullsized":"2024-03-26T04:13:44.665007+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:58.920789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:12:58.920789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:13.165271+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"448'64","reported_seq":171,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664095+0000","last_change":"2024-03-26T04:13:37.607925+0000","last_active":"2024-03-26T04:13:44.664095+0000","last_peered":"2024-03-26T04:13:44.664095+0000","last_clean":"2024-03-26T04:13:44.664095+0000","last_became_active":"2024-03-26T04:12:21.683241+0000","last_became_peered":"2024-03-26T04:12:21.683241+0000","last_unstale":"2024-03-26T04:13:44.664095+0000","last_undegraded":"2024-03-26T04:13:44.664095+0000","last_fullsized":"2024-03-26T04:13:44.664095+0000","mapping_epoch":526,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":527,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'64","last_scrub_stamp":"2024-03-26T04:13:37.607799+0000","last_deep_scrub":"448'64","last_deep_scrub_stamp":"2024-03-26T04:12:22.799227+0000","last_clean_scrub_stamp":"2024-03-26T04:13:37.607799+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.297848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":86,"num_read_kb":72,"num_write":56,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1115,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663701+0000","last_change":"2024-03-26T04:13:24.919940+0000","last_active":"2024-03-26T04:13:44.663701+0000","last_peered":"2024-03-26T04:13:44.663701+0000","last_clean":"2024-03-26T04:13:44.663701+0000","last_became_active":"2024-03-26T04:12:11.881608+0000","last_became_peered":"2024-03-26T04:12:11.881608+0000","last_unstale":"2024-03-26T04:13:44.663701+0000","last_undegraded":"2024-03-26T04:13:44.663701+0000","last_fullsized":"2024-03-26T04:13:44.663701+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:24.919819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:04:06.297701+0000","last_clean_scrub_stamp":"2024-03-26T04:13:24.919819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:36.479562+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"79'128","reported_seq":433,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200893+0000","last_change":"2024-03-26T04:13:17.559995+0000","last_active":"2024-03-26T04:13:45.200893+0000","last_peered":"2024-03-26T04:13:45.200893+0000","last_clean":"2024-03-26T04:13:45.200893+0000","last_became_active":"2024-03-26T04:12:18.659657+0000","last_became_peered":"2024-03-26T04:12:18.659657+0000","last_unstale":"2024-03-26T04:13:45.200893+0000","last_undegraded":"2024-03-26T04:13:45.200893+0000","last_fullsized":"2024-03-26T04:13:45.200893+0000","mapping_epoch":524,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":525,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:13:17.559860+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:00:35.340085+0000","last_clean_scrub_stamp":"2024-03-26T04:13:17.559860+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:45.570707+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":153,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.665830+0000","last_change":"2024-03-26T04:13:57.665880+0000","last_active":"2024-03-26T04:13:57.665830+0000","last_peered":"2024-03-26T04:13:57.665830+0000","last_clean":"2024-03-26T04:13:57.665830+0000","last_became_active":"2024-03-26T04:12:29.656864+0000","last_became_peered":"2024-03-26T04:12:29.656864+0000","last_unstale":"2024-03-26T04:13:57.665830+0000","last_undegraded":"2024-03-26T04:13:57.665830+0000","last_fullsized":"2024-03-26T04:13:57.665830+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.665730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:53:28.635838+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.665730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:13.842790+0000","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"510'14","reported_seq":1416,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:59.395803+0000","last_change":"2024-03-26T04:12:57.939664+0000","last_active":"2024-03-26T04:13:59.395803+0000","last_peered":"2024-03-26T04:13:59.395803+0000","last_clean":"2024-03-26T04:13:59.395803+0000","last_became_active":"2024-03-26T04:12:03.838321+0000","last_became_peered":"2024-03-26T04:12:03.838321+0000","last_unstale":"2024-03-26T04:13:59.395803+0000","last_undegraded":"2024-03-26T04:13:59.395803+0000","last_fullsized":"2024-03-26T04:13:59.395803+0000","mapping_epoch":509,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"510'14","last_scrub_stamp":"2024-03-26T04:12:57.939567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:57.939567+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:18.031183+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664359+0000","last_change":"2024-03-26T04:12:49.876477+0000","last_active":"2024-03-26T04:13:44.664359+0000","last_peered":"2024-03-26T04:13:44.664359+0000","last_clean":"2024-03-26T04:13:44.664359+0000","last_became_active":"2024-03-26T04:12:11.902422+0000","last_became_peered":"2024-03-26T04:12:11.902422+0000","last_unstale":"2024-03-26T04:13:44.664359+0000","last_undegraded":"2024-03-26T04:13:44.664359+0000","last_fullsized":"2024-03-26T04:13:44.664359+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:49.876374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:31.922297+0000","last_clean_scrub_stamp":"2024-03-26T04:12:49.876374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:03.259627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"448'39","reported_seq":1300,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:58.610278+0000","last_change":"2024-03-26T04:13:58.610330+0000","last_active":"2024-03-26T04:13:58.610278+0000","last_peered":"2024-03-26T04:13:58.610278+0000","last_clean":"2024-03-26T04:13:58.610278+0000","last_became_active":"2024-03-26T04:12:11.892439+0000","last_became_peered":"2024-03-26T04:12:11.892439+0000","last_unstale":"2024-03-26T04:13:58.610278+0000","last_undegraded":"2024-03-26T04:13:58.610278+0000","last_fullsized":"2024-03-26T04:13:58.610278+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'39","last_scrub_stamp":"2024-03-26T04:13:58.610208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:58.610208+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:17.322978+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":119,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.666170+0000","last_change":"2024-03-26T04:12:42.869017+0000","last_active":"2024-03-26T04:13:44.666170+0000","last_peered":"2024-03-26T04:13:44.666170+0000","last_clean":"2024-03-26T04:13:44.666170+0000","last_became_active":"2024-03-26T04:12:15.655609+0000","last_became_peered":"2024-03-26T04:12:15.655609+0000","last_unstale":"2024-03-26T04:13:44.666170+0000","last_undegraded":"2024-03-26T04:13:44.666170+0000","last_fullsized":"2024-03-26T04:13:44.666170+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:42.868898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:08:30.133040+0000","last_clean_scrub_stamp":"2024-03-26T04:12:42.868898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:51.336079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":143,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.201041+0000","last_change":"2024-03-26T04:13:27.482011+0000","last_active":"2024-03-26T04:13:45.201041+0000","last_peered":"2024-03-26T04:13:45.201041+0000","last_clean":"2024-03-26T04:13:45.201041+0000","last_became_active":"2024-03-26T04:12:15.664011+0000","last_became_peered":"2024-03-26T04:12:15.664011+0000","last_unstale":"2024-03-26T04:13:45.201041+0000","last_undegraded":"2024-03-26T04:13:45.201041+0000","last_fullsized":"2024-03-26T04:13:45.201041+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:27.481912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:52:42.283958+0000","last_clean_scrub_stamp":"2024-03-26T04:13:27.481912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:31.004240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":1089,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.663688+0000","last_change":"2024-03-26T04:12:56.978763+0000","last_active":"2024-03-26T04:13:44.663688+0000","last_peered":"2024-03-26T04:13:44.663688+0000","last_clean":"2024-03-26T04:13:44.663688+0000","last_became_active":"2024-03-26T04:12:08.682229+0000","last_became_peered":"2024-03-26T04:12:08.682229+0000","last_unstale":"2024-03-26T04:13:44.663688+0000","last_undegraded":"2024-03-26T04:13:44.663688+0000","last_fullsized":"2024-03-26T04:13:44.663688+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:56.978663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:49:48.864997+0000","last_clean_scrub_stamp":"2024-03-26T04:12:56.978663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:06.093627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"448'33","reported_seq":1364,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664582+0000","last_change":"2024-03-26T04:13:35.055258+0000","last_active":"2024-03-26T04:13:44.664582+0000","last_peered":"2024-03-26T04:13:44.664582+0000","last_clean":"2024-03-26T04:13:44.664582+0000","last_became_active":"2024-03-26T04:12:29.654543+0000","last_became_peered":"2024-03-26T04:12:29.654543+0000","last_unstale":"2024-03-26T04:13:44.664582+0000","last_undegraded":"2024-03-26T04:13:44.664582+0000","last_fullsized":"2024-03-26T04:13:44.664582+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'33","last_scrub_stamp":"2024-03-26T04:13:35.055131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:35.055131+0000","objects_scrubbed":6,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:40.214551+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":45,"num_write":34,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":1210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"79'128","reported_seq":510,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.200557+0000","last_change":"2024-03-26T04:12:47.630769+0000","last_active":"2024-03-26T04:13:45.200557+0000","last_peered":"2024-03-26T04:13:45.200557+0000","last_clean":"2024-03-26T04:13:45.200557+0000","last_became_active":"2024-03-26T04:12:14.661582+0000","last_became_peered":"2024-03-26T04:12:14.661582+0000","last_unstale":"2024-03-26T04:13:45.200557+0000","last_undegraded":"2024-03-26T04:13:45.200557+0000","last_fullsized":"2024-03-26T04:13:45.200557+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:47.630674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:46:31.055931+0000","last_clean_scrub_stamp":"2024-03-26T04:12:47.630674+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:07.766240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1032,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:57.874573+0000","last_change":"2024-03-26T04:13:57.874596+0000","last_active":"2024-03-26T04:13:57.874573+0000","last_peered":"2024-03-26T04:13:57.874573+0000","last_clean":"2024-03-26T04:13:57.874573+0000","last_became_active":"2024-03-26T04:12:11.878355+0000","last_became_peered":"2024-03-26T04:12:11.878355+0000","last_unstale":"2024-03-26T04:13:57.874573+0000","last_undegraded":"2024-03-26T04:13:57.874573+0000","last_fullsized":"2024-03-26T04:13:57.874573+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:57.874497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:12:37.500679+0000","last_clean_scrub_stamp":"2024-03-26T04:13:57.874497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:03.148212+0000","scrub_duration":175,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1073,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.725953+0000","last_change":"2024-03-26T04:13:03.292505+0000","last_active":"2024-03-26T04:13:44.725953+0000","last_peered":"2024-03-26T04:13:44.725953+0000","last_clean":"2024-03-26T04:13:44.725953+0000","last_became_active":"2024-03-26T03:48:48.026625+0000","last_became_peered":"2024-03-26T03:48:48.026625+0000","last_unstale":"2024-03-26T04:13:44.725953+0000","last_undegraded":"2024-03-26T04:13:44.725953+0000","last_fullsized":"2024-03-26T04:13:44.725953+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:03.292382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:01:29.373227+0000","last_clean_scrub_stamp":"2024-03-26T04:13:03.292382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:25.987480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":118,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.734561+0000","last_change":"2024-03-26T04:13:44.734583+0000","last_active":"2024-03-26T04:13:44.734561+0000","last_peered":"2024-03-26T04:13:44.734561+0000","last_clean":"2024-03-26T04:13:44.734561+0000","last_became_active":"2024-03-26T04:12:29.654073+0000","last_became_peered":"2024-03-26T04:12:29.654073+0000","last_unstale":"2024-03-26T04:13:44.734561+0000","last_undegraded":"2024-03-26T04:13:44.734561+0000","last_fullsized":"2024-03-26T04:13:44.734561+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:44.734499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:44.734499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.687565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"565'53","reported_seq":1159,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:56.981469+0000","last_change":"2024-03-26T04:13:56.981536+0000","last_active":"2024-03-26T04:13:56.981469+0000","last_peered":"2024-03-26T04:13:56.981469+0000","last_clean":"2024-03-26T04:13:56.981469+0000","last_became_active":"2024-03-26T04:05:29.520566+0000","last_became_peered":"2024-03-26T04:05:29.520566+0000","last_unstale":"2024-03-26T04:13:56.981469+0000","last_undegraded":"2024-03-26T04:13:56.981469+0000","last_fullsized":"2024-03-26T04:13:56.981469+0000","mapping_epoch":378,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":379,"parent":"0.0","parent_split_bits":0,"last_scrub":"565'53","last_scrub_stamp":"2024-03-26T04:13:56.981397+0000","last_deep_scrub":"565'53","last_deep_scrub_stamp":"2024-03-26T04:13:56.981397+0000","last_clean_scrub_stamp":"2024-03-26T04:13:56.981397+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:15:23.609533+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"79'128","reported_seq":1019,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:45.049648+0000","last_change":"2024-03-26T04:12:52.803427+0000","last_active":"2024-03-26T04:13:45.049648+0000","last_peered":"2024-03-26T04:13:45.049648+0000","last_clean":"2024-03-26T04:13:45.049648+0000","last_became_active":"2024-03-26T04:12:14.661744+0000","last_became_peered":"2024-03-26T04:12:14.661744+0000","last_unstale":"2024-03-26T04:13:45.049648+0000","last_undegraded":"2024-03-26T04:13:45.049648+0000","last_fullsized":"2024-03-26T04:13:45.049648+0000","mapping_epoch":520,"log_start":"0'0","ondisk_log_start":"0'0","created":271,"last_epoch_clean":521,"parent":"0.0","parent_split_bits":4,"last_scrub":"79'128","last_scrub_stamp":"2024-03-26T04:12:52.803327+0000","last_deep_scrub":"79'128","last_deep_scrub_stamp":"2024-03-26T04:07:48.931565+0000","last_clean_scrub_stamp":"2024-03-26T04:12:52.803327+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:14.096349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":1083,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664428+0000","last_change":"2024-03-26T04:12:48.090953+0000","last_active":"2024-03-26T04:13:44.664428+0000","last_peered":"2024-03-26T04:13:44.664428+0000","last_clean":"2024-03-26T04:13:44.664428+0000","last_became_active":"2024-03-26T04:12:29.653566+0000","last_became_peered":"2024-03-26T04:12:29.653566+0000","last_unstale":"2024-03-26T04:13:44.664428+0000","last_undegraded":"2024-03-26T04:13:44.664428+0000","last_fullsized":"2024-03-26T04:13:44.664428+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:12:48.090845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:54:28.365009+0000","last_clean_scrub_stamp":"2024-03-26T04:12:48.090845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:13:52.933804+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"515'16","reported_seq":1344,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:54.394484+0000","last_change":"2024-03-26T04:12:55.531644+0000","last_active":"2024-03-26T04:13:54.394484+0000","last_peered":"2024-03-26T04:13:54.394484+0000","last_clean":"2024-03-26T04:13:54.394484+0000","last_became_active":"2024-03-26T04:12:08.696335+0000","last_became_peered":"2024-03-26T04:12:08.696335+0000","last_unstale":"2024-03-26T04:13:54.394484+0000","last_undegraded":"2024-03-26T04:13:54.394484+0000","last_fullsized":"2024-03-26T04:13:54.394484+0000","mapping_epoch":514,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":515,"parent":"0.0","parent_split_bits":0,"last_scrub":"515'16","last_scrub_stamp":"2024-03-26T04:12:55.531545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:47.006138+0000","last_clean_scrub_stamp":"2024-03-26T04:12:55.531545+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:19.410810+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":113,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.665265+0000","last_change":"2024-03-26T04:13:07.871715+0000","last_active":"2024-03-26T04:13:44.665265+0000","last_peered":"2024-03-26T04:13:44.665265+0000","last_clean":"2024-03-26T04:13:44.665265+0000","last_became_active":"2024-03-26T04:12:11.902647+0000","last_became_peered":"2024-03-26T04:12:11.902647+0000","last_unstale":"2024-03-26T04:13:44.665265+0000","last_undegraded":"2024-03-26T04:13:44.665265+0000","last_fullsized":"2024-03-26T04:13:44.665265+0000","mapping_epoch":517,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":518,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:13:07.871615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:42.999718+0000","last_clean_scrub_stamp":"2024-03-26T04:13:07.871615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:15.672879+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"448'30","reported_seq":1261,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-03-26T04:13:44.664059+0000","last_change":"2024-03-26T04:13:42.106430+0000","last_active":"2024-03-26T04:13:44.664059+0000","last_peered":"2024-03-26T04:13:44.664059+0000","last_clean":"2024-03-26T04:13:44.664059+0000","last_became_active":"2024-03-26T04:12:15.669351+0000","last_became_peered":"2024-03-26T04:12:15.669351+0000","last_unstale":"2024-03-26T04:13:44.664059+0000","last_undegraded":"2024-03-26T04:13:44.664059+0000","last_fullsized":"2024-03-26T04:13:44.664059+0000","mapping_epoch":521,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":522,"parent":"0.0","parent_split_bits":0,"last_scrub":"448'30","last_scrub_stamp":"2024-03-26T04:13:42.106315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T03:48:45.002460+0000","last_clean_scrub_stamp":"2024-03-26T04:13:42.106315+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-26T04:14:53.243456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":122,"num_read_kb":117,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12,"ondisk_log_size":12,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":1817,"num_read_kb":1542,"num_write":1181,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1032,"num_bytes_recovered":17884,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},"log_size":1298,"ondisk_log_size":1298,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070773,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1012028,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733124,"statfs":{"total":95995035648,"available":94958718976,"internally_reserved":0,"allocated":2150400,"data_stored":1309827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":1034156490},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365115,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37084,"kb_used_data":1748,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708068,"statfs":{"total":95995035648,"available":95957061632,"internally_reserved":0,"allocated":1789952,"data_stored":979735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575187,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1188884,"kb_used_data":1804,"kb_used_omap":1,"kb_used_meta":1187070,"kb_avail":92556268,"statfs":{"total":95995035648,"available":94777618432,"internally_reserved":0,"allocated":1847296,"data_stored":981300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1215560133},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738712,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1295740,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":1293950,"kb_avail":92449412,"statfs":{"total":95995035648,"available":94668197888,"internally_reserved":0,"allocated":1822720,"data_stored":981876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1325005253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":378,"seq":1623497637993,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":666280,"kb_used_data":2016,"kb_used_omap":6,"kb_used_meta":664249,"kb_avail":93078872,"statfs":{"total":95995035648,"available":95312764928,"internally_reserved":0,"allocated":2064384,"data_stored":1242771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6875,"internal_metadata":680191269},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182729,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565260,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93179892,"statfs":{"total":95995035648,"available":95416209408,"internally_reserved":0,"allocated":2428928,"data_stored":1572341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":576387527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953796,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1161212,"kb_used_data":1780,"kb_used_omap":1,"kb_used_meta":1159422,"kb_avail":92583940,"statfs":{"total":95995035648,"available":94805954560,"internally_reserved":0,"allocated":1822720,"data_stored":980353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1187248581},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":180,"seq":773094113499,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487076,"kb_used_data":2332,"kb_used_omap":4,"kb_used_meta":484731,"kb_avail":93258076,"statfs":{"total":95995035648,"available":95496269824,"internally_reserved":0,"allocated":2387968,"data_stored":1570429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4448,"internal_metadata":496365216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1485,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2431,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:14:09.372 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-26T04:14:09.373 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003 shell --fsid de980954-eb22-11ee-95d0-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-26T04:14:09.967 INFO:teuthology.orchestra.run.smithi006.stderr:Inferring config /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:09.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:09 smithi086 ceph-mon[32965]: 3.15 scrub starts 2024-03-26T04:14:09.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:09 smithi086 ceph-mon[32965]: 3.15 scrub ok 2024-03-26T04:14:09.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:09 smithi086 ceph-mon[32965]: from='client.26147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:09.995 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:09 smithi086 ceph-mon[32965]: pgmap v1243: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:10.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[37501]: 3.15 scrub starts 2024-03-26T04:14:10.089 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[37501]: 3.15 scrub ok 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[37501]: from='client.26147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[37501]: pgmap v1243: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[29446]: 3.15 scrub starts 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[29446]: 3.15 scrub ok 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[29446]: from='client.26147 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:14:10.090 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:09 smithi006 ceph-mon[29446]: pgmap v1243: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:10.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:10 smithi086 ceph-mon[32965]: 1.8 scrub starts 2024-03-26T04:14:10.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:10 smithi086 ceph-mon[32965]: 1.8 scrub ok 2024-03-26T04:14:10.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:10 smithi086 ceph-mon[32965]: 6.13 scrub starts 2024-03-26T04:14:10.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:10 smithi086 ceph-mon[32965]: 6.13 scrub ok 2024-03-26T04:14:10.983 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:10 smithi086 ceph-mon[32965]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[37501]: 1.8 scrub starts 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[37501]: 1.8 scrub ok 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[37501]: 6.13 scrub starts 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[37501]: 6.13 scrub ok 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[37501]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[29446]: 1.8 scrub starts 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[29446]: 1.8 scrub ok 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[29446]: 6.13 scrub starts 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[29446]: 6.13 scrub ok 2024-03-26T04:14:11.139 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:10 smithi006 ceph-mon[29446]: from='mgr.14580 172.21.15.6:0/2674840067' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:14:11.889 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 3.19 scrub starts 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 3.19 scrub ok 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: pgmap v1244: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 6.11 scrub starts 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 1.5 scrub starts 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 1.5 scrub ok 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: 6.11 scrub ok 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[29446]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:11.890 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 3.19 scrub starts 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 3.19 scrub ok 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: pgmap v1244: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 6.11 scrub starts 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 1.5 scrub starts 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 1.5 scrub ok 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: 6.11 scrub ok 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:11.891 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-mon[37501]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 3.19 scrub starts 2024-03-26T04:14:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 3.19 scrub ok 2024-03-26T04:14:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: pgmap v1244: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:14:12.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 6.11 scrub starts 2024-03-26T04:14:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 1.5 scrub starts 2024-03-26T04:14:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 1.5 scrub ok 2024-03-26T04:14:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: 6.11 scrub ok 2024-03-26T04:14:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:12.088 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:11 smithi086 ceph-mon[32965]: from='mgr.14580 ' entity='mgr.y' 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:11 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:11] ENGINE Bus STOPPING 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:12] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:12] ENGINE Bus STOPPED 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:12] ENGINE Bus STARTING 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:12] ENGINE Serving on http://:::9283 2024-03-26T04:14:12.285 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: [26/Mar/2024:04:14:12] ENGINE Bus STARTED 2024-03-26T04:14:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T04:14:12.889 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-mon[37501]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T04:14:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T04:14:12.890 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:12 smithi006 ceph-mon[29446]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T04:14:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-26T04:14:13.087 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:12 smithi086 ceph-mon[32965]: from='client.? 172.21.15.6:0/3048389931' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-26T04:14:13.280 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T04:14:13.289 INFO:tasks.cephadm:Teardown begin 2024-03-26T04:14:13.290 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:14:13.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:14:13.647 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T04:14:13.647 DEBUG:teuthology.orchestra.run.smithi006:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:14:13.703 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:14:13.730 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T04:14:13.730 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T04:14:13.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:14:13.780 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 systemd[1]: Stopping Ceph mon.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[37501]: pgmap v1245: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:14.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[37501]: 6.1f scrub starts 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[37501]: 3.9 scrub starts 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[37501]: 3.9 scrub ok 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[37501]: 6.1f scrub ok 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[29446]: pgmap v1245: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[29446]: 6.1f scrub starts 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[29446]: 3.9 scrub starts 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[29446]: 3.9 scrub ok 2024-03-26T04:14:14.140 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:13 smithi006 ceph-mon[29446]: 6.1f scrub ok 2024-03-26T04:14:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:13 smithi086 ceph-mon[32965]: pgmap v1245: 142 pgs: 142 active+clean; 582 KiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:14:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:13 smithi086 ceph-mon[32965]: 6.1f scrub starts 2024-03-26T04:14:14.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:13 smithi086 ceph-mon[32965]: 3.9 scrub starts 2024-03-26T04:14:14.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:13 smithi086 ceph-mon[32965]: 3.9 scrub ok 2024-03-26T04:14:14.338 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:13 smithi086 ceph-mon[32965]: 6.1f scrub ok 2024-03-26T04:14:14.639 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y[29678]: ::ffff:172.21.15.86 - - [26/Mar/2024:04:14:14] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:14:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:14:14.234+0000 7fd46b796700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:14 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a[29442]: 2024-03-26T04:14:14.234+0000 7fd46b796700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T04:14:14.640 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:14 smithi006 podman[213613]: 2024-03-26 04:14:14.418331534 +0000 UTC m=+0.306832311 container died 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-26T04:14:15.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:14 smithi006 podman[213613]: 2024-03-26 04:14:14.801723583 +0000 UTC m=+0.690224352 container cleanup 621b7e019e792c89a9d8bc1dd79952033ee7dedc907430410febe33e3b6d2e0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux ) 2024-03-26T04:14:15.069 INFO:journalctl@ceph.mon.a.smithi006.stdout:Mar 26 04:14:14 smithi006 bash[213613]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:14:15.169 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T04:14:15.209 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:15.209 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T04:14:15.209 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T04:14:15.210 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.c 2024-03-26T04:14:15.389 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 systemd[1]: Stopping Ceph mon.c for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:15.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c[37494]: 2024-03-26T04:14:15.514+0000 7fe6a44d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:15.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c[37494]: 2024-03-26T04:14:15.514+0000 7fe6a44d9700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:14:15.857 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 podman[213767]: 2024-03-26 04:14:15.637546216 +0000 UTC m=+0.210128172 container died e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:14:16.087 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:14:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug there is no tcmu-runner data available 2024-03-26T04:14:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 podman[213767]: 2024-03-26 04:14:15.956853156 +0000 UTC m=+0.529435061 container cleanup e80875f9a6b4b7b7125b5af18b0494f08db64fab1e1da096bf7b674ce5f9160f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-26T04:14:16.139 INFO:journalctl@ceph.mon.c.smithi006.stdout:Mar 26 04:14:15 smithi006 bash[213767]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:14:16.304 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T04:14:16.341 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:16.341 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T04:14:16.341 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T04:14:16.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.b 2024-03-26T04:14:16.646 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:16 smithi086 systemd[1]: Stopping Ceph mon.b for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:17.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-b[32948]: 2024-03-26T04:14:16.645+0000 7fea26704700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:17.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-b[32948]: 2024-03-26T04:14:16.645+0000 7fea26704700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:14:17.051 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:16 smithi086 podman[166882]: 2024-03-26 04:14:16.738327466 +0000 UTC m=+0.200153571 container died af6c19465bfa6962cf824be0557a3ce62215355b5c83047a2b8aa48383b38474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:14:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:17 smithi086 podman[166882]: 2024-03-26 04:14:17.246802173 +0000 UTC m=+0.708628285 container cleanup af6c19465bfa6962cf824be0557a3ce62215355b5c83047a2b8aa48383b38474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T04:14:17.337 INFO:journalctl@ceph.mon.b.smithi086.stdout:Mar 26 04:14:17 smithi086 bash[166882]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:14:17.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T04:14:17.634 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:17.635 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T04:14:17.635 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T04:14:17.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:14:17.954 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:17 smithi006 systemd[1]: Stopping Ceph mgr.y for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:18.212 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:17 smithi006 podman[213890]: 2024-03-26 04:14:17.959162724 +0000 UTC m=+0.128622254 container died 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-26T04:14:18.470 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:18 smithi006 podman[213890]: 2024-03-26 04:14:18.294667343 +0000 UTC m=+0.464126863 container cleanup 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-26T04:14:18.471 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:18 smithi006 bash[213890]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:14:18.728 INFO:journalctl@ceph.mgr.y.smithi006.stdout:Mar 26 04:14:18 smithi006 podman[213904]: 2024-03-26 04:14:18.521117531 +0000 UTC m=+0.558986038 container remove 84e1717f6ba222f90211d5c80d413a60158063cc55c00c71cf782f0cd5e18444 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e42b8ea316eec6c6f4fae026f6d6b62fa5fe7003, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-26T04:14:18.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T04:14:18.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:18.797 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T04:14:18.797 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T04:14:18.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.x 2024-03-26T04:14:19.087 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 04:14:18 smithi086 systemd[1]: Stopping Ceph mgr.x for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:19.441 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 04:14:19 smithi086 podman[167003]: 2024-03-26 04:14:19.157139852 +0000 UTC m=+0.155679003 container died ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-26T04:14:19.827 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 04:14:19 smithi086 podman[167003]: 2024-03-26 04:14:19.53461228 +0000 UTC m=+0.533151428 container cleanup ebcf4d341f340e3daa438266e1a21a89a7c22fa0648e57958dd636f91f1d0eab (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-26T04:14:19.827 INFO:journalctl@ceph.mgr.x.smithi086.stdout:Mar 26 04:14:19 smithi086 bash[167003]: ceph-de980954-eb22-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:14:19.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T04:14:19.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:19.987 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T04:14:19.987 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T04:14:19.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.0 2024-03-26T04:14:20.342 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:20 smithi006 systemd[1]: Stopping Ceph osd.0 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:20.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0[41524]: 2024-03-26T04:14:20.340+0000 7f472c101700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:20.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0[41524]: 2024-03-26T04:14:20.340+0000 7f472c101700 -1 osd.0 565 *** Got signal Terminated *** 2024-03-26T04:14:20.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:20 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0[41524]: 2024-03-26T04:14:20.340+0000 7f472c101700 -1 osd.0 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:14:26.211 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:25 smithi006 podman[214010]: 2024-03-26 04:14:25.946427665 +0000 UTC m=+5.749929137 container died 6e15672afbf6943ad91e84d80b3d8a4853b1fc2073ebeb8eabd843dfc57ea436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T04:14:26.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:26 smithi006 podman[214010]: 2024-03-26 04:14:26.368433455 +0000 UTC m=+6.171934924 container cleanup 6e15672afbf6943ad91e84d80b3d8a4853b1fc2073ebeb8eabd843dfc57ea436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:14:26.639 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:26 smithi006 bash[214010]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:14:27.131 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:26 smithi006 podman[214027]: 2024-03-26 04:14:26.644857251 +0000 UTC m=+0.696022692 container remove 6e15672afbf6943ad91e84d80b3d8a4853b1fc2073ebeb8eabd843dfc57ea436 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:14:27.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:27 smithi006 podman[214118]: 2024-03-26 04:14:27.033797715 +0000 UTC m=+0.017726823 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:14:27.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:27 smithi006 podman[214118]: 2024-03-26 04:14:27.137425543 +0000 UTC m=+0.121354649 container create e47223499fa72bb80b80cef479eedd8f17a923db81a00215fa3dd00fb3f06e21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:14:27.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:27 smithi006 podman[214118]: 2024-03-26 04:14:27.356986048 +0000 UTC m=+0.340915164 container init e47223499fa72bb80b80cef479eedd8f17a923db81a00215fa3dd00fb3f06e21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T04:14:27.390 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:27 smithi006 podman[214118]: 2024-03-26 04:14:27.360477705 +0000 UTC m=+0.344406812 container start e47223499fa72bb80b80cef479eedd8f17a923db81a00215fa3dd00fb3f06e21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T04:14:28.001 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:27 smithi006 podman[214118]: 2024-03-26 04:14:27.715309 +0000 UTC m=+0.699238121 container attach e47223499fa72bb80b80cef479eedd8f17a923db81a00215fa3dd00fb3f06e21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T04:14:28.293 INFO:journalctl@ceph.osd.0.smithi006.stdout:Mar 26 04:14:28 smithi006 podman[214118]: 2024-03-26 04:14:28.0017304 +0000 UTC m=+0.985659512 container died e47223499fa72bb80b80cef479eedd8f17a923db81a00215fa3dd00fb3f06e21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:14:28.602 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T04:14:28.644 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:28.644 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T04:14:28.645 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T04:14:28.645 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.1 2024-03-26T04:14:28.889 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:28 smithi006 systemd[1]: Stopping Ceph osd.1 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:29.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:29 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1[47368]: 2024-03-26T04:14:29.014+0000 7fd25761c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:29.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:29 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1[47368]: 2024-03-26T04:14:29.014+0000 7fd25761c700 -1 osd.1 565 *** Got signal Terminated *** 2024-03-26T04:14:29.390 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:29 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1[47368]: 2024-03-26T04:14:29.014+0000 7fd25761c700 -1 osd.1 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:14:34.756 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:34 smithi006 podman[214224]: 2024-03-26 04:14:34.486966241 +0000 UTC m=+5.619001517 container died 7121d2eea1229d13e614a430d8232ea57d0f8494933b19277cab75a231b13e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:14:35.132 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:34 smithi006 podman[214224]: 2024-03-26 04:14:34.847682001 +0000 UTC m=+5.979717264 container cleanup 7121d2eea1229d13e614a430d8232ea57d0f8494933b19277cab75a231b13e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T04:14:35.132 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:34 smithi006 bash[214224]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:14:35.389 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214244]: 2024-03-26 04:14:35.132355439 +0000 UTC m=+0.642193993 container remove 7121d2eea1229d13e614a430d8232ea57d0f8494933b19277cab75a231b13e31 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True) 2024-03-26T04:14:35.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214338]: 2024-03-26 04:14:35.374307264 +0000 UTC m=+0.017451655 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:14:35.810 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214338]: 2024-03-26 04:14:35.483428483 +0000 UTC m=+0.126572873 container create 123c451cd976162d8b94402d7649738c594821d7cdc085a62c584f4dc80af630 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T04:14:35.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214338]: 2024-03-26 04:14:35.711229117 +0000 UTC m=+0.354373517 container init 123c451cd976162d8b94402d7649738c594821d7cdc085a62c584f4dc80af630 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:14:35.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214338]: 2024-03-26 04:14:35.714738726 +0000 UTC m=+0.357883116 container start 123c451cd976162d8b94402d7649738c594821d7cdc085a62c584f4dc80af630 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T04:14:35.811 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:35 smithi006 podman[214338]: 2024-03-26 04:14:35.810781113 +0000 UTC m=+0.453925520 container attach 123c451cd976162d8b94402d7649738c594821d7cdc085a62c584f4dc80af630 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-26T04:14:36.139 INFO:journalctl@ceph.osd.1.smithi006.stdout:Mar 26 04:14:36 smithi006 podman[214338]: 2024-03-26 04:14:36.01622185 +0000 UTC m=+0.659366241 container died 123c451cd976162d8b94402d7649738c594821d7cdc085a62c584f4dc80af630 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T04:14:36.586 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T04:14:36.626 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:36.627 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T04:14:36.627 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:14:36.627 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:14:36.889 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:36 smithi006 systemd[1]: Stopping Ceph osd.2 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:37.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:36 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:14:36.975+0000 7fd40d3b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:37.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:36 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:14:36.975+0000 7fd40d3b1700 -1 osd.2 565 *** Got signal Terminated *** 2024-03-26T04:14:37.281 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:36 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2[168203]: 2024-03-26T04:14:36.975+0000 7fd40d3b1700 -1 osd.2 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:14:42.676 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:42 smithi006 podman[214456]: 2024-03-26 04:14:42.413847508 +0000 UTC m=+5.559118775 container died 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:14:43.010 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:42 smithi006 podman[214456]: 2024-03-26 04:14:42.797521043 +0000 UTC m=+5.942792311 container cleanup 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:14:43.010 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:42 smithi006 bash[214456]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:14:43.346 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214475]: 2024-03-26 04:14:43.010817459 +0000 UTC m=+0.599872558 container remove 646532395a78d076e41405b2cccd9da94be80a913694c049d451a3b674d69f9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-26T04:14:43.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.24968701 +0000 UTC m=+0.017860913 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:14:43.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.369969754 +0000 UTC m=+0.138143646 container create 507fc104258f397bb5f0be85b27d8226ee4c6b3d8621eda53474d70287c034d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:14:43.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.572254954 +0000 UTC m=+0.340428846 container init 507fc104258f397bb5f0be85b27d8226ee4c6b3d8621eda53474d70287c034d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-26T04:14:43.640 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.575379836 +0000 UTC m=+0.343553726 container start 507fc104258f397bb5f0be85b27d8226ee4c6b3d8621eda53474d70287c034d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-26T04:14:44.139 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.647290861 +0000 UTC m=+0.415464764 container attach 507fc104258f397bb5f0be85b27d8226ee4c6b3d8621eda53474d70287c034d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:14:44.140 INFO:journalctl@ceph.osd.2.smithi006.stdout:Mar 26 04:14:43 smithi006 podman[214564]: 2024-03-26 04:14:43.868927234 +0000 UTC m=+0.637101124 container died 507fc104258f397bb5f0be85b27d8226ee4c6b3d8621eda53474d70287c034d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:14:44.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:14:44.563 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:44.563 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:14:44.564 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T04:14:44.564 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.3 2024-03-26T04:14:44.863 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:44 smithi006 systemd[1]: Stopping Ceph osd.3 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:45.139 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:44.862+0000 7fa5e4634700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:45.140 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:44.862+0000 7fa5e4634700 -1 osd.3 565 *** Got signal Terminated *** 2024-03-26T04:14:45.140 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:44 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:44.862+0000 7fa5e4634700 -1 osd.3 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:14:46.889 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:46 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:46.588+0000 7fa5dcc20700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:22.480369+0000 front 2024-03-26T04:14:22.480434+0000 (oldest deadline 2024-03-26T04:14:46.580523+0000) 2024-03-26T04:14:47.338 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:46 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:46.858+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:47.338 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:46 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:46.837+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:47.889 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:47 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:47.575+0000 7fa5dcc20700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:22.480369+0000 front 2024-03-26T04:14:22.480434+0000 (oldest deadline 2024-03-26T04:14:46.580523+0000) 2024-03-26T04:14:48.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:47 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:47.836+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:48.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:47 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:47.877+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:48.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:47 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:47.770+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:48.639 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:48 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:48.576+0000 7fa5dcc20700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:22.480369+0000 front 2024-03-26T04:14:22.480434+0000 (oldest deadline 2024-03-26T04:14:46.580523+0000) 2024-03-26T04:14:49.087 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:48 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:48.836+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:49.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:48 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:48.826+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:49.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:48 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:48.763+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:49.837 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:49 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:49.743+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:49.889 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:49 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3[58789]: 2024-03-26T04:14:49.531+0000 7fa5dcc20700 -1 osd.3 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:22.480369+0000 front 2024-03-26T04:14:22.480434+0000 (oldest deadline 2024-03-26T04:14:46.580523+0000) 2024-03-26T04:14:50.337 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:49 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:49.867+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:50.338 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:49 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:49.844+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:50.735 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:50 smithi006 podman[214669]: 2024-03-26 04:14:50.461113469 +0000 UTC m=+5.702333597 container died 2e46d5c7e2692a577a1a9a0b262854b432ff0283edccace6733d2754f1530042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T04:14:51.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:50.815+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:51.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:50.904+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:51.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:50 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:50.777+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:51.096 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:50 smithi006 podman[214669]: 2024-03-26 04:14:50.845600975 +0000 UTC m=+6.086821102 container cleanup 2e46d5c7e2692a577a1a9a0b262854b432ff0283edccace6733d2754f1530042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T04:14:51.096 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:50 smithi006 bash[214669]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3 2024-03-26T04:14:51.389 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214686]: 2024-03-26 04:14:51.096868289 +0000 UTC m=+0.619238181 container remove 2e46d5c7e2692a577a1a9a0b262854b432ff0283edccace6733d2754f1530042 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0) 2024-03-26T04:14:51.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:51.185+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:51.708 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.325754546 +0000 UTC m=+0.017840782 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:14:51.709 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.430973039 +0000 UTC m=+0.123059263 container create cb1c5173ea60b596d9cf07b95f673510c8b67948479bef3255151ee3e9d85dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T04:14:51.709 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.650208388 +0000 UTC m=+0.342294615 container init cb1c5173ea60b596d9cf07b95f673510c8b67948479bef3255151ee3e9d85dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-26T04:14:51.709 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.653260031 +0000 UTC m=+0.345346258 container start cb1c5173ea60b596d9cf07b95f673510c8b67948479bef3255151ee3e9d85dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-26T04:14:51.709 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.708507218 +0000 UTC m=+0.400593461 container attach cb1c5173ea60b596d9cf07b95f673510c8b67948479bef3255151ee3e9d85dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:14:52.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:51.821+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:52.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:51.755+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:52.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:51.894+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:52.088 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:14:51.866Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-26T04:14:52.088 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:14:51.868Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-26T04:14:52.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:14:51.868Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-26T04:14:52.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:14:51.870Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-26T04:14:52.089 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:14:51 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:14:51.870Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.6:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.6:8765: connect: connection refused" 2024-03-26T04:14:52.139 INFO:journalctl@ceph.osd.3.smithi006.stdout:Mar 26 04:14:51 smithi006 podman[214779]: 2024-03-26 04:14:51.949658583 +0000 UTC m=+0.641744801 container died cb1c5173ea60b596d9cf07b95f673510c8b67948479bef3255151ee3e9d85dd3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:14:52.565 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T04:14:52.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:52.138+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:52.607 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:14:52.607 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T04:14:52.607 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T04:14:52.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.4 2024-03-26T04:14:52.890 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:52 smithi086 systemd[1]: Stopping Ceph osd.4 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:14:52.891 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:52.842+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:52.891 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:52.761+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:53.180 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:52.890+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:53.180 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:52.921+0000 7fb88b868700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:14:53.181 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:52.921+0000 7fb88b868700 -1 osd.4 565 *** Got signal Terminated *** 2024-03-26T04:14:53.181 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:52 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:52.921+0000 7fb88b868700 -1 osd.4 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:14:53.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:53 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:53.180+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:54.087 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:53 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:53.867+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:54.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:53 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:53.836+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:54.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:53 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:53.735+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:54.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:54.134+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:55.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:54.851+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:55.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:54.852+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:55.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:54.852+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:33.081291+0000 front 2024-03-26T04:14:33.081348+0000 (oldest deadline 2024-03-26T04:14:54.781325+0000) 2024-03-26T04:14:55.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:54 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:54.737+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:55.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:55.093+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:55.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:55.093+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:14:56.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:55.822+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:56.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:55.787+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:56.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:55.898+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:56.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:55 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:55.898+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:33.081291+0000 front 2024-03-26T04:14:33.081348+0000 (oldest deadline 2024-03-26T04:14:54.781325+0000) 2024-03-26T04:14:56.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:56.117+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:56.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:56.117+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:14:57.087 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:56.836+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:56.871+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:56.871+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:56.858+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:56 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:56.858+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:33.081291+0000 front 2024-03-26T04:14:33.081348+0000 (oldest deadline 2024-03-26T04:14:54.781325+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:57.085+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:57.088 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:57.085+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:14:58.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:57.815+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:58.089 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:57.812+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.079736+0000 front 2024-03-26T04:14:25.079692+0000 (oldest deadline 2024-03-26T04:14:46.179650+0000) 2024-03-26T04:14:58.089 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4[37858]: 2024-03-26T04:14:57.812+0000 7fb887e5c700 -1 osd.4 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:33.081291+0000 front 2024-03-26T04:14:33.081348+0000 (oldest deadline 2024-03-26T04:14:54.781325+0000) 2024-03-26T04:14:58.089 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:57.911+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:58.089 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:57.911+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:14:58.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:58.103+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:58.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:58.103+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:14:58.872 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:58 smithi086 podman[167143]: 2024-03-26 04:14:58.615486761 +0000 UTC m=+5.807294254 container died 0ac4f4f875665ad8ff7fe81b124131a30a7d02d9b00ff2e33fcaf460facc915e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-26T04:14:58.872 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:58.782+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:58.872 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:58.782+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:14:59.271 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:58 smithi086 podman[167143]: 2024-03-26 04:14:58.995327421 +0000 UTC m=+6.187134915 container cleanup 0ac4f4f875665ad8ff7fe81b124131a30a7d02d9b00ff2e33fcaf460facc915e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-26T04:14:59.272 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:58 smithi086 bash[167143]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4 2024-03-26T04:14:59.272 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:58.871+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:14:59.272 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:58 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:58.871+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:14:59.272 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:59.111+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:14:59.272 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:14:59.111+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:14:59.566 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167160]: 2024-03-26 04:14:59.271698828 +0000 UTC m=+0.629484274 container remove 0ac4f4f875665ad8ff7fe81b124131a30a7d02d9b00ff2e33fcaf460facc915e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-26T04:14:59.837 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:59.792+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:14:59.837 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:14:59.792+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:14:59.838 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167251]: 2024-03-26 04:14:59.469481741 +0000 UTC m=+0.017956218 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:14:59.838 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167251]: 2024-03-26 04:14:59.597413086 +0000 UTC m=+0.145887554 container create 4ed0509825a524148c39100539d4d9ba6d08eddc262cebe496732b44d5253a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-26T04:15:00.097 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:59.847+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:00.097 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:14:59 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:14:59.847+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:00.097 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167251]: 2024-03-26 04:14:59.841705468 +0000 UTC m=+0.390179941 container init 4ed0509825a524148c39100539d4d9ba6d08eddc262cebe496732b44d5253a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:15:00.098 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167251]: 2024-03-26 04:14:59.845068267 +0000 UTC m=+0.393542743 container start 4ed0509825a524148c39100539d4d9ba6d08eddc262cebe496732b44d5253a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T04:15:00.098 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:14:59 smithi086 podman[167251]: 2024-03-26 04:14:59.941656626 +0000 UTC m=+0.490131094 container attach 4ed0509825a524148c39100539d4d9ba6d08eddc262cebe496732b44d5253a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS) 2024-03-26T04:15:00.463 INFO:journalctl@ceph.osd.4.smithi086.stdout:Mar 26 04:15:00 smithi086 podman[167251]: 2024-03-26 04:15:00.191803432 +0000 UTC m=+0.740277892 container died 4ed0509825a524148c39100539d4d9ba6d08eddc262cebe496732b44d5253a0f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:15:00.464 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:00.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:00.464 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:00.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:00.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T04:15:00.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:00.873 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T04:15:00.873 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T04:15:00.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5 2024-03-26T04:15:01.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:00.886+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:01.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:00.886+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:01.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:00.835+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:01.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:00 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:00.835+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:01.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:00 smithi086 systemd[1]: Stopping Ceph osd.5 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:01.433 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:01.089+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:01.433 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:01.089+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:01.433 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:01.248+0000 7f0776950700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:15:01.433 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:01.248+0000 7f0776950700 -1 osd.5 565 *** Got signal Terminated *** 2024-03-26T04:15:01.433 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:01.248+0000 7f0776950700 -1 osd.5 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:15:02.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:02.089+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:02.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:02.089+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:02.338 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:01.864+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:02.338 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:01.864+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:02.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:01.881+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:02.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:01 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:01.881+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:03.098 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:02.839+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:03.098 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:02.839+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:03.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:02.929+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:03.099 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:02 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:02.929+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:03.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:03.097+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:03.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:03.097+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:03.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:03.097+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:04.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:03.814+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:04.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:03.814+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:04.087 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:03.935+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:04.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:03.935+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:04.088 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:03 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:03.935+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.544951+0000 front 2024-03-26T04:14:41.544970+0000 (oldest deadline 2024-03-26T04:15:03.844896+0000) 2024-03-26T04:15:04.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:04.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:04.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:04.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:04.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:04.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:05.085 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:04.769+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:05.086 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:04.769+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:05.086 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:04.964+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:05.086 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:04.964+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:05.086 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:04 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:04.964+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.544951+0000 front 2024-03-26T04:14:41.544970+0000 (oldest deadline 2024-03-26T04:15:03.844896+0000) 2024-03-26T04:15:05.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:05.086+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:05.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:05.086+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:05.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:05.086+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:06.004 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:05.749+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:06.004 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:05 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:05.749+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:06.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:06.056+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:06.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:06.056+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:06.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:06.056+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:06.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:06.004+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:25.242714+0000 front 2024-03-26T04:14:25.242868+0000 (oldest deadline 2024-03-26T04:14:47.542574+0000) 2024-03-26T04:15:06.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:06.004+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:32.144130+0000 front 2024-03-26T04:14:32.143985+0000 (oldest deadline 2024-03-26T04:14:58.044010+0000) 2024-03-26T04:15:06.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5[91581]: 2024-03-26T04:15:06.004+0000 7f076ef3c700 -1 osd.5 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.544951+0000 front 2024-03-26T04:14:41.544970+0000 (oldest deadline 2024-03-26T04:15:03.844896+0000) 2024-03-26T04:15:07.067 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:06.795+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:07.067 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:06.795+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:07.067 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:06 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:06.795+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:07.068 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:06 smithi086 podman[167368]: 2024-03-26 04:15:06.752911961 +0000 UTC m=+5.638804090 container died a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T04:15:07.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:07.067+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:07.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:07.067+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:07.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:07.067+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:07.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167368]: 2024-03-26 04:15:07.073076 +0000 UTC m=+5.958968127 container cleanup a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:15:07.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 bash[167368]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5 2024-03-26T04:15:07.338 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167390]: 2024-03-26 04:15:07.299403745 +0000 UTC m=+0.532261677 container remove a7c853c9f7127626a95cc86dfbb23b85b4cf1532012672ed3ae087b92bca990f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:07.818+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:07.818+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:07 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:07.818+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167481]: 2024-03-26 04:15:07.616870664 +0000 UTC m=+0.103784813 container create eac2cc0d68b36916dedb7497bf5c48e38e126a8fde0786c9d9d50dc6ca3e1828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167481]: 2024-03-26 04:15:07.530712928 +0000 UTC m=+0.017627085 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167481]: 2024-03-26 04:15:07.861457915 +0000 UTC m=+0.348372076 container init eac2cc0d68b36916dedb7497bf5c48e38e126a8fde0786c9d9d50dc6ca3e1828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-03-26T04:15:07.944 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167481]: 2024-03-26 04:15:07.865040979 +0000 UTC m=+0.351955127 container start eac2cc0d68b36916dedb7497bf5c48e38e126a8fde0786c9d9d50dc6ca3e1828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-26T04:15:07.945 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:07 smithi086 podman[167481]: 2024-03-26 04:15:07.944348718 +0000 UTC m=+0.431262903 container attach eac2cc0d68b36916dedb7497bf5c48e38e126a8fde0786c9d9d50dc6ca3e1828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:15:08.337 INFO:journalctl@ceph.osd.5.smithi086.stdout:Mar 26 04:15:08 smithi086 podman[167481]: 2024-03-26 04:15:08.161705584 +0000 UTC m=+0.648619732 container died eac2cc0d68b36916dedb7497bf5c48e38e126a8fde0786c9d9d50dc6ca3e1828 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T04:15:08.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:08.078+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:08.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:08.078+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:08.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:08.078+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:08.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T04:15:08.786 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:08.786 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T04:15:08.786 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T04:15:08.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.6 2024-03-26T04:15:09.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:08.829+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:09.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:08.829+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:09.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:08 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:08.829+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:09.088 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:08 smithi086 systemd[1]: Stopping Ceph osd.6 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:09.587 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.119+0000 7f3f75606700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:15:09.587 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.119+0000 7f3f75606700 -1 osd.6 565 *** Got signal Terminated *** 2024-03-26T04:15:09.587 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.119+0000 7f3f75606700 -1 osd.6 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:15:09.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:09.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:09.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:09.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:09.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:09.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:10.125 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:10.124+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:10.125 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:10.124+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:10.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.852+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:10.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.852+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:10.125 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:09 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:09.852+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:10.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:10.124+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:11.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:10.823+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:11.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:10.823+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:11.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:10 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:10.823+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:11.504 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:11.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:11.504 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:11.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:11.504 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:11.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:12.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:11.821+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:12.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:11.821+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:12.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:11 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:11.821+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:12.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:12.138+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:12.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:12.138+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:12.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:12.138+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:12.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:12.138+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:13.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:12.794+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:13.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:12.794+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:13.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:12 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:12.794+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:13.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:13.158+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:13.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:13.158+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:13.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:13.158+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:13.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:13.158+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:14.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:13.779+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.468909+0000 front 2024-03-26T04:14:24.468961+0000 (oldest deadline 2024-03-26T04:14:46.168968+0000) 2024-03-26T04:15:14.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:13.779+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.269919+0000 front 2024-03-26T04:14:30.269823+0000 (oldest deadline 2024-03-26T04:14:56.169757+0000) 2024-03-26T04:15:14.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:13.779+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.971314+0000 front 2024-03-26T04:14:41.971419+0000 (oldest deadline 2024-03-26T04:15:06.071212+0000) 2024-03-26T04:15:14.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:13 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6[49905]: 2024-03-26T04:15:13.779+0000 7f3f6dbf2700 -1 osd.6 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:48.972087+0000 front 2024-03-26T04:14:48.972098+0000 (oldest deadline 2024-03-26T04:15:13.071925+0000) 2024-03-26T04:15:14.492 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:14.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:14.493 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:14.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:14.493 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:14.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:14.493 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:14 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:14.127+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:14.770 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:14 smithi086 podman[167585]: 2024-03-26 04:15:14.49243165 +0000 UTC m=+5.486723537 container died f1a3b6441e71d9936f2eecaa521fb8b0265839508c4b1be0f1cd788aad203558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-26T04:15:15.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:14 smithi086 podman[167585]: 2024-03-26 04:15:14.858546644 +0000 UTC m=+5.852838527 container cleanup f1a3b6441e71d9936f2eecaa521fb8b0265839508c4b1be0f1cd788aad203558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-26T04:15:15.087 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:14 smithi086 bash[167585]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6 2024-03-26T04:15:15.450 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167606]: 2024-03-26 04:15:15.143219092 +0000 UTC m=+0.633326816 container remove f1a3b6441e71d9936f2eecaa521fb8b0265839508c4b1be0f1cd788aad203558 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319) 2024-03-26T04:15:15.451 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:15.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:15.451 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:15.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:15.451 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:15.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:15.451 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:15 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:15.095+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:15.712 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167698]: 2024-03-26 04:15:15.353435997 +0000 UTC m=+0.017735533 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:15:15.712 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167698]: 2024-03-26 04:15:15.460685429 +0000 UTC m=+0.124984983 container create 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T04:15:15.712 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167698]: 2024-03-26 04:15:15.71289122 +0000 UTC m=+0.377190731 container init 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:15:16.020 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167698]: 2024-03-26 04:15:15.71599728 +0000 UTC m=+0.380296793 container start 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:15:16.020 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:15 smithi086 podman[167698]: 2024-03-26 04:15:15.787905895 +0000 UTC m=+0.452205406 container attach 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T04:15:16.307 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:16 smithi086 podman[167698]: 2024-03-26 04:15:16.021394677 +0000 UTC m=+0.685694199 container died 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T04:15:16.307 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.071+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:16.307 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.071+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:16.307 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.071+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:16.307 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.071+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:16.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T04:15:16.570 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:16 smithi086 podman[167724]: 2024-03-26 04:15:16.524361924 +0000 UTC m=+0.497208314 container remove 4a8355b8668c7c35da603abced42c17563c5ddcdc7aad4e9625a525675f0fc85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:15:16.570 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:16 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.6.service: Deactivated successfully. 2024-03-26T04:15:16.570 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:16 smithi086 systemd[1]: Stopped Ceph osd.6 for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:16.570 INFO:journalctl@ceph.osd.6.smithi086.stdout:Mar 26 04:15:16 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.6.service: Consumed 8.611s CPU time. 2024-03-26T04:15:16.586 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:16.587 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T04:15:16.587 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T04:15:16.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.7 2024-03-26T04:15:16.837 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 systemd[1]: Stopping Ceph osd.7 for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:17.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.934+0000 7f86ffb54700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:15:17.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.934+0000 7f86ffb54700 -1 osd.7 565 *** Got signal Terminated *** 2024-03-26T04:15:17.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:16 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:16.934+0000 7f86ffb54700 -1 osd.7 565 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:15:17.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:17.068+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:17.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:17.068+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:17.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:17.068+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:17.338 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:17 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:17.068+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:18.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:18.033+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:18.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:18.033+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:18.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:18.033+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:18.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:18 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:18.033+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:19.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:19.072+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:19.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:19.072+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:19.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:19.072+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:19.337 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:19 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:19.072+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:20.587 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:20.119+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:20.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:20.119+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:20.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:20.119+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:20.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:20.119+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:20.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:20 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:20.119+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.86:6802 osd.4 since back 2024-03-26T04:14:54.522519+0000 front 2024-03-26T04:14:54.522497+0000 (oldest deadline 2024-03-26T04:15:19.222542+0000) 2024-03-26T04:15:21.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:21.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6803 osd.0 since back 2024-03-26T04:14:24.917716+0000 front 2024-03-26T04:14:24.917750+0000 (oldest deadline 2024-03-26T04:14:50.817766+0000) 2024-03-26T04:15:21.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:21.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6807 osd.1 since back 2024-03-26T04:14:30.818433+0000 front 2024-03-26T04:14:30.818343+0000 (oldest deadline 2024-03-26T04:14:54.918281+0000) 2024-03-26T04:15:21.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:21.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6811 osd.2 since back 2024-03-26T04:14:41.820157+0000 front 2024-03-26T04:14:41.820110+0000 (oldest deadline 2024-03-26T04:15:02.919949+0000) 2024-03-26T04:15:21.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:21.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.6:6815 osd.3 since back 2024-03-26T04:14:46.321602+0000 front 2024-03-26T04:14:46.321557+0000 (oldest deadline 2024-03-26T04:15:11.621488+0000) 2024-03-26T04:15:21.588 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:21 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7[55822]: 2024-03-26T04:15:21.122+0000 7f86fc148700 -1 osd.7 565 heartbeat_check: no reply from 172.21.15.86:6802 osd.4 since back 2024-03-26T04:14:54.522519+0000 front 2024-03-26T04:14:54.522497+0000 (oldest deadline 2024-03-26T04:15:19.222542+0000) 2024-03-26T04:15:22.861 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:22 smithi086 podman[167802]: 2024-03-26 04:15:22.609333258 +0000 UTC m=+5.799766921 container died 1f4af2bc7d3f5c522953df178bae1bf9d0f497d79b217d826e56708a3f3848cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:15:23.161 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:22 smithi086 podman[167802]: 2024-03-26 04:15:22.952233606 +0000 UTC m=+6.142667271 container cleanup 1f4af2bc7d3f5c522953df178bae1bf9d0f497d79b217d826e56708a3f3848cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:15:23.161 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:22 smithi086 bash[167802]: ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7 2024-03-26T04:15:23.560 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167831]: 2024-03-26 04:15:23.161355004 +0000 UTC m=+0.552239330 container remove 1f4af2bc7d3f5c522953df178bae1bf9d0f497d79b217d826e56708a3f3848cf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:15:23.837 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167923]: 2024-03-26 04:15:23.464089178 +0000 UTC m=+0.017388228 image pull f18c6672b5a35a9268569822301af4f68c7c1f5d599252bb322be37fdc7da354 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717 2024-03-26T04:15:23.837 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167923]: 2024-03-26 04:15:23.578967688 +0000 UTC m=+0.132266780 container create 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-26T04:15:24.139 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167923]: 2024-03-26 04:15:23.847781929 +0000 UTC m=+0.401080980 container init 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea) 2024-03-26T04:15:24.139 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167923]: 2024-03-26 04:15:23.850898079 +0000 UTC m=+0.404197137 container start 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-26T04:15:24.139 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:23 smithi086 podman[167923]: 2024-03-26 04:15:23.914419771 +0000 UTC m=+0.467718836 container attach 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T04:15:24.400 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:24 smithi086 podman[167923]: 2024-03-26 04:15:24.139475654 +0000 UTC m=+0.692774714 container died 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:15:24.651 INFO:journalctl@ceph.osd.7.smithi086.stdout:Mar 26 04:15:24 smithi086 podman[167950]: 2024-03-26 04:15:24.628064256 +0000 UTC m=+0.483394391 container remove 26c0fe08d22716e8de8b0e6668368cd61384c0db7125774ebedd9389f4508341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:15:24.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T04:15:24.695 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:24.695 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T04:15:24.695 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T04:15:24.695 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T04:15:24.994 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:24 smithi006 systemd[1]: Stopping Ceph rgw.foo.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:25.261 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a[63504]: 2024-03-26T04:15:24.993+0000 7f81af351700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:15:25.282 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:24 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a[63504]: 2024-03-26T04:15:24.994+0000 7f81bdbd7ac0 -1 shutting down 2024-03-26T04:15:35.378 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:34 smithi006 bash[214887]: time="2024-03-26T04:15:34Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:15:35.378 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:35 smithi006 podman[214887]: 2024-03-26 04:15:35.024976978 +0000 UTC m=+10.126607364 container died cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-26T04:15:35.378 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:35 smithi006 podman[214887]: 2024-03-26 04:15:35.37745836 +0000 UTC m=+10.479088793 container cleanup cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:15:35.378 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:35 smithi006 bash[214887]: ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T04:15:35.639 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:35 smithi006 podman[214914]: 2024-03-26 04:15:35.587167034 +0000 UTC m=+0.554938795 container remove cdd20f56a02bbedaa60c87eed7e5e76b5301cda3599ceed5c3bdd9299271acf9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-26T04:15:35.640 INFO:journalctl@ceph.rgw.foo.a.smithi006.stdout:Mar 26 04:15:35 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T04:15:35.808 DEBUG:teuthology.orchestra.run.smithi006:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T04:15:35.847 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:35.847 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T04:15:35.847 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T04:15:35.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-de980954-eb22-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T04:15:36.144 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:35 smithi086 systemd[1]: Stopping Ceph prometheus.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:36.439 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:15:36.439 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:15:36.439 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.144Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.155Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.156Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.156Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a[66533]: ts=2024-03-26T04:15:36.156Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:15:36.440 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 podman[168029]: 2024-03-26 04:15:36.165126361 +0000 UTC m=+0.125050698 container died 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:36.795 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 podman[168029]: 2024-03-26 04:15:36.552423683 +0000 UTC m=+0.512348015 container cleanup 6235e0a48a9f13a94229dbe1aee409834dcc22fd31cbe8220e749fbdd421c797 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:36.795 INFO:journalctl@ceph.prometheus.a.smithi086.stdout:Mar 26 04:15:36 smithi086 bash[168029]: ceph-de980954-eb22-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:15:36.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-de980954-eb22-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T04:15:36.949 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:15:36.949 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T04:15:36.949 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid de980954-eb22-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:15:37.148 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T04:15:39.625 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:39 smithi006 systemd[1]: Stopping Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:39.889 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:39 smithi006 ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a[75652]: ts=2024-03-26T04:15:39.624Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:15:39.890 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:39 smithi006 podman[215396]: 2024-03-26 04:15:39.665747427 +0000 UTC m=+0.163895684 container died eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:40.309 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 podman[215396]: 2024-03-26 04:15:40.108374886 +0000 UTC m=+0.606523183 container cleanup eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:40.309 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 bash[215396]: ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:15:40.595 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 podman[215410]: 2024-03-26 04:15:40.309827574 +0000 UTC m=+0.631954658 container remove eba375f3b0eee83c856fb8b75b07bc85d7ba55316263f02982ed10967d94f8f2 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:40.596 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 podman[215410]: 2024-03-26 04:15:40.391247925 +0000 UTC m=+0.713374973 volume remove f0408a240d868d02129df5476b426e30c7536f19bd390bc63fb676cf49a35c73 2024-03-26T04:15:40.889 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:15:40.889 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 systemd[1]: Stopped Ceph alertmanager.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:40.889 INFO:journalctl@ceph.alertmanager.a.smithi006.stdout:Mar 26 04:15:40 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 1.704s CPU time. 2024-03-26T04:15:40.890 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:40 smithi006 systemd[1]: Stopping Ceph node-exporter.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:41.313 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 podman[215520]: 2024-03-26 04:15:41.056896936 +0000 UTC m=+0.111638411 container died 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:41.563 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 podman[215520]: 2024-03-26 04:15:41.396339569 +0000 UTC m=+0.451081049 container cleanup 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:41.564 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 bash[215520]: ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:15:41.828 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 podman[215534]: 2024-03-26 04:15:41.564013437 +0000 UTC m=+0.508062582 container remove 8ab9cc5d3ea31a9bc7e13208d4641ac8755445b484ceb62a81e73911b8a93977 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:15:41.828 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:15:41.828 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:15:41.828 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 systemd[1]: Stopped Ceph node-exporter.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:41.828 INFO:journalctl@ceph.node-exporter.a.smithi006.stdout:Mar 26 04:15:41 smithi006 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.154s CPU time. 2024-03-26T04:15:42.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid de980954-eb22-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:15:42.685 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T04:15:44.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:44 smithi086 systemd[1]: Stopping Ceph iscsi.iscsi.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:45.337 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:44 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a[60646]: debug Shutdown received 2024-03-26T04:15:55.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:54 smithi086 bash[168442]: time="2024-03-26T04:15:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:15:55.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:54 smithi086 podman[168442]: 2024-03-26 04:15:54.86783506 +0000 UTC m=+10.122937241 container died 13cf7e641c20cadd6b9d995ac60b5a0076ed12445fb1619d98d54c1333b57297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-26T04:15:55.121 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 podman[168442]: 2024-03-26 04:15:55.121756269 +0000 UTC m=+10.376858463 container cleanup 13cf7e641c20cadd6b9d995ac60b5a0076ed12445fb1619d98d54c1333b57297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_BRANCH=HEAD) 2024-03-26T04:15:55.122 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 bash[168442]: ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T04:15:55.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 podman[168458]: 2024-03-26 04:15:55.43976043 +0000 UTC m=+0.572044111 container remove 13cf7e641c20cadd6b9d995ac60b5a0076ed12445fb1619d98d54c1333b57297 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:036ff263f269d03295f88ce388558d8d1cbe14b2ee02cd78a973d9f233151717, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-adk-testing-2024-03-25-1628-e42b8ea, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-26T04:15:55.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T04:15:55.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T04:15:55.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 systemd[1]: Stopped Ceph iscsi.iscsi.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:55.752 INFO:journalctl@ceph.iscsi.iscsi.a.smithi086.stdout:Mar 26 04:15:55 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 3.525s CPU time. 2024-03-26T04:15:57.293 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 systemd[1]: Stopping Ceph grafana.a for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:57.572 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=server t=2024-03-26T04:15:57.305163439Z level=info msg="Shutdown started" reason="System signal: terminated" 2024-03-26T04:15:57.572 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=ticker t=2024-03-26T04:15:57.30549866Z level=info msg=stopped last_tick=2024-03-26T04:15:50Z 2024-03-26T04:15:57.572 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a[64122]: logger=tracing t=2024-03-26T04:15:57.305504091Z level=info msg="Closing tracing" 2024-03-26T04:15:57.572 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 podman[168720]: 2024-03-26 04:15:57.315445078 +0000 UTC m=+0.103961808 container died bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T04:15:57.837 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 podman[168720]: 2024-03-26 04:15:57.654639083 +0000 UTC m=+0.443155812 container cleanup bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T04:15:57.837 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 bash[168720]: ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:15:58.138 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:57 smithi086 podman[168734]: 2024-03-26 04:15:57.874869611 +0000 UTC m=+0.558105777 container remove bc49658358fab7ec7bbfb3c421a0c6361f7fa4330b04fa5082ec6472e0f31a88 (image=quay.io/ceph/grafana:9.4.12, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-grafana-a, maintainer=Grafana Labs ) 2024-03-26T04:15:58.139 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:58 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@grafana.a.service: Deactivated successfully. 2024-03-26T04:15:58.139 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:58 smithi086 systemd[1]: Stopped Ceph grafana.a for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:58.139 INFO:journalctl@ceph.grafana.a.smithi086.stdout:Mar 26 04:15:58 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@grafana.a.service: Consumed 17.602s CPU time. 2024-03-26T04:15:58.577 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:58 smithi086 systemd[1]: Stopping Ceph node-exporter.b for de980954-eb22-11ee-95d0-87774f69a715... 2024-03-26T04:15:58.837 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:58 smithi086 podman[168857]: 2024-03-26 04:15:58.578030494 +0000 UTC m=+0.116990171 container died bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:15:59.288 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:58 smithi086 podman[168857]: 2024-03-26 04:15:58.942362308 +0000 UTC m=+0.481321983 container cleanup bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:15:59.289 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:58 smithi086 bash[168857]: ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:15:59.289 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:59 smithi086 podman[168872]: 2024-03-26 04:15:59.184915129 +0000 UTC m=+0.611256559 container remove bd98a9d50e994cb709934536cb847fa9f95dddda65839397c3bf2ff12f5e24d2 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-de980954-eb22-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:15:59.289 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:59 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:15:59.581 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:59 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:15:59.581 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:59 smithi086 systemd[1]: Stopped Ceph node-exporter.b for de980954-eb22-11ee-95d0-87774f69a715. 2024-03-26T04:15:59.581 INFO:journalctl@ceph.node-exporter.b.smithi086.stdout:Mar 26 04:15:59 smithi086 systemd[1]: ceph-de980954-eb22-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 4.374s CPU time. 2024-03-26T04:16:00.029 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:16:00.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:16:00.097 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T04:16:00.099 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi006/crash 2024-03-26T04:16:00.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:16:00.165 INFO:teuthology.orchestra.run.smithi006.stderr:tar: /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:16:00.165 INFO:teuthology.orchestra.run.smithi006.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:16:00.167 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi086/crash 2024-03-26T04:16:00.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:16:00.217 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/de980954-eb22-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:16:00.217 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:16:00.218 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T04:16:00.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T04:16:00.315 INFO:teuthology.orchestra.run.smithi006.stdout:2024-03-26T03:51:25.619070+0000 mon.a (mon.0) 889 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-26T04:16:00.316 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T04:16:00.316 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[SEC\]' /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T04:16:00.351 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]' /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v MON_DOWN | egrep -v 'mons down, quorum' | egrep -v 'is down \(out of quorum\)' | egrep -v OSDMAP_FLAGS | egrep -v PG_DEGRADED | egrep -v OSD_DOWN | egrep -v POOL_APP_NOT_ENABLED | egrep -v CACHE_POOL_NO_HIT_SET | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-26T04:16:00.423 INFO:teuthology.orchestra.run.smithi006.stdout:2024-03-26T04:02:29.491553+0000 mon.a (mon.0) 1430 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL) 2024-03-26T04:16:00.423 INFO:tasks.cephadm:Compressing logs... 2024-03-26T04:16:00.423 DEBUG:teuthology.orchestra.run.smithi006:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:16:00.465 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:16:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-26T04:16:00.522 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:16:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T04:16:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:16:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T04:16:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.a.log: /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T04:16:00.535 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr: 56.7% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log 2024-03-26T04:16:00.536 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T04:16:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:16:00.539 INFO:teuthology.orchestra.run.smithi086.stderr:: No such file or directory 2024-03-26T04:16:00.546 INFO:teuthology.orchestra.run.smithi006.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log: /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.c.log: 91.8% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:16:00.551 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T04:16:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:16:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T04:16:00.552 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T04:16:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T04:16:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T04:16:00.553 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T04:16:00.554 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:16:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:16:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:16:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T04:16:00.555 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log 2024-03-26T04:16:00.569 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.b.log: /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log: 67.8% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T04:16:00.673 INFO:teuthology.orchestra.run.smithi086.stderr: 92.2% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T04:16:00.703 INFO:teuthology.orchestra.run.smithi006.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:16:00.705 INFO:teuthology.orchestra.run.smithi086.stderr: 91.1% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:16:00.714 INFO:teuthology.orchestra.run.smithi086.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:16:00.721 INFO:teuthology.orchestra.run.smithi006.stderr: 89.0% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:16:00.755 INFO:teuthology.orchestra.run.smithi086.stderr: 81.5% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:16:00.761 INFO:teuthology.orchestra.run.smithi086.stderr: 90.1% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:16:00.787 INFO:teuthology.orchestra.run.smithi086.stderr: 85.5% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:16:00.793 INFO:teuthology.orchestra.run.smithi006.stderr: 91.0% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:16:01.031 INFO:teuthology.orchestra.run.smithi006.stderr: 94.0% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:16:01.408 INFO:teuthology.orchestra.run.smithi086.stderr: 94.5% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log.gz 2024-03-26T04:16:01.909 INFO:teuthology.orchestra.run.smithi006.stderr: 94.4% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/objectstore_tool.2.log.gz 2024-03-26T04:16:02.842 INFO:teuthology.orchestra.run.smithi086.stderr: 92.9% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T04:16:03.484 INFO:teuthology.orchestra.run.smithi086.stderr: 91.1% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T04:16:04.006 INFO:teuthology.orchestra.run.smithi006.stderr: 89.7% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T04:16:09.871 INFO:teuthology.orchestra.run.smithi006.stderr: 91.1% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T04:16:19.463 INFO:teuthology.orchestra.run.smithi006.stderr: 90.3% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T04:17:33.845 INFO:teuthology.orchestra.run.smithi086.stderr: 93.5% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T04:17:51.555 INFO:teuthology.orchestra.run.smithi086.stderr: 93.8% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T04:18:51.834 INFO:teuthology.orchestra.run.smithi086.stderr: 93.1% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T04:18:51.835 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-26T04:18:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:real 2m51.355s 2024-03-26T04:18:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:user 3m55.164s 2024-03-26T04:18:51.835 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m9.129s 2024-03-26T04:27:43.137 INFO:teuthology.orchestra.run.smithi006.stderr: 93.6% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T04:36:14.172 INFO:teuthology.orchestra.run.smithi006.stderr: 92.9% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T04:42:59.129 INFO:teuthology.orchestra.run.smithi006.stderr: 92.6% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T04:43:44.484 INFO:teuthology.orchestra.run.smithi006.stderr: 92.8% -- replaced with /var/log/ceph/de980954-eb22-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T04:43:44.485 INFO:teuthology.orchestra.run.smithi006.stderr: 2024-03-26T04:43:44.485 INFO:teuthology.orchestra.run.smithi006.stderr:real 27m44.009s 2024-03-26T04:43:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:user 9m40.740s 2024-03-26T04:43:44.486 INFO:teuthology.orchestra.run.smithi006.stderr:sys 0m27.550s 2024-03-26T04:43:44.486 INFO:tasks.cephadm:Archiving logs... 2024-03-26T04:43:44.486 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi006/log 2024-03-26T04:43:44.488 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:45:09.424 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi086/log 2024-03-26T04:45:09.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:45:45.665 INFO:tasks.cephadm:Removing cluster... 2024-03-26T04:45:45.665 DEBUG:teuthology.orchestra.run.smithi006:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid de980954-eb22-11ee-95d0-87774f69a715 --force 2024-03-26T04:45:45.920 INFO:teuthology.orchestra.run.smithi006.stdout:Deleting cluster with fsid: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T04:45:46.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid de980954-eb22-11ee-95d0-87774f69a715 --force 2024-03-26T04:45:47.223 INFO:teuthology.orchestra.run.smithi086.stdout:Deleting cluster with fsid: de980954-eb22-11ee-95d0-87774f69a715 2024-03-26T04:45:47.834 INFO:tasks.cephadm:Removing cephadm ... 2024-03-26T04:45:47.835 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:45:47.850 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-26T04:45:47.866 INFO:tasks.cephadm:Teardown complete 2024-03-26T04:45:47.867 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T04:45:47.876 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T04:45:47.877 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T04:45:47.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T04:45:48.003 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T04:45:48.020 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T04:45:48.020 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:45:48.023 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:45:48.034 INFO:teuthology.orchestra.run.smithi006.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:45:48.039 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 1 +539us[ +539us] +/- 86ms 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 6 -558us[ -562us] +/- 53ms 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 64 +260us[ +253us] +/- 87ms 2024-03-26T04:45:48.054 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 4 +553us[ +554us] +/- 86ms 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 2 -544us[ -542us] +/- 53ms 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 4 +261us[ +263us] +/- 87ms 2024-03-26T04:45:48.076 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:45:48.077 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T04:45:48.125 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T04:45:48.126 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T04:45:48.211 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:45:48.297 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:45:48.383 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:45:48.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:45:48.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:45:48.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:45:48.441 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T04:45:48.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T04:45:49.032 INFO:teuthology.task.internal:Duration was 4605.382475 seconds 2024-03-26T04:45:49.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T04:45:49.219 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T04:45:49.219 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:45:49.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:45:49.278 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:45:49.306 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:45:49.713 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T04:45:49.714 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2024-03-26T04:45:49.714 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:45:49.735 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-26T04:45:49.735 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:45:49.773 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T04:45:49.773 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:45:49.778 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:45:49.844 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T04:45:49.845 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:45:49.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:45:51.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T04:45:51.257 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T04:45:51.257 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:45:51.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:45:51.340 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T04:45:51.349 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:45:51.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:45:51.402 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2024-03-26T04:45:51.476 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-26T04:45:51.498 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:45:51.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:45:51.512 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:45:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:45:51.527 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T04:45:51.536 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T04:45:51.537 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi006 2024-03-26T04:45:51.537 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:45:51.619 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451/remote/smithi086 2024-03-26T04:45:51.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:45:51.661 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T04:45:51.661 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:45:51.664 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:45:51.710 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T04:45:51.720 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T04:45:51.720 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T04:45:51.743 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T04:45:51.744 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:45:51.746 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:45:51.757 INFO:teuthology.orchestra.run.smithi006.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:45 /home/ubuntu/cephtest 2024-03-26T04:45:51.764 INFO:teuthology.orchestra.run.smithi086.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:45 /home/ubuntu/cephtest 2024-03-26T04:45:51.766 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T04:45:51.777 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T04:45:51.844 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T04:45:51.868 DEBUG:teuthology.nuke:shortname: smithi006 2024-03-26T04:45:51.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:45:51.892 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-26T04:45:51.893 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:45:51.896 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:20:17.101050', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyiWJ0FQ6XSMIajEJ/7O0IKWmXYqMAG1UOylFEqrCzvT3Q3DPyIL30yjheKqmJShRyRKzzSQXBGZkrtKcYRA/k='} 2024-03-26T04:45:51.919 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/adking-2024-03-26_01:50:53-orch:cephadm-wip-adk-testing-2024-03-25-1628-distro-default-smithi/7622451', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:20:17.102271', 'locked_by': 'scheduled_adking@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL+f2GjBWCT1ydzKcEaQJWK0YesqjDKanNpmvIou6emAhOAk6hAE/YoDz1vu2Z7hFgRLWIEt89/J800Sl87Ey6g='} 2024-03-26T04:45:51.943 INFO:teuthology.orchestra.console.smithi006:Power off 2024-03-26T04:45:51.944 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:45:51.971 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2024-03-26T04:45:51.971 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:45:51.986 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-26T04:45:51.986 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:45:51.997 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-26T04:45:52.009 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-26T04:45:52.009 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:45:52.039 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-26T04:45:55.998 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:45:56.040 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:45:56.108 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-26T04:45:56.149 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-26T04:46:00.111 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:46:00.150 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:46:00.221 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2024-03-26T04:46:00.260 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-26T04:46:00.361 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-26T04:46:04.222 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:46:04.332 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2024-03-26T04:46:04.433 INFO:teuthology.orchestra.console.smithi006:Power off completed 2024-03-26T04:46:04.537 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 4605.382475376129 failure_reason: '"2024-03-26T04:02:29.491553+0000 mon.a (mon.0) 1430 : cluster [ERR] Health check failed: full ratio(s) out of order (OSD_OUT_OF_ORDER_FULL)" in cluster log' flavor: default owner: scheduled_adking@teuthology success: false 2024-03-26T04:46:04.537 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:46:04.631 INFO:teuthology.run:FAIL